Login | Register
My pages Projects Community openCollabNet

Discussions > dev > Fwd: Current with Oracle backend

Project highlights: Stable Version: 1.6.1, Development Version: 1.7.6

current
Discussion topic

Hide all messages in topic

All messages in topic

Re: Fwd: Current with Oracle backend

Reply

Author theslack
Full name Jack Neely
Date 2006-09-01 10:07:06 PDT
Message On 9/1/06, Pauline Middelink <middelink at polyware dot nl> wrote:
> On Fri, 01 Sep 2006 around 11:14:13 -0400, Jack Neely wrote:
> > >Well, the interface does not seem to change a lot,
> > > db -> connection -> cursor -> execute.
> > >They do claim to have an exceutemany, which in some cases could
> > >really be useful!
> > ><pauze>
> > >It does seem to be more general in use, translating the
> > >queries and stuff to the selected engine on the fly.
> > >Not a lot of work though...
> > >
> >
> > Folks,
> >
> > Seems I hadn't quite gotten to the part where SQLAlchemy documents how
> > to execute standard SQL. Which does make it less work than having to
> > port SQL to the python sql constructors. Could it be setup has
> > another database type?
>
> Well, probably, but we will be missing out on lots of features we
> than can't use because we need to program for the stupidest interface :(
> (like the use of the ?-bindings and executemany)
>
> Whereby sqlalchemy seems to have a feature rich interface and does the
> translating to 'stupid' databases below the surface.
>
> > Here's the thing: For this dev cycle (and my own goals) I don't have
> > time to do any major re-work of the database layer. I am interested
> > in SQLAlchemy and if I was starting from scratch now, that's probably
> > what I would use. So, if folks want to work on a SQLAlchemy branch I
> > will be supportive. But I don't see accepting it until the next dev
> > cycle.
>
> But... Err, how to put this. We are already 2+ years in dev cycle 1.7.x?
> What will be the timeframe for 1.9.x? I mean if we have to wait like
> 1.5 years, than its not worth the effort to track the changes in the
> base 1.7 to the SQLAlchemy branch.
>
> I think, before we go anyfurther, we need to see how big an impact
> it will be. Judging for the docs, it's a low-level impact. So, i'm
> willing to wrap up a little patch during the weekend so we know what
> we are talking about. The decision to include it yes/no can than be
> made on more facts.
>

I'd definitely like a better idea of the size of the impact. Thanks.

I need channels + yum + website to tackle whats coming down the pipe.
That's what I'd like 1.8 to be. Hopefully, sooner rather than later.
The 1.8.0 goals are in the TODO file...which I did just tweak
slightly. That's what I'm trying to stick with.

I really do appreciate folks' help and patches. Its a big project. :-)

> > Also, all the filesystem scanning/manipulation needs to come out of
> > currentdb.py and go into the Application layer. Something else to
> > keep in mind.
>
> Like putting _scanfilesystem and consorts in channels.py?
>
> (It still seems like lots of bouncing between app-layer and db-layer)
>
> Take for example the join I do in updateInstaledPackages(), do I need
> to replace that with 2 calls from the app layer to get the installed
> and package nvre's. Do the sort and do yet another third db-call to put
> the information back?

No, this is fine. You've got the right idea.

The filesystem manipulations are at this point pretty intertwined with
the database. However, its a significant part of the code in
currentdb.py and I feel goes beyond being data-centric. Its something
I need to spend time thinking about how to handle. So its not
happening soon.

Jack

>
> Dont get me wrong, I'm all for seperation of logic and db, but some
> data-centric stuff seems ok in the db-layer.
>
> Comments?
>
> Met vriendelijke groet,
> Pauline Middelink
> --
> GPG Key fingerprint = 2D5B 87A7 DDA6 0378 5DEA BD3B 9A50 B416 E2D0 C3C2
> For more details look at my website http://www.polyware.​nl/~middelink
>
>
>

Re: Fwd: Current with Oracle backend

Reply

Author Pauline Middelink <middelink at polyware dot nl>
Full name Pauline Middelink <middelink at polyware dot nl>
Date 2006-09-01 09:05:00 PDT
Message On Fri, 01 Sep 2006 around 11:14:13 -0400, Jack Neely wrote:
> >Well, the interface does not seem to change a lot,
> > db -> connection -> cursor -> execute.
> >They do claim to have an exceutemany, which in some cases could
> >really be useful!
> ><pauze>
> >It does seem to be more general in use, translating the
> >queries and stuff to the selected engine on the fly.
> >Not a lot of work though...
> >
>
> Folks,
>
> Seems I hadn't quite gotten to the part where SQLAlchemy documents how
> to execute standard SQL. Which does make it less work than having to
> port SQL to the python sql constructors. Could it be setup has
> another database type?

Well, probably, but we will be missing out on lots of features we
than can't use because we need to program for the stupidest interface :(
(like the use of the ?-bindings and executemany)

Whereby sqlalchemy seems to have a feature rich interface and does the
translating to 'stupid' databases below the surface.

> Here's the thing: For this dev cycle (and my own goals) I don't have
> time to do any major re-work of the database layer. I am interested
> in SQLAlchemy and if I was starting from scratch now, that's probably
> what I would use. So, if folks want to work on a SQLAlchemy branch I
> will be supportive. But I don't see accepting it until the next dev
> cycle.

But... Err, how to put this. We are already 2+ years in dev cycle 1.7.x?
What will be the timeframe for 1.9.x? I mean if we have to wait like
1.5 years, than its not worth the effort to track the changes in the
base 1.7 to the SQLAlchemy branch.

I think, before we go anyfurther, we need to see how big an impact
it will be. Judging for the docs, it's a low-level impact. So, i'm
willing to wrap up a little patch during the weekend so we know what
we are talking about. The decision to include it yes/no can than be
made on more facts.

> Also, all the filesystem scanning/manipulation needs to come out of
> currentdb.py and go into the Application layer. Something else to
> keep in mind.

Like putting _scanfilesystem and consorts in channels.py?

(It still seems like lots of bouncing between app-layer and db-layer)

Take for example the join I do in updateInstaledPackages(), do I need
to replace that with 2 calls from the app layer to get the installed
and package nvre's. Do the sort and do yet another third db-call to put
the information back?

Dont get me wrong, I'm all for seperation of logic and db, but some
data-centric stuff seems ok in the db-layer.

Comments?

    Met vriendelijke groet,
        Pauline Middelink
--
GPG Key fingerprint = 2D5B 87A7 DDA6 0378 5DEA BD3B 9A50 B416 E2D0 C3C2
For more details look at my website http://www.polyware.​nl/~middelink
Attachments

Re: Fwd: Current with Oracle backend

Reply

Author theslack
Full name Jack Neely
Date 2006-09-01 08:14:13 PDT
Message On 9/1/06, Pauline Middelink <middelink at polyware dot nl> wrote:
> On Thu, 31 Aug 2006 around 13:30:11 -0400, Jack Neely wrote:
> > Comments?
>
> Well, the interface does not seem to change a lot,
> db -> connection -> cursor -> execute.
> They do claim to have an exceutemany, which in some cases could
> really be useful!
> <pauze>
> It does seem to be more general in use, translating the
> queries and stuff to the selected engine on the fly.
> Not a lot of work though...
>

Folks,

Seems I hadn't quite gotten to the part where SQLAlchemy documents how
to execute standard SQL. Which does make it less work than having to
port SQL to the python sql constructors. Could it be setup has
another database type?

Here's the thing: For this dev cycle (and my own goals) I don't have
time to do any major re-work of the database layer. I am interested
in SQLAlchemy and if I was starting from scratch now, that's probably
what I would use. So, if folks want to work on a SQLAlchemy branch I
will be supportive. But I don't see accepting it until the next dev
cycle.

Also, all the filesystem scanning/manipulation needs to come out of
currentdb.py and go into the Application layer. Something else to
keep in mind.

Jack

Re: Fwd: Current with Oracle backend

Reply

Author Pauline Middelink <middelink at polyware dot nl>
Full name Pauline Middelink <middelink at polyware dot nl>
Date 2006-09-01 03:25:09 PDT
Message On Thu, 31 Aug 2006 around 13:30:11 -0400, Jack Neely wrote:
> Comments?

Well, the interface does not seem to change a lot,
    db -> connection -> cursor -> execute.
They do claim to have an exceutemany, which in some cases could
really be useful!
<pauze>
It does seem to be more general in use, translating the
queries and stuff to the selected engine on the fly.
Not a lot of work though...

> ---------- Forwarded message ----------
> From: Jared Greenwald <jared.greenwald@​oracle.com>
> Date: Aug 31, 2006 1:15 PM
> Subject: Re: Current with Oracle backend
> To: Jack Neely <jjneely at gmail dot com>
>
>
> Actually, re: DB APIs, might want to consider using SQLAlchemy... Looks
> like it has built-in support for the following:
>
> * Postgres: psycopg2
> * SQLite: pysqlite
> * MySQL: MySQLDB
> * Oracle: cx_Oracle
> * MS-SQL: adodbapi pymssql
> * Firebird: kinterbasdb
>
> This would probably allow you to eliminate most of the db layer - it
> also takes into account differences in the way that bindvars are
> handled, so the only thing that would be db-specific would be the
> schemas....
>
> Just a thought.
>
> -Jared
>
> Jack Neely wrote:
> > The logger.py module that everything imports has been ported to use
> > python's logging stuff.
> >
> > Jack
> >
> > On 8/30/06, Jared Greenwald <jared.greenwald@​oracle.com> wrote:
> > Something I just noticed. You mentioned transitioning to the logger
> > module, but in reading the pysqlite.py code, it seems to be still using
> > the home-brew logger method. Is this intentional?
> >
> > -Jared
> >
> > Jack Neely wrote:
> >> Yeah...in theory you could create the database by hand. You got it.
> >
> >> The pyc/pyo files you don't need to worry about. Python will manage
> >> those as it needs. (Of course, propper RPM packaging takes a bit more
> >> work, but the current spec file should work. There have been a lot of
> >> changes sence 1.7.4 so not sure...)
> >
> >> Jack
> >
> >> On 8/29/06, Jared Greenwald <jared.greenwald@​oracle.com> wrote:
> >> But other than just calling the initdb routine, there's really nothing
> >> more to it, right? (As in, if I created the db off to the side, it
> >> shouldn't matter, right? As long as it had the correct layout.)
> >
> >> I had previously made changes to currentdb.py (to add the new db type)
> >> and created my own oracle sub-dir (of db) and created the __init__.py
> >> and oracle.py files.
> >
> >> -Jared
> >
> >> Jack Neely wrote:
> >>> cinstall calls current.db.currentdb​.CurrentDB().initdb(​) which calls
> >>> the initdb() method of the specificDB object that is associated with
> >>> that specific database, in your case Oracle.
> >
> >>> You may also want to see current/db/__init__.py.
> >
> >>> Jack
> >
> >>> On 8/29/06, Jared Greenwald <jared.greenwald@​oracle.com> wrote:
> >>> Where is the initdb code located? I'm thinking that there might be
> >> some
> >>> issues with how it creates the db, but I want to check over the code
> >>> first.
> >
> >>> Thanks for all the help. I really appreciate it.
> >
> >>> -Jared
> >
> >>> Jack Neely wrote:
> >>>> Jared,
> >
> >>>> Yeah...you need to edit the current.conf config file, cinstall
> > initdb,
> >>>> create_channel, add_dir, and scan. Current will still touch the
> >>>> filesystem (some files are named from a datestamp) but it knows what
> >>>> to do if stuff is already there.
> >
> >>>> Jack
> >
> >>>> On 8/29/06, Jared Greenwald <jared.greenwald@​oracle.com> wrote:
> >>>> Excellent. Works like a charm. If I'm going to change databases,
> >> so I
> >>>> need to re-configure anything other than having the schema setup? I
> >>>> imagine that I'll have to create_channel, add_dir, scan, etc... to
> >>>> re-setup this channel...
> >
> >>>> -Jared
> >
> >>>> Jack Neely wrote:
> >>>>> Jarad,
> >
> >>>>> up2date --showall wont work until the initial cadmin scan. I can
> >> tell
> >>>>> you haven't done the scan because of this
> >
> >>>>> 'lastupdate': None
> >
> >>>>> in the logs.
> >
> >>>>> The scan wont work until apache has the proper permissions for
> >>>>> /export/current
> >
> >>>>> Jack
> >
> >
> >
> >>
>
> --------------------​--------------------​--------------------​---------
> To unsubscribe, e-mail: dev-unsubscribe@curr​ent.tigris.org
> For additional commands, e-mail: dev-help at current dot tigris dot org

    Met vriendelijke groet,
        Pauline Middelink
--
GPG Key fingerprint = 2D5B 87A7 DDA6 0378 5DEA BD3B 9A50 B416 E2D0 C3C2
For more details look at my website http://www.polyware.​nl/~middelink
Attachments

Fwd: Current with Oracle backend

Reply

Author theslack
Full name Jack Neely
Date 2006-08-31 10:30:11 PDT
Message Comments?

---------- Forwarded message ----------
From: Jared Greenwald <jared.greenwald@​oracle.com>
Date: Aug 31, 2006 1:15 PM
Subject: Re: Current with Oracle backend
To: Jack Neely <jjneely at gmail dot com>


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Actually, re: DB APIs, might want to consider using SQLAlchemy... Looks
like it has built-in support for the following:

    * Postgres: psycopg2
    * SQLite: pysqlite
    * MySQL: MySQLDB
    * Oracle: cx_Oracle
    * MS-SQL: adodbapi pymssql
    * Firebird: kinterbasdb

This would probably allow you to eliminate most of the db layer - it
also takes into account differences in the way that bindvars are
handled, so the only thing that would be db-specific would be the
schemas....

Just a thought.

- -Jared

Jack Neely wrote:
> The logger.py module that everything imports has been ported to use
> python's logging stuff.
>
> Jack
>
> On 8/30/06, Jared Greenwald <jared.greenwald@​oracle.com> wrote:
> Something I just noticed. You mentioned transitioning to the logger
> module, but in reading the pysqlite.py code, it seems to be still using
> the home-brew logger method. Is this intentional?
>
> -Jared
>
> Jack Neely wrote:
>> Yeah...in theory you could create the database by hand. You got it.
>
>> The pyc/pyo files you don't need to worry about. Python will manage
>> those as it needs. (Of course, propper RPM packaging takes a bit more
>> work, but the current spec file should work. There have been a lot of
>> changes sence 1.7.4 so not sure...)
>
>> Jack
>
>> On 8/29/06, Jared Greenwald <jared.greenwald@​oracle.com> wrote:
>> But other than just calling the initdb routine, there's really nothing
>> more to it, right? (As in, if I created the db off to the side, it
>> shouldn't matter, right? As long as it had the correct layout.)
>
>> I had previously made changes to currentdb.py (to add the new db type)
>> and created my own oracle sub-dir (of db) and created the __init__.py
>> and oracle.py files.
>
>> -Jared
>
>> Jack Neely wrote:
>>> cinstall calls current.db.currentdb​.CurrentDB().initdb(​) which calls
>>> the initdb() method of the specificDB object that is associated with
>>> that specific database, in your case Oracle.
>
>>> You may also want to see current/db/__init__.py.
>
>>> Jack
>
>>> On 8/29/06, Jared Greenwald <jared.greenwald@​oracle.com> wrote:
>>> Where is the initdb code located? I'm thinking that there might be
>> some
>>> issues with how it creates the db, but I want to check over the code
>>> first.
>
>>> Thanks for all the help. I really appreciate it.
>
>>> -Jared
>
>>> Jack Neely wrote:
>>>> Jared,
>
>>>> Yeah...you need to edit the current.conf config file, cinstall
> initdb,
>>>> create_channel, add_dir, and scan. Current will still touch the
>>>> filesystem (some files are named from a datestamp) but it knows what
>>>> to do if stuff is already there.
>
>>>> Jack
>
>>>> On 8/29/06, Jared Greenwald <jared.greenwald@​oracle.com> wrote:
>>>> Excellent. Works like a charm. If I'm going to change databases,
>> so I
>>>> need to re-configure anything other than having the schema setup? I
>>>> imagine that I'll have to create_channel, add_dir, scan, etc... to
>>>> re-setup this channel...
>
>>>> -Jared
>
>>>> Jack Neely wrote:
>>>>> Jarad,
>
>>>>> up2date --showall wont work until the initial cadmin scan. I can
>> tell
>>>>> you haven't done the scan because of this
>
>>>>> 'lastupdate': None
>
>>>>> in the logs.
>
>>>>> The scan wont work until apache has the proper permissions for
>>>>> /export/current
>
>>>>> Jack
>
>
>
>>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFE9xlDu4z7Xptg​TUYRAkf7AKCA3NoGNkpp​jZaCeEG5x+1OOuAekQCf​Qc4B
ts43FaLQ7naBvE4TZZcjZGM=
=7+vP
-----END PGP SIGNATURE-----
Messages per page: