Public spec is needed for writing ORM adapters
jeffself
recently mentioned that IBM has written a DB2 driver for Python. This
driver uses the SQLAlchemy ORM. I asked him if IBM had any plans to
write a driver that would work with Django and here is his response:
"The API team (which currently maintains all the dynamic language APIs
for DB2) considers the Django adapter a priority. Due to IBM's strict
rules in terms of open source participation though, the developers
can't simply look at the code of the other adapters (not even the
dummy one) and adapt it for DB2. Therefore, in their case, the lack of
a public spec, that outlines method by method the requirements for any
new Django ORM adapters, is the biggest obstacle. If such a document
was freely published on the web (without restrictions), the API team
could come up with a Django adapter for CPython and for Jython in a
short amount of time."
Are there any plans to implement a public spec? While almost no one
uses DB2 in the community, it is still used quite a bit in the
government and commercial sector. I would love to have one more
barrier removed so I can bring Django into my work.
Ivan Sagalaev
> Are there any plans to implement a public spec? While almost no one
> uses DB2 in the community, it is still used quite a bit in the
> government and commercial sector. I would love to have one more
> barrier removed so I can bring Django into my work.
I can confirm from my recent experience of writing a DB backend that
some spec is needed. There is now a de-facto public interface
(".connection", "cursor()", "_commit()", "_rollback()") but nobody knows
what's going to change.
Malcolm Tredinnick
On Thu, 2008-03-20 at 06:20 -0700, jeffself wrote:
> My employer currently uses DB2 for its database. Antonio Cangiano
> recently mentioned that IBM has written a DB2 driver for Python. This
> driver uses the SQLAlchemy ORM. I asked him if IBM had any plans to
> write a driver that would work with Django and here is his response:
>
> "The API team (which currently maintains all the dynamic language APIs
> for DB2) considers the Django adapter a priority.
Such a priority that they've never brought this request to us in the
past. Guess that's why the word "high" wasn't used in that sentence. :-)
> Due to IBM's strict
> rules in terms of open source participation though, the developers
> can't simply look at the code of the other adapters (not even the
> dummy one) and adapt it for DB2. Therefore, in their case, the lack of
> a public spec, that outlines method by method the requirements for any
> new Django ORM adapters, is the biggest obstacle. If such a document
> was freely published on the web (without restrictions),
Except for copyright and normal open source licensing requirements,
presumably.
> the API team
> could come up with a Django adapter for CPython and for Jython in a
> short amount of time."
>
> Are there any plans to implement a public spec?
Well, there already is a spec. It's written in Python, which is mostly
Dutch pseudo-code rather than colloquial English, but it's fairly well
commented and the method names are sensible. Interestingly that's
enough, for example, for IBM's kernel developers, who regularly read
other parts of the kernel when constructing their APIs and driver
contributions.
Still, there's no accounting for some people's wish for paperwork. The
best way to continue would be for you to write up something based on the
current backends, send it here and we can review it. It's probably
something that would fit nicely alongside DevModelCreation in the wiki
and would be nice to have just for documenting how things work. We could
tidy up the DeveloperDocs page and promote it to "official" in some
sense. There'll probably be a bit of back-and-forth needed after the
first version to work out which features really are to be relied upon,
but certainly stuff in django/db/backends/__init__.py is going to be
pretty much an accurate representation of the state of play at the
moment. In fact, that file shows precisely how much duplicated effort
this "specification" requirement will require: it's got all the methods
you need, a honking great docstring on almost all of them -- adding
docstrings to the remaining ones will be easy and satisfy, say, Ivan's
requirements I would thinking -- and is laid out in a fashion that a
competent Python programmer can scan and implement really fast.
Regards,
Malcolm
--
The early bird may get the worm, but the second mouse gets the cheese.
http://www.pointy-stick.com/blog/
Jacob Kaplan-Moss
> [...] Due to IBM's strict
> rules in terms of open source participation though, the developers
> can't simply look at the code of the other adapters (not even the
> dummy one) and adapt it for DB2. Therefore, in their case, the lack of
> a public spec, that outlines method by method the requirements for any
> new Django ORM adapters, is the biggest obstacle.
You do realize that any document so written and released would be (a)
released under the same BSD license as the rest of Django's source and
would (b) contain any supposed tainted IP that IBM's so terrified of.
Keep in mind that the BSD license is the most permissive software
license to stand up to legal tests; IBM is perfect within their rights
as licensees to read, modify, and distribute -- commercially, even --
Django's source. I encourage them to do so.
I *highly* doubt that IBM engineers are not allowed to look at Open
Source -- in fact, I know IBM engineers who contribute to Open Source
daily. Perhaps they do it while wearing blinders So IBM is insinuating
here that Django's IP provenance is tainted. This is FUD, pure and
simple.
I'll bet at least someone from IBM reads this list. If that's the
case, wanna let me know -- publicly or privately -- exactly what the
problem with Django's IP is?
Jacob
Jacob Kaplan-Moss
No, I just wanted to add:
Of course we'd like the database backend API to be documented; it's on
my list of "internals" documents I want to start working on, and it's
fairly high up. I'm not, however, going to write it just because of
some vague legal threats of a has-been computer company.
Jacob
James Bennett
<mal...@pointy-stick.com> wrote:
> Well, there already is a spec. It's written in Python, which is mostly
> Dutch pseudo-code rather than colloquial English, but it's fairly well
> commented and the method names are sensible. Interestingly that's
> enough, for example, for IBM's kernel developers, who regularly read
> other parts of the kernel when constructing their APIs and driver
> contributions.
See also Zen of Python #14:
"Although that way may not be obvious at first unless you're Dutch."
Perhaps IBM needs more Dutchmen on its dev team?
Also, none of these concerns were raised six months ago when Antonio
Cangiano announced[1] that IBM would be contributing DB2 adapters for
Python and Python ORMs; Antonio, if you're on this list could you
chime in on this issue?
[1] http://antoniocangiano.com/2007/09/04/python-django-and-db2-status-update/
--
"Bureaucrat Conrad, you are technically correct -- the best kind of correct."
jeffself
On Mar 20, 10:53 am, "Jacob Kaplan-Moss" <jacob.kaplanm...@gmail.com>
wrote:
I don't think there was any legal threat directed at anyone.
Basically, it sounds like IBM is actually covering their own ass by
not allowing their developers to look at existing code for fear of
another lawsuit from someone like SCO. I think thats why they "need"
a public spec. Remember, IBM employs hundreds of developers that work
on open source projects including many who work on the Apache HTTP
Server.
But I'm glad to hear that you are planning to write documentation for
this. Good job!
Jacob Kaplan-Moss
> I don't think there was any legal threat directed at anyone.
Not directly, no. Think it though, though. If Django's IP is clean,
then there's nothing wrong with looking, using, adopting, modifying,
or distributing Django. That's the point of the BSD API. If, however,
there's a problem with Django's IP, then looking at the tainted IP
could be construed as copyright infringement. Thus, if IBM really has
this policy, it must be because someone there distrusts Django's IP.
I'd encourage that someone to contact me directly instead of spreading
insinuations.
> Basically, it sounds like IBM is actually covering their own ass by
> not allowing their developers to look at existing code for fear of
> another lawsuit from someone like SCO.
Who won that lawsuit, again?
> I think thats why they "need" a public spec.
Again, when I write that spec it will be released under the same BSD
license as Django itself, and will contain the same IP as Django does.
If Django's IP is tainted, one level of indirection won't protect IBM.
> Remember, IBM employs hundreds of developers that work
> on open source projects including many who work on the Apache HTTP
> Server.
I'm curious how they do this without reading anyone else's code.
Jacob
Jacob Kaplan-Moss
> Also, none of these concerns were raised six months ago when Antonio
> Cangiano announced[1] that IBM would be contributing DB2 adapters for
I think it's coming up now because IBM has customers who want to use
Django and who are dropping DB2 in favor of an Open Source database.
IBM wants in on the hot action [*] but wants us to do the heavy
lifting. Yay.
Jacob
Alex P
It's probably not that easy to make this context perfectly understood,
but I will make another attempt to straighten the record at least a
bit:
1. IBM does indeed have an incredible large number of developers
involved in the Open Source Software (OSS) development and one can
easily find them everywhere from apache.org to Linux kernel group. The
only difference between those fully OSS engaged developers and those
in the Python, Ruby, PHP, Perl support for IBM Data Servers
development is the scope of their work: the first are working solely
on OSS development, while the later might be involved in proprietary
developments, too. The idea is that no OSS code fragment gets spilled
"contaminating" proprietary IBM developments. So, it's solely an IBM's
IP pedigree preservation that could be at risk if some isolated
developer, without even intention (!) codes some proprietary component
in a similar way as something seen maybe years ago in an OSS
component. I hope this clears up the concerns about any perceived
insinuation; it's not the case.
2. What Python (and Ruby, PHP, etc.) developers in IBM are really
looking for is some level of development doc/guide that would get them
started (being able to start running compliance test suite).
SQLAlchemy, for example, has quite an extensive documentation, but was
still a bit of a struggle to extract the subset of API impl required
for a DB vendor. Subsequently, while fixing failures exhibited in the
tests, it's probably relatively easy to find what needs to be
overridden in the default implementation, and usually the community is
great in providing hints and real help. And yes, all these developers
are indeed in a very awkward, blind position that no OSS developer
would envy, and they usually ask: "why don't you check the existing
code, mate?"... well, yes one would think that to be easiest thing to
do, but that would expose one to the risk of later "contaminating"
some proprietary code with some OSS bits, and that is certainly
unacceptable, right?
In the end, to provide a concrete example of how members of the
community have helped the "blind" IBM developers implement SQLAlchemy
adapter: they started by just listing the subset of API that need to
be implemented (pointing to the signatures and API contract), and also
mentioning default APIs that likely needed to be overridden. Once
those were implemented and the first drat of the code started to spew
test case errors, it was relatively easy to see were SQL generated was
malformed, and needed to be overridden.
Hopefully this is clearing up some of the concerns on this issue in
this community, and I also hope our Dutch gets better with time :-)
Jacob Kaplan-Moss
Thanks for jumping on here to help us figure this stuff out. I'd love
to see a DB2 adaptor make its way into Django or be developed as a
third-party project.
On Thu, Mar 20, 2008 at 11:42 AM, Alex P <apit...@ca.ibm.com> wrote:
> 1. IBM does indeed have an incredible large number of developers
> involved in the Open Source Software (OSS) development and one can
> easily find them everywhere from apache.org to Linux kernel group. The
> only difference between those fully OSS engaged developers and those
> in the Python, Ruby, PHP, Perl support for IBM Data Servers
> development is the scope of their work: the first are working solely
> on OSS development, while the later might be involved in proprietary
> developments, too. The idea is that no OSS code fragment gets spilled
> "contaminating" proprietary IBM developments. So, it's solely an IBM's
> IP pedigree preservation that could be at risk if some isolated
> developer, without even intention (!) codes some proprietary component
> in a similar way as something seen maybe years ago in an OSS
> component. I hope this clears up the concerns about any perceived
> insinuation; it's not the case.
I think I must be dense here 'cause I've read this paragraph about
three times and don't understand what you're saying. So is the idea
that you've got essentially a firewall between those contributing to
Open Source and those merely using it? If so, is the problem that
nobody in the former group wants to work on Django/DB2?
Or is it deeper, and is Antonio correct that no IBM developer is
allowed to look at open source?
Either way, how would a spec -- released under the *same license* --
be any different in terms of IP than the code? If I copy a DB backend
into a .txt file and call it a spec can we move forward? If I rewrite
it in psudocode then is it OK?
> 2. What Python (and Ruby, PHP, etc.) developers in IBM are really
> looking for is some level of development doc/guide that would get them
> started (being able to start running compliance test suite).
See, this is a different problem then "we can't help Django because of
IP problems". Yes, we need to document the internals better; that's
never been in doubt. What I'm cheesed about is mixing that issue up
with all this IP business.
Which is the real problem?
> And yes, all these developers
> are indeed in a very awkward, blind position that no OSS developer
> would envy, and they usually ask: "why don't you check the existing
> code, mate?"... well, yes one would think that to be easiest thing to
> do, but that would expose one to the risk of later "contaminating"
> some proprietary code with some OSS bits, and that is certainly
> unacceptable, right?
Once again, Django's license explicitly allows this kind of
derivation. The BSD license is, in general, the most reuse-friendly
license you'll find. That is, you *have the copyright owner's
permission* to read the code. Do you think we're not qualified to give
that permssion because of IP provenance?
Also, please give my condolences to those poor developers. That's a
ridiculous position for IBM to put them in. Mention to them that there
are dozens of jobs coding Python and Django at companies without such
silly restrictions and that I'd be happy to help them make that move.
> In the end, to provide a concrete example of how members of the
> community have helped the "blind" IBM developers implement SQLAlchemy
> adapter: they started by just listing the subset of API that need to
> be implemented (pointing to the signatures and API contract), and also
> mentioning default APIs that likely needed to be overridden.
Right now, for better or for worse, the spec is the existing code. IIf
you're concerned about IP, you're fucked. No spec we release will have
different IP provenance, so I still don't see how a spec would help.
If, howeer, you just want better documentation, then it's message
heard loud and clear. Let's all shut up about IP and write some
documentation :)
Jacob
James Bennett
> IP pedigree preservation that could be at risk if some isolated
> developer, without even intention (!) codes some proprietary component
> in a similar way as something seen maybe years ago in an OSS
> component. I hope this clears up the concerns about any perceived
> insinuation; it's not the case.
As Jacob has pointed out, any documentation you receive will be under
the BSD license. If your developers aren't allowed to look at
BSD-licensed content for fear of getting contaminated with its evil
permissiveness, too bad for you, and honestly I wonder how IBM gets
anything done -- don't you kind of have to have a TCP/IP stack at some
point?
> 2. What Python (and Ruby, PHP, etc.) developers in IBM are really
> looking for is some level of development doc/guide that would get them
> started (being able to start running compliance test suite).
There's a dummy backend in django/db/backends which simply raises
errors on all the parts of the API. Of course, you say you're not
allowed to look at that.
> And yes, all these developers
> are indeed in a very awkward, blind position that no OSS developer
> would envy, and they usually ask: "why don't you check the existing
> code, mate?"... well, yes one would think that to be easiest thing to
> do, but that would expose one to the risk of later "contaminating"
> some proprietary code with some OSS bits, and that is certainly
> unacceptable, right?
Awkward and blind? Not really. Try "impossible".
> Hopefully this is clearing up some of the concerns on this issue in
> this community, and I also hope our Dutch gets better with time :-)
Again: any docs you'll get will be BSD licensed, because the docs will
be part of Django and Django is BSD licensed. If you can't look at
anything that's BSD licensed, that's your problem and your product --
DB2 -- will get left behind.
And, again, why didn't Antonio raise these issues six months ago when
he announced IBM would be working on these adapters?
Justin Lilly
I think what Alex is saying is that there are two subsets of IBM developers. OSS contributers (apache contributers, etc), and those who aren't (these people who want to create the db2 adapter). OSS contribs don't deal with IBM's IP. The people in the second category do deal with IP and are covered under a giant blanket rule that says "You are not allowed to look at source code!". And as giant corporations are often fans of blanket rules, by offering some non-code for them to look at (peudo-code) they are then escaping this blanket rule. While you and I realize that the license makes the blanket rule not matter, its a giant coproration and everyone knows they're slow to pick up on things like this.
To address James's post, I think they're not allowed to look at BSD licensed source code. BSD licensed documents are fine. See: Giant blanket statement. I'm sure all of this was in the Lawyer's Keynote at Pycon... if only I had stayed to the end...
I think that should clear it up. If not, I'd be happy to attempt a second time.
-justin
Justin Lilly
Web Developer/Designer
http://justinlilly.com
Rajeev J Sebastian
> look at source code!". And as giant corporations are often fans of blanket
> rules, by offering some non-code for them to look at (peudo-code) they are
> then escaping this blanket rule.
Question of course is, once they write the adapter with the
psuedo-code, how will they test it ? Wont they they accidentally see
the code in the debugger, or when they add print statements all over
the codebase ?
Regards
Rajeev J Sebastian
Jacob Kaplan-Moss
> I think what Alex is saying is that there are two subsets of IBM
> developers. OSS contributers (apache contributers, etc), and those who
> aren't (these people who want to create the db2 adapter). OSS contribs don't
> deal with IBM's IP. The people in the second category do deal with IP and
> are covered under a giant blanket rule that says "You are not allowed to
> look at source code!". And as giant corporations are often fans of blanket
> rules, by offering some non-code for them to look at (peudo-code) they are
> then escaping this blanket rule. While you and I realize that the license
> makes the blanket rule not matter, its a giant coproration and everyone
> knows they're slow to pick up on things like this.
Thanks for the clarification -- I understand now. Or rather, I don't
really understand why they continue to operate this way, but whatever,
not my fight.
> To address James's post, I think they're not allowed to look at BSD licensed
> source code. BSD licensed documents are fine. See: Giant blanket statement.
IANAL, so I'm asking this sincerely: is there a difference between
code and English such that IP provenance isn't an issue in a spec but
is in code?
Jacob
Alex P
development docs/guides. It's certainly not pseudo-code that we are
looking for, and yes that would be of concern. What would qualify as
public, unrestricted documentation or developer guide is solely the
required API signature and contract. While API signature is probably
well understood, the contract really means the API scope and
expectations in terms of input & entry conditions, output & exit
conditions.
This is really just basic design expectations that has nothing to do
with the actual code implementation. It's really not different from
asking a question on the forum about why some specific API doesn't
accept certain input or generate expected output, while the docs don't
mention anything on that. That being said, I can assure you that no
"blanket rule" will cover any of these, and such public documentation
is usually first checked by legal counselling in terms of
restrictions.
So, yes there's quite a difference between reading and "sucking in"
OSS code versus implementing an API contract specifically published
without restriction in order to get vendors contribute implementations
(i.e. think in terms on implementing any RFC or JSR).
James Bennett
> It seems it's now the time to clarify what we understand through
> development docs/guides. It's certainly not pseudo-code that we are
> looking for, and yes that would be of concern. What would qualify as
> public, unrestricted documentation or developer guide is solely the
> required API signature and contract. While API signature is probably
> well understood, the contract really means the API scope and
> expectations in terms of input & entry conditions, output & exit
> conditions.
Any such document provided by Django would be BSD licensed. You have
indicated that you are not able to look at any material which is BSD
licensed, for fear of contamination.
Unless you actually *are* allowed to look at BSD licensed material in
some cases, in which case something doesn't add up here.
Jacob Kaplan-Moss
> development docs/guides. It's certainly not pseudo-code that we are
> looking for, and yes that would be of concern.
Um, have you *seen* our documentation yet? It's chock-full of example
code. I'm not a good enough technical writer to write about code
without writing code.
If this is really your policy, your developers are violating it
whenever they open the Django documentation, read the book I wrote, or
participate on this mailing list or in IRC. All contain code taken
directly from the Open Source projects.
When I write the db backend documentation -- which, by the way, I just
started; thanks for the push! -- I'm going to use code to explain
things better explained in code. If IBM seriously can't deal with
that, it's not our problem; it's yours.
Jacob
jeffself
On Mar 20, 1:59 pm, "Jacob Kaplan-Moss" <jacob.kaplanm...@gmail.com>
wrote:
James Bennett
> I think I can sum all this up: Lawyers. We have to keep them employed.
A better summary: Employers. You can always find another one ;)
Alex P
easily :-P
Oh man... I guess it's my Dutch, but hey, I might find at least one
translator that could understand my bubbling :-D
Yes, that was always "our problem", and I don't recall ever trying to
imply otherwise. That being said, I am still confident we can find a
way out with a bit of understanding and maybe some community goodwill.
I know it's rather hard to deal with those different, if not "really
odd" ones, but it's users like Jeff that look for help more than those
few odd developers, so they "can bring Django into [their] work"
environments. It's definitely not Django's community responsibility by
any stretch of imagination, to deliver on this, but sometimes a bit of
community goodwill can go a long way, and might turn into win-win
situations.
So, what if keep this request at a very undemanding level: just
"expected vendor API signature and contract"? ... the way RFC and JSR
are provided for the community, but certainly not the same kind of
uber-detail. Is that outside the scope of this community?
P.S. oh, and yeah, these odd developers are also pretty much human,
maybe not that eager to find the perfect employer, but still likely to
have passions and admire technical excellence when they encounter, so
often in the Open Source.
Tom Tobin
> So, what if keep this request at a very undemanding level: just
> "expected vendor API signature and contract"? ... the way RFC and JSR
> are provided for the community, but certainly not the same kind of
> uber-detail. Is that outside the scope of this community?
Maybe I can distill this: the "expected vendor API signature and
contract" will be part of BSD-licensed documentation which will
include BSD-licensed code. There will be no public domain
documentation. Unless they can look at BSD-licensed documentation
(which will include BSD-licensed code), Django cannot help them.
Period.
::tries not to headdesk::
Jacob Kaplan-Moss
> So, what if keep this request at a very undemanding level: just
> "expected vendor API signature and contract"? ... the way RFC and JSR
> are provided for the community, but certainly not the same kind of
> uber-detail. Is that outside the scope of this community?
Can you point me at *any* documentation of Open Source that conforms
to this? I can't think of a single OSS project that somehow manages to
document their internals without using any code examples. Python's
documentation contains source, as does Linux's, PostgreSQL's, MySQL's,
and all the other ones I've ever seen.
I *still* can't believe that IBM seriously expects to operate in this
way. If this really is IBM's corporate policy w/r/t Open Source y'all
are in a world of hurt.
Jacob
James Bennett
> So, what if keep this request at a very undemanding level: just
> "expected vendor API signature and contract"? ... the way RFC and JSR
> are provided for the community, but certainly not the same kind of
> uber-detail. Is that outside the scope of this community?
If we're your vendor, when do we get a check for our work? ;)
Antonio Cangiano
dev mailing list, so this morning my IM lit up like a Christmas tree,
with everyone pointing me in the direction of this thread :) I do in
fact follow (albeit as an occasional lurker) this ML, as well as
Django Users and the group I founded (Django-it).
I'd like to take the opportunity to thank jeffself for bringing this
issue up and I want to clarify a few things from my standpoint.
When, six months ago, I announced that IBM would be working on a
Python driver and a Django adapter, all I did was provide the inside
scoop to interested developers outside of IBM. I'm not in the API
development team, even though I have great relationship with them and
they usually ask me for advice and feedback. After my initial
announcement, I let the team and their management do the work. And
that includes dealing with paperwork and legal counseling.
Some people love dealing with legality and paperwork. I don't. I, just
like you, love to code and I'm deeply passionate about technology.
Technical matters give me the adrenaline I need for my work, writing
about and promoting them trills me too, but I don't really care for
the red tape. Within IBM, just like with most big corporations, there
are certain rules that are defined by management on, and with, the
advice of lawyers. I don't have to like these rules myself, but I
can't violate them either.
The rules are the same though for any of the APIs for DB2. This means
that there is no insinuation that Django's particular IP is tainted.
In IBM's eyes, Django or Rails make no difference from a legal
standpoint. I believe, and again IANAL, that this is just a policy
enforced for legal reasons by IBM with any frameworks. DB2 on Rails[1]
[2], PHP or Perl are no different.
I have personally pushed the idea at IBM of having support for DB2
Express-C[3] and Django. In my eyes, having support for Rails but not
for Django, doesn't make any sense. Moreover, I'd like to see Django
continue to penetrate further within the enterprise world. If we
really want to change the enterprise, we need to inject technologies
like Rails and Django. However, in these types of environments, DB2 is
extremely popular and support for DB2 is fundamental if we want this
to happen (jeffself's scenario).
I believe in Django, and that's why I want to see it running on DB2 as
soon as possible. Even if it means that I have to write the adapter
myself at 3 AM. So don't construe this as a way of saying "do the work
for us". Looking at the other adapters' code and changing it to
reflect the particular DB2 datatypes and SQL dialect would be a walk
in the park. Python code is so nice and readable that it's almost the
same type of pseudo-code that's used to describe algorithms. But as an
IBMer, even if I'm not in the API team, I'm still bound by the same
rules and can't simply code up the adapter over a couple of nights -
however trivial it may seem to an outsider.
I want to clarify that it's not just a matter of getting Django in the
enterprise, or getting Django on DB2 in the enterprise environment.
DB2 Express-C is a fully functional free (as in beer) database, with a
lot of cool (technical) features. It has no limitations on data or
connections and can therefore be used in production by many, without
paying IBM a cent. For those who want to pay for support, the cost per
sever per year is the same as MySQL (which is not much). Therefore
even outside of the enterprise world, DB2 Express-C was able to gain
traction and a great number of downloads. The end result is that in
the last few months, I've been bombed by requests about DB2 and X,
where X is a language or a particular framework. Most of these
requests are for two languages: Ruby and Python. And as you may guess,
the Python requests are almost always for the best framework around,
Django.
A spec or better documentation, whatever the lawyers want to call it,
would have a copyright and a licensing agreement, just like code does,
but that'd be okay AFAIK. Of course we all agree here, that without a
specification and without the possibility of looking at the code, it's
extremely challenging, borderline impossible, to pull this one off.
I want to close this series of personal comments made strictly from my
viewpoint, by saying that I appreciate the hard work that's gone into
making Django one of, if not the, best web framework(s) available
today. I know you are very busy and I don't expect you to come up with
a spec that's ready for the team to use. If you do, that's terrific,
if someone like jeffself offers to do it with some help from you,
it'll be very appreciated, and I think it'll be a win-win situation.
But please don't think that we expect this from you. This is open
source we're talking about, you're not being paid from IBM, so if you
want to and can, do it because a better documented API may be good for
many people, but don't feel obligated, pressured or threatened in any
way.
Cheers,
Antonio Cangiano
--
http://stacktrace.it - Aperiodico di resistenza informatica.
http://antoniocangiano.com - Zen and the Art of Ruby Programming
Antonio Cangiano
> enforced for legal reasons by IBM with any frameworks. DB2 on Rails[1]
> [2], PHP or Perl are no different.
>
> I have personally pushed the idea at IBM of having support for DB2
> Express-C[3] and Django.
[1] http://rubyforge.org/projects/rubyibm/
[2] http://db2onrails.com
[3] http://www-306.ibm.com/software/data/db2/express/download.html
Cheers,
Antonio
Leon
to figure out where to insert the post.
Let me introduce myself. I am Leon Katsnelson and I am responsible for
Product Management for the IBM Data Management portfolio which
includes DB2 and IDS database servers as well as Data Studio to
mention just very few products. In other words, if there is anything
that is wrong with our approach you can blame me and leave poor Alex
and Antonio be. They are really good developers who thrive on
technology and have been pushing us to fully embrace Django.
Let me get right to the genesis of this thread. Will IBM be releasing
commercial grade Django driver for its databases? I can't say this in
an open forum. I can however confirm that we have been watching Django
with a great deal of interest as part of our overall commitment to
support of the scripting languages (PHP, Ruby, Python, Perl etc.). So,
if you follow what we have done with Ruby on Rails (DB2onRails.com) it
is a definite maybe :-)
A bit to the method to our madness. As you would expect IBM provides
solutions to corporations large and small and what we provide is
commercial software. IBM is a very big contributor of Open Source but
our business is "commercial" software. The lines between commercial
software and open source do get blurry from time to time as is the
case with DB2 Express-C (www.ibm.com/db2/express) for example. This is
a product that follows open source business model i.e. free license
with availability of inexpensive support and extra features. However,
DB2 Express-C is not open source. For most of our corporate customers
this is very important as they have restrictions on the use of open
source. I am not hear to argue the wisdom of having such restrictions,
I am simply stating the fact. One of the reasons (there are more) many
corporations have such restrictions is lack of "indemnification". In
other words, they don't like acquiring a product and getting it used
within their infrastructure and than finding out that they are a
target of a law suit claiming that they have misappropriated some IP.
If you think it does not happen just look at the case of SCO suing
Diamler-Chrysler over Linux. This does not happen to individual
developers and budding start-ups but if you are a large company with a
lot of money things are different. It is like you have a bulls eye
painted on your corporate logo and it is in the cross sight of every
IP lawyer who thinks he/she 1% chance of even having a semblance of a
case. it is sad that we live in such a litigious society but it is a
reality of IP law.
When IBM provides commercial software we guarantee its pedigree.
Before we ship any software we put together a Certificate of
Originality. This is one of the processes we have to ensure that all
of the code we are shipping to our customers was either written by IBM
employees and IBM has full rights to it or if the code came from some
place (e.g. open source) we are absolutely sure we have the rights to
use it. I am simplifying things a bit but this is the essence of the
due diligence of what we must do.
If you read so far you should get an idea by now that for IBM
developers taking an existing Django adapter and just creating one for
DB2 or IDS is not a simple proposition. That is not to say that we do
not leverage open source in IBM development; we do. But we do it with
great care. Yes, Django has is a fairly liberal license that would
allow us to do this i.e create what lawyers call "derivative work".
And thankfully it is not a viral license that would force us to open
source the entire product (DB2 or IDS) that would ship the adapter.
For us to look at Django code we need to be sure that we understand
the pedigree of that code so we inadvertently don't infringe on some
third party rights. For example, the code in the Django adapter could
have had contributions from someone who has not explicitly assigned
the rights and may not wish to have IBM use this code. Maybe this
person expects to be compensated for their contribution if a company
like IBM was to benefit from it. Again, this is not a hypothetical
scenario. This happens quite frequently.
The way we and other protect ourselves is by coding to a publicly
available specification. If there was such a spec for Django database
interfaces it would make it a lot simpler for us to commit to
developing a Django adapter for IBM DBMS.
My reason for making this post was to explain the need for a spec. I
am not hear to argue the wisdom of IBM practices; this is way out of
my domain of responsibility. Yes, we can read the "Dutch pseudo-code"
but we need to make sure that when if someone launches a lawsuit we
can clearly demonstrate that there are no lines of the pseudo-code
found their way in to IBM-supplied code. That is a lot more difficult
to do with the "Dutch pseudo-code" than with English. And it is really
difficult to argue with or our lawyers that the code we were looking
at is really a spec. Our developers will have to indicate that they
looked at the 3rd party code and not a spec and that is pretty much
where things will end i.e. no certificate of originality and no
ability to ship the product.
On Mar 20, 10:11 am, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:
Justin Bronn
IBM should understand our frustration when a spec exists (in
`django.db.backends`) but, because it is in source code form, your
particular IBM department cannot look at it. Thus, IBM is attempting
to shift the burden onto our community to produce onerous technical
documentation to _your_ requirements. This is the source of friction
you are experiencing on this list.
Moreover, there are actually several legitimate reasons for the lack
of a spec: (i) the database internals are being refactored, and (ii)
each RDBMS is different and has its own quirks that are dealt with in
its respective backend. For example, Oracle doesn't provide LIKE and
OFFSET capabilities, so those have to be worked around with a custom
QuerySet. Producing a heavy-weight public-domain spec will bind us to
a model that won't allow us to adapt to the different variations of
SQL dialects -- and I'm sure that DB2 and Informix will have their
fair share of quirks.
How was this problem solved with other OSS projects that IBM worked
with -- e.g, was this process been recorded for other OSS projects
(like Django) to examine? Was this on the SQLAlchemy mailing lists, in
private discussions with their developers, and/or all of the above?
Was this somehow less of an issue becaus SQLAlchemy and its
documentation are provided under the MIT license?
Due to the scarcity of the core developer's time don't expect this
documentation to appear absent some incentive. My last experience
trying to work with IBM's commercial databases was very painful[1],
and I personally would not invest myself again unless I was paid to.
If IBM is truly dedicated to having a Django database backend consider
transferring this to a department that can look at OSS code, or be
willing to hire someone to produce the documentation to your own
standards. Other risk-reduction measures, such as indemnification
insurance, are also available for your corporate customers.
Regards,
-Justin
[1] "Informix", django-users, http://groups.google.com/group/django-users/browse_thread/thread/3f0f772e84befaa1
Alex P
that needed to be addressed, I am now resurrecting this thread once
more:
1. First, surprisingly but consistently, the message gets
distorted, and for the benefit of the original poster and at least 5
or 6 other Python and DB2 enthusiasts in the community, I'd like to
clean up the noise once again and address anyone in the community that
has a vested interest in the subject
2. And second, does the majority of Django community really think a
web framework documentation as SQLAlchemy has is "onerous" (i.e.
according to dictionary: "troublesome or oppressive; burdensome ") ? I
hope not, but some may disagree and I can understand that, if given
some context.
So, the 2 issues that cry for clarification:
1. The original message was in brief: there is a growing interest in
the Python community for DB2 support, and we (developers behind IBM_DB
driver, DB-API wrapper and SQLAlchemy adapter) are interested to help
in the Django context if some _minimal_ documentation is provided in
an unrestricted form, even under a Open Source license like BSD (even
with sample/example API usage code, for that matter).
If those interested parties, which may unlikely include core Django
developers will be able to produce such minimal subset docs (of
external APIs required), and this docs could be reviewed informally by
Django gurus through this forum, then we can probably have "lift-off"
relatively soon after.
Conclusion: it's not a call to Django core developers, and we do
respect their opinion and limited time on the subject, it's a call to
interested parties already asking for ways to get this started,
therefore it's certainly far from any perceived "friction". I can only
hope, core Django developers would not feel any such pressure, but
just acknowledge the trend.
2. In more than one occasion, the _minimal_ "DB vendor specific" doc
(dev guide) was construed as "documenting internals", "burden", "heavy-
weight", while the request was specifically targeting the opposite:
solely API _externals_ (signature, and contract, i.e. input/output
params and entry/exit conditions), as-lightweight-as-possible-to-get-
started to run compliance test suite.
It is what worked with Rails and SQLAlchemy, and I can hardly see why
it wouldn't work with Django. The way it worked with SQLAlchemy was to
have one of the most motivated community member (Thanks Ken!), not a
core SA dev, submit a minimal set of API signatures that were a "must
implement" priority, and then step-by-step through direct dialogue we
found missing pieces of info that helped us fix the test suite
failures. That's were I also need to thank Florian Bösch and Michael
Bayer for their help with entry point support in SQLAlchemy. Really
wonderful people in the SQLAlchemy dev community !
But I think Jacob got it quite right in a post above: "Let's all shut
up ... and write some documentation :) "... It's a message mostly for
the Python and DB2 enthusiasts in this Django community, and we hope
to help them, too.
Thanks for reading this through with patience :-)
Alex P
James Bennett
> 1. The original message was in brief: there is a growing interest in
> the Python community for DB2 support, and we (developers behind IBM_DB
> driver, DB-API wrapper and SQLAlchemy adapter) are interested to help
> in the Django context if some _minimal_ documentation is provided in
> an unrestricted form, even under a Open Source license like BSD (even
> with sample/example API usage code, for that matter).
The thing is, though, that there *is* a minimal API documented:
django.db.backends.dummy. If this were Java, you could take that as
laying out the interface to be implemented. The confusion came from
the apparent refusal to deal with this because it's in the form of
open-source code.
Ned Batchelder
Unfortunately, django.db.backends.dummy falls below the bar for minimal API documentation. While it may have the names of the entry points, they are all defined as complain(*args, **kwargs), and they have no docstrings or comments, so there is no guidance about what they expect, do, or produce.
Perhaps James mis-spoke: django.db.backends contains base classes, and has much more useful information.
--Ned.
http://nedbatchelder.com/blog
-- Ned Batchelder, http://nedbatchelder.com
Niels
Given the existance of a django.db.backends.dummy module that probably
contains, in *some* form, all of the required information to produce a
document with "API signatures and entry/exit conditions", i would say
the easiest to go is to hire (or lend) a developer and have the module
code translated into Dutch, Spanish, English or whatever lingo IBM
lawyers be happy with and IBM devs can read and code to.
I think (but i do not speak for them) at least one core django
developer will be happy to look at the resulting document and comment
on it.
IBM is a very big contributor of Open Source but the business is
"commercial" software. Lines between commercial software and open
So, "Let's all shut up ... and write some documentation :) "... is a
message that can be acted up on by open source developers in IBM
offices as well. It will broaden the business opportunities for the
DB2 product so some investment for the lift off seems easily
justified.
Best regards
Niels
(wow, i'm Dutch :)
On Mar 25, 10:59 am, Ned Batchelder <n...@nedbatchelder.com> wrote:
> I thought I would end this standoff by simply going where James pointed,
> and turning it into the minimal "spec" Alex describes. I know next to
> nothing about db backends, but it sounded like this was a job for a
> scribe, not a developer.
>
> Unfortunately, django.db.backends.dummy falls below the bar for minimal
> API documentation. While it may have the names of the entry points,
> they are all defined as complain(*args, **kwargs), and they have no
> docstrings or comments, so there is no guidance about what they expect,
> do, or produce.
>
> Perhaps James mis-spoke: django.db.backends contains base classes, and
> has much more useful information.
>
>
>
>
>
>
> James Bennett wrote:
>
> >> 1. The original message was in brief: there is a growing interest in
> >> the Python community for DB2 support, and we (developers behind IBM_DB
> >> driver, DB-API wrapper and SQLAlchemy adapter) are interested to help
> >> in the Django context if some _minimal_ documentation is provided in
> >> an unrestricted form, even under a Open Source license like BSD (even
> >> with sample/example API usage code, for that matter).
>
> > The thing is, though, that there *is* a minimal API documented:
> > django.db.backends.dummy. If this were Java, you could take that as
> > laying out the interface to be implemented. The confusion came from
> > the apparent refusal to deal with this because it's in the form of
> > open-source code.
>
> --
>
> - Show quoted text -
James Bennett
> Unfortunately, django.db.backends.dummy falls below the bar for minimal API
> documentation. While it may have the names of the entry points, they are
> all defined as complain(*args, **kwargs), and they have no docstrings or
> comments, so there is no guidance about what they expect, do, or produce.
For more information, the SQLite backend is nicely implemented, and
fills in quite a bit if/when needed. And, amusingly, SQLite itself has
a policy of only implementing techniques that are demonstrably older
than the term of patent, so it's a great little DB to study for people
who have IP concerns ;)
jeffself
be glad to help out in whatever ways possible. I know Jacob said he
was going to write the documentation for this. What kind of
assistance can I provide?
I'll be glad to test the driver that eventually gets written as well.
Especially since I'm the one who needs it.
Robert
- ibm guys are developers
- community guys are supporters
- developers may NOT look existing code
- developers may look at technical specification
- since django hasn't got specification for writing backend -
developers can't do anything
- supporters may look at code and write specification based on this,
and provide it to developers (so. supporters are eyes for
developers ;) )
- developers can start job based on this - implement api based on
specification
- developers can test backend running django backend/all unittests
- supporters can help when developers hit problem that can't be
solved without reading the code or asking django developers directly
- supporters can do tests too
- after all tests are passed - we have "beta" - django developers
review this and decide if to integrate in trunk - or some adjustments
should be done before integration
- if this is going to be part of django trunk - there is license
issue which should be solved - django license and ibm license are not
the same
I needed db2 personally and I wrote one db2 driver myself (http://
code.google.com/p/ibm-db/issues/detail?id=10, http://code.djangoproject.com/ticket/6995),
and later I found that someone have done this before me (http://
code.djangoproject.com/ticket/5052). Maybe this could be of use.
= Maintaining problem =
On the other hand, IMHO this ibm license problem is pretty "strange"
for Open source project, you develop something open/free for project
having "source open" for reading ... That could be the problem in
future, each backend api improvement/change needs writing
specification - just for ibm guys.
I see two things that can be done:
- (easiest) if someone from ibm can do something to remove license
obstacle
- if django team can integrate specification which will is to be done
(if I have understood correctly Jacob suggestion, putting it in
backends\__init__.py as pydoc ...) and someone should maintain this
= Getting started =
For the start we could provide ibm team some backend api specification
to get things started, e.g.
generating pydoc for:
- http://code.djangoproject.com/browser/django/trunk/django/db/backends/__init__.py
- http://code.djangoproject.com/browser/django/trunk/django/db/backends/creation.py
- http://code.djangoproject.com/browser/django/trunk/django/db/backends/util.py
- http://code.djangoproject.com/browser/django/trunk/django/db/backends/dummy/
- ...
and explain shortly what is to be done.
If this suites ibm team and django team doesn't have complains, then
we could start (jeffself, Alex P from ibm, I'm willing to help
too, ...).
Mario Briggs
'Getting Started'.
Regarding the 'Maintaining problem', this is not a problem as
evidenced by the IBM Rails adaptor project as Rails has progressed
from 1.x to 2.x
On Apr 14, 2:26 pm, Robert <trebor7...@gmail.com> wrote:
> If have I understood problem well:
> - ibm guys are developers
> - community guys are supporters
> - developers may NOT look existing code
> - developers may look at technical specification
> - since django hasn't got specification for writing backend -
> developers can't do anything
> - supporters may look at code and write specification based on this,
> and provide it to developers (so. supporters are eyes for
> developers ;) )
> - developers can start job based on this - implement api based on
> specification
> - developers can test backend running django backend/all unittests
> - supporters can help when developers hit problem that can't be
> solved without reading the code or asking django developers directly
> - supporters can do tests too
> - after all tests are passed - we have "beta" - django developers
> review this and decide if to integrate in trunk - or some adjustments
> should be done before integration
> - if this is going to be part of django trunk - there is license
> issue which should be solved - django license and ibm license are not
> the same
>
> I neededdb2personally and I wrote onedb2driver myself (http://
> code.djangoproject.com/ticket/5052). Maybe this could be of use.
>
> = Maintaining problem =
> On the other hand, IMHO this ibm license problem is pretty "strange"
> for Open source project, you develop something open/free for project
> having "source open" for reading ... That could be the problem in
> future, each backend api improvement/change needs writing
> specification - just for ibm guys.
>
> I see two things that can be done:
> - (easiest) if someone from ibm can do something to remove license
> obstacle
> - if django team can integrate specification which will is to be done
> (if I have understood correctly Jacob suggestion, putting it in
> backends\__init__.py as pydoc ...) and someone should maintain this
>
> = Getting started =
> For the start we could provide ibm team some backend api specification
> to get things started, e.g.
> generating pydoc for:
> -http://code.djangoproject.com/browser/django/trunk/django/db/backends...
> -http://code.djangoproject.com/browser/django/trunk/django/db/backends...
> -http://code.djangoproject.com/browser/django/trunk/django/db/backends...