Changes today
Christopher Lenz
I've finally had a chance to work on couchdb-python today, and have
checked in a number of changes. In particular:
* The Database.update() has been changed as previously discussed on
this list [r151]
* A number of fixes to schema.ListField
* client.Server now has a delete() method (in addition to __delitem__)
* Added a copy() method to client.Database [r155,r159]
* Added a streaming MIME multipart writer for the couchdb-dump tool
[162], and fixes for the multipart parser.
I'd be grateful for any feedback and testing, in particular with
respect to Database.update and the streaming couchdb-dump.
If testing/review doesn't surface any really nasty issues, there
should be a release later this week (f-f-finally!)
Thanks,
--
Christopher Lenz
cmlenz at gmx.de
http://www.cmlenz.net/
Matt Goodall
>
> Hey folks,
>
> I've finally had a chance to work on couchdb-python today, and have
> checked in a number of changes. In particular:
>
> * The Database.update() has been changed as previously discussed on
> this list [r151]
The new Database.update() API looks good to me, but then I guess you'd
expect me to say that ;-).
However, when the doc revs are updated (the successful ones at least)
is it updating schema Documents too? I don't use schema at all but
they're not dict subclasses so I don't think the new rev will be set.
> * A number of fixes to schema.ListField
> * client.Server now has a delete() method (in addition to __delitem__)
> * Added a copy() method to client.Database [r155,r159]
> * Added a streaming MIME multipart writer for the couchdb-dump tool
> [162], and fixes for the multipart parser.
I found a couple of problems when testing against r163. I was going to
attach patches to this message but thought I might as well just commit
them. Feel free to revert or change anything you don't like.
* The required self arg is missing from
MultipartWriter._make_boundary. See r164.
* Missing method MultipartWriter.end() is called at the end of dump. See r165.
* End of line handling was a bit messed up (only a bit!) so I
refactored it a little to ensure it's using '\r\n' everywhere. See
r166.
* Documents with attachments fail. *Not* fixed yet.
Other than that, memory usage during dump and load stayed around the
10MB mark (as "measured" with Gnome's system monitor).
>
> I'd be grateful for any feedback and testing, in particular with
> respect to Database.update and the streaming couchdb-dump.
Unittests with Python 2.6 all succeed, but I see some failing doctests
with Python 2.5.
> If testing/review doesn't surface any really nasty issues, there
> should be a release later this week (f-f-finally!)
That would be great! I'll do what I can to test and help out.
>
> Thanks,
Thank you!
I need to get some "real" work done now but I'll try to go through the
changes later as well as test dump/load with some different databases.
Hope all this helps.
- Matt
Christopher Lenz
thanks for the reviewing and the fixes. Just a quick note:
On 30.06.2009, at 13:46, Matt Goodall wrote:
> * The required self arg is missing from
> MultipartWriter._make_boundary. See r164.
> * Missing method MultipartWriter.end() is called at the end of dump.
> See r165.
I renamed end() to close() pretty late in the process, and this was a
leftover. But we do still need to call close().
> * End of line handling was a bit messed up (only a bit!) so I
> refactored it a little to ensure it's using '\r\n' everywhere. See
> r166.
Hmm, we need to be careful then that the fileobj is not open in
universal-newline mode ('U') for the parser. Need to look closer, later.
> * Documents with attachments fail. *Not* fixed yet.
Hmm, I'm sure I did test those :P
Cheers,
Christopher Lenz
> On 30.06.2009, at 13:46, Matt Goodall wrote:
>> * The required self arg is missing from
>> MultipartWriter._make_boundary. See r164.
>> * Missing method MultipartWriter.end() is called at the end of dump.
>> See r165.
>
> I renamed end() to close() pretty late in the process, and this was a
> leftover. But we do still need to call close().
>
>> * End of line handling was a bit messed up (only a bit!) so I
>> refactored it a little to ensure it's using '\r\n' everywhere. See
>> r166.
>
> Hmm, we need to be careful then that the fileobj is not open in
> universal-newline mode ('U') for the parser. Need to look closer,
> later.
Okay, after some investigation I think we to make the parsing more
tolerant. Previously it expected just \n linebreaks, now it expects \r
\n linebreaks, but it should really be possible to hand it either. But
we still need to be careful not to mess up linebreaks in attachments.
I have a change ready locally, but currently can't check anything in
because svn@googlecode seems to be down. Will try later.
>> * Documents with attachments fail. *Not* fixed yet.
>
> Hmm, I'm sure I did test those :P
Doh, that was probably due to another leftover call to end() instead
of close().
Dirkjan Ochtman
> Okay, after some investigation I think we to make the parsing more
> tolerant. Previously it expected just \n linebreaks, now it expects \r
> \n linebreaks, but it should really be possible to hand it either. But
> we still need to be careful not to mess up linebreaks in attachments.
> I have a change ready locally, but currently can't check anything in
> because svn@googlecode seems to be down. Will try later.
Semi-relatedly: what about switching to h...@code.google instead?
Cheers,
Dirkjan
Christopher Lenz
> On 30.06.2009, at 15:05, Christopher Lenz wrote:
>> On 30.06.2009, at 13:46, Matt Goodall wrote:
>>> * The required self arg is missing from
>>> MultipartWriter._make_boundary. See r164.
>>> * Missing method MultipartWriter.end() is called at the end of dump.
>>> See r165.
>>
>> I renamed end() to close() pretty late in the process, and this was a
>> leftover. But we do still need to call close().
>>
>>> * End of line handling was a bit messed up (only a bit!) so I
>>> refactored it a little to ensure it's using '\r\n' everywhere. See
>>> r166.
>>
>> Hmm, we need to be careful then that the fileobj is not open in
>> universal-newline mode ('U') for the parser. Need to look closer,
>> later.
>
> Okay, after some investigation I think we to make the parsing more
> tolerant. Previously it expected just \n linebreaks, now it expects \r
> \n linebreaks, but it should really be possible to hand it either. But
> we still need to be careful not to mess up linebreaks in attachments.
> I have a change ready locally, but currently can't check anything in
> because svn@googlecode seems to be down. Will try later.
Okay, it's checked in now. I've also added Content-MD5 headers for all
the leaf parts in the dump, so that we can verify the integrity of the
content on load.
Cheers,
Chris
Christopher Lenz
> with Python 2.5.
Maybe due to an old version of simplejson? See
<http://code.google.com/p/couchdb-python/issues/detail?id=59&can=1&q=simplejson#c2
>
Cheers,
Matt Goodall
Yep, dump/load seems to work fine now, including attachments. Thanks.
- Matt
Matt Goodall
>
> On 30.06.2009, at 13:46, Matt Goodall wrote:
>> Unittests with Python 2.6 all succeed, but I see some failing doctests
>> with Python 2.5.
>
> Maybe due to an old version of simplejson? See
>
> <http://code.google.com/p/couchdb-python/issues/detail?id=59&can=1&q=simplejson#c2
> >
>
You were correct, I had a pre-2.0 version of simplejson installed for
Python 2.5. With the latest simplejson the doctests all pass.
- Matt
Christopher Lenz
I don't have a strong opinion either way. Might look into a switch
sometime after the 0.6 release. What do others think?
Sergey Shepelev
contributor, still, checkout/update speed is far better.
Jeff Hinrichs - DM&T
Regards,
Jeff
Dirkjan Ochtman
> I don't have a strong opinion either way. Might look into a switch
> sometime after the 0.6 release. What do others think?
Not much noise here, but people who responded were in favor. What do you think?
Cheers,
Dirkjan
Christopher Lenz
I'd like to make the move. Any tips on how that would go?
Dirkjan Ochtman
> I'd like to make the move. Any tips on how that would go?
Not sure how it works at GCode, but I'd be happy to make it work for you.
Cheers,
Dirkjan
Dirkjan Ochtman
> Not sure how it works at GCode, but I'd be happy to make it work for you.
Just found a tutorial at
http://code.google.com/p/support/wiki/ConvertingSvnToHg.
I'll prepare a hg repo you can pull from and host it somewhere.
Cheers,
Dirkjan
Dirkjan Ochtman
> I'll prepare a hg repo you can pull from and host it somewhere.
Okay, I've prepared three repos and put them on Bitbucket:
You might want to read up on the differences between clone- and named
branches in Mercurial (use 1.3 for the best named branches support).
-all has everything, from a straight hgsubversion conversion with a
small author map, -trunkrel has the default branch and all release
branches, but not the feature branch. -httplib is a clone feature
branch off of the default branch.
Since GCode currently doesn't support multiple clones per project, you
probably want to use named branches for release branches, but I'd
advise the use of clone branches for feature branches. See also
Python's PEP 385 for my reasoning concerning this issue for the Python
repo.
Let me know if anyone wants to know more/doesn't understand something.
Cheers,
Dirkjan
Christopher Lenz
> On Thu, Jul 16, 2009 at 15:27, Dirkjan Ochtman<dir...@ochtman.nl>
> wrote:
>> I'll prepare a hg repo you can pull from and host it somewhere.
>
> Okay, I've prepared three repos and put them on Bitbucket:
>
> http://bitbucket.org/djc/
>
> You might want to read up on the differences between clone- and named
> branches in Mercurial (use 1.3 for the best named branches support).
> -all has everything, from a straight hgsubversion conversion with a
> small author map, -trunkrel has the default branch and all release
> branches, but not the feature branch. -httplib is a clone feature
> branch off of the default branch.
Thanks a lot. I'll investigate the migration process over the next few
days.
> Since GCode currently doesn't support multiple clones per project, you
> probably want to use named branches for release branches, but I'd
> advise the use of clone branches for feature branches. See also
> Python's PEP 385 for my reasoning concerning this issue for the Python
> repo.
Just to make sure I understand... the feature branching approach you
suggest we use isn't actually supported by Google Code?
Christopher Lenz
> On 16.07.2009, at 16:18, Dirkjan Ochtman wrote:
>> On Thu, Jul 16, 2009 at 15:27, Dirkjan Ochtman<dir...@ochtman.nl>
>> wrote:
>>> I'll prepare a hg repo you can pull from and host it somewhere.
>>
>> Okay, I've prepared three repos and put them on Bitbucket:
>>
>> http://bitbucket.org/djc/
>>
>> You might want to read up on the differences between clone- and named
>> branches in Mercurial (use 1.3 for the best named branches support).
>> -all has everything, from a straight hgsubversion conversion with a
>> small author map, -trunkrel has the default branch and all release
>> branches, but not the feature branch. -httplib is a clone feature
>> branch off of the default branch.
>
> Thanks a lot. I'll investigate the migration process over the next few
> days.
So following the migration guide, I "hg convert"ed the repository, and
then switched the VCS on Google Code to Mercurial. After which
everything is now resulting in an internal server error :( Can't even
switch back to Subversion right now.
I'm hoping this will just take a couple of hours or something,
otherwise… ugh.
Dirkjan Ochtman
> So following the migration guide, I "hg convert"ed the repository, and
> then switched the VCS on Google Code to Mercurial. After which
> everything is now resulting in an internal server error :( Can't even
> switch back to Subversion right now.
Why didn't you use the repos I setup on Bitbucket? I used a better
conversion tool to do it. :)
Cheers,
Dirkjan
Christopher Lenz
The only thing I really *did* was change the dropdown on Google Code.
This is obviously a pretty bad bug on their side.
Anyway, what exactly does the conversion process you used do better
than hg convert? (Keeping in mind that the svn history of CouchDB-
Python is pretty darn simple).
Dirkjan Ochtman
> The only thing I really *did* was change the dropdown on Google Code.
> This is obviously a pretty bad bug on their side.
>
> Anyway, what exactly does the conversion process you used do better
> than hg convert? (Keeping in mind that the svn history of CouchDB-
> Python is pretty darn simple).
It keeps the order of revisions straight and simple. I also used an
author map (you can do that with convert, too, not sure if you did?).
And yeah, it's a pretty bad bug on GCode's site. Have you filed a bug
with them about it? If so, where is it?
Cheers,
Dirkjan
Christopher Lenz
> On 25/07/2009 21:30, Christopher Lenz wrote:
>> The only thing I really *did* was change the dropdown on Google Code.
>> This is obviously a pretty bad bug on their side.
>>
>> Anyway, what exactly does the conversion process you used do better
>> than hg convert? (Keeping in mind that the svn history of CouchDB-
>> Python is pretty darn simple).
>
> It keeps the order of revisions straight and simple. I also used an
> author map (you can do that with convert, too, not sure if you did?).
No, I forgot about that. But luckily I haven't pushed anything yet :)
The order of revisions in the hg converted repos looks straight and
simple to me.
Anyway, stupid question, how would I go about "using" the repos you
pushed to bitbucket? Just pull and push to googlecode?
(I'm still a newbie to hg and have so far only used it for local
development. I thought couchdb-python would be a good way to extend
that usage and learn.)
> And yeah, it's a pretty bad bug on GCode's site. Have you filed a bug
> with them about it? If so, where is it?
Apparently there's at least one other project with the same problem:
<http://code.google.com/p/support/issues/detail?id=2896>
Star away ;)
Dirkjan Ochtman
> Anyway, stupid question, how would I go about "using" the repos you
> pushed to bitbucket? Just pull and push to googlecode?
Yes, that should do the job. You can use the hgsubversion extension
(requires hg 1.3; get it from bitbucket, see also hg help extensions) to
pull additional changesets since my conversion.
> (I'm still a newbie to hg and have so far only used it for local
> development. I thought couchdb-python would be a good way to extend
> that usage and learn.)
You may want to review the hgbook online. There's also a dead-tree
version, called Mercurial: The Definitive Guide.
I'll star the issue.
Cheers,
Dirkjan
Dirkjan Ochtman
> No, I forgot about that. But luckily I haven't pushed anything yet :)
Any news on this? Here's a nice guide to distinguish named branches and clones:
http://stevelosh.com/blog/entry/2009/8/30/a-guide-to-branching-in-mercurial/
Cheers,
Dirkjan