--You received this message because you are subscribed to the Google Groups "zodb" group.To unsubscribe from this group and stop receiving emails from it, send an email to zodb+uns...@googlegroups.com.For more options, visit https://groups.google.com/d/optout.
I'm working on updating Plone to support ZODB 4 and 5.So far most of the blockers are because BTrees 4.x no longer allows using keys whose ordering is not well-defined, including None, and there are various places where Plone was using None as a key. I understand the reason for the restriction and am making progress on making Plone no longer do this.
If there are unorderable values in the tree, is that going to cause corruption on adding new keys or removing other keys?
Some such keys may not be removable, at least in extreme cases.
Here I am back in ZODB3 again:
In [6]: class Bad(object):
...: def __eq__(self, other):
...: return False # Corner case extreme example of broken object
...:
In [8]: bt[Bad()] = 42
But it seems safe to say there are at least some discrepancies and some clear bugs in the handling of unorderable objects still.
> On Nov 20, 2016, at 10:25, Jim Fulton <j...@jimfulton.info> wrote:
>
>> But it seems safe to say there are at least some discrepancies and some clear bugs in the handling of unorderable objects still.
>
> Yup. Partly I suspect this is the related to the many ways one can express comparison in Python. (Talk about more than one way to do it.)
>
> I think this may also be related to inserting into an empty BTree, where there's noting to compare the new item to and thus no opportunity to fail. Perhaps, if we're paranoid, we should compare the first item to itself on insert.
You're right, under Python 3, this only happens in an empty tree. Once there are other items, you get "TypeError: unorderable types: Bad() < ..." errors. Under Python 2, of course, no such luck.
> In any case, I think the focus for this seat belt should be on insertion.
I've opened a series of issues in the repository (https://github.com/zopefoundation/BTrees/issues) that I think capture the discussion here.
I may be able to come up with some PRs over the next week for them.
On 11/20/16 9:31 AM, Jim Fulton wrote:
I finally had time to look at this again today. I've got a branch (check-obj-cmp-on-insert-only) that makes the CPython implementation only do the check on insertion so it's more like the Python implementation. However, trying to delete None as a key still raises "TypeError: unorderable types: NoneType() < NoneType()" in Python 3 (both implementations; we weren't testing the pure-Python implementation on Python 3 except for PyPy3). This is presumably Python itself complaining when trying to search for the bucket. I suppose the workaround is: for the search during delete only, if the keys use default comparison, compare them using a function that mimics Python 2 comparison and thus skips the check for unorderable types. Gaaa...
On 11/27/16 9:43 AM, Jim Fulton wrote:
On Sat, Nov 26, 2016 at 8:55 PM, David Glick <dgl...@gmail.com> wrote:
On 11/20/16 9:31 AM, Jim Fulton wrote:
I was thinking the same thing, and I've opened a pull request: https://github.com/zopefoundation/BTrees/pull/54It looks like David was going to try a fix. David, did this discussion help?
I finally had time to look at this again today. I've got a branch (check-obj-cmp-on-insert-only) that makes the CPython implementation only do the check on insertion so it's more like the Python implementation. However, trying to delete None as a key still raises "TypeError: unorderable types: NoneType() < NoneType()" in Python 3 (both implementations; we weren't testing the pure-Python implementation on Python 3 except for PyPy3). This is presumably Python itself complaining when trying to search for the bucket. I suppose the workaround is: for the search during delete only, if the keys use default comparison, compare them using a function that mimics Python 2 comparison and thus skips the check for unorderable types. Gaaa...
Well
a) We don't have a way to use databases created in Python 2 in Python 3 (do we?).
b) It's impossible to insert None as a key in Python 3.
If a & b, then this seems to be a non-issue and the tests should be Python version dependent.
But surely people have built apps with BTrees 4, because BTrees 4 was released a loooooooong time ago (late 2012), and this seatbelt was introduced in BTrees 4.0.When indexing content, you will very often encounter content without a value set, typically defaulting to None.When such values are indexed, you'll get an error. I don't see any guard against this error in, for example, zope.index.Have people built newer apps with indexing on BTrees 4? If so, how have you dealt with this issue?
On 1/5/17 11:30 AM, Jim Fulton wrote:
Not sure I agree that's simpler than doing nothing,
On Thu, Jan 5, 2017 at 2:25 PM, David Glick (Glick Software) <da...@glicksoftware.com> wrote:
On 1/5/17 11:18 AM, Jim Fulton wrote:
On Thu, Jan 5, 2017 at 1:11 PM, David Glick (Glick Software) <da...@glicksoftware.com> wrote:
On 1/5/17 10:01 AM, Jim Fulton wrote:
So, now the problem will move up through the application stack. :)Thanks for the release.
And perhaps become harder.
But surely people have built apps with BTrees 4, because BTrees 4 was released a loooooooong time ago (late 2012), and this seatbelt was introduced in BTrees 4.0.
When indexing content, you will very often encounter content without a value set, typically defaulting to None.
When such values are indexed, you'll get an error. I don't see any guard against this error in, for example, zope.index.
Have people built newer apps with indexing on BTrees 4? If so, how have you dealt with this issue?
Products.ZCatalog was updated some time ago to work with BTrees 4. It explictly checks for None and silently skips indexing it, to avoid erroring in that scenario: https://github.com/zopefoundation/Products.ZCatalog/blob/master/src/Products/PluginIndexes/unindex.py#L246
Thanks David (and Hanno).
Good to know. It's sad that zope.index hasn't gotten a fix.
In Postgres (and I assume other RDBMSs), you can specify how nulls are handled, so null values are still handled.
I wonder if that's something that should be done here as well. That is, I wonder if indexes should have some special None accommodation.
It certainly seems like a valid use case to support querying for items that don't have a value set. That could be done by storing a separate treeset of ids with no value, outside the BTree, of course.
The simplest way to do this, would be to add special handling of None in handling object keys, to treat None as always greater than everything but itself. This would be similar to Postgres' NULLS LAST in CREATE INDEX. (or less than / NULL FIRST).
but that does sound nicer than rejecting None just because Python doesn't know how to order it,
if we can enforce a reasonable ordering ourselves. Ordering None first would be the way to go if we care about backwards compatibility with how it got ordered in BTrees 3 on Python 2.
I'm going to work on PR to special case None, treating it as less than anything but itself.
Jim, there's a junk file in BTrees-4.4.0 (also in 4.3.2, I see) which is causing me problems with Windows packaging: #BTreeTemplate.c#.
Also saw a lot of junk files in the ZEO 5.0.4 tar file. A couple of checkpoint files, and a lot of Emacs backup ~ files.
Bill
On Tuesday, January 24, 2017 at 12:28:07 PM UTC-8, Jim Fulton wrote:On Tue, Jan 24, 2017 at 3:23 PM, Bill Janssen <bill.j...@gmail.com> wrote:Jim, there's a junk file in BTrees-4.4.0 (also in 4.3.2, I see) which is causing me problems with Windows packaging: #BTreeTemplate.c#.Gaaa. Someone configured packaging to include everything and then added a (inevitably incomplete) blacklist.Thanks. I'll make a 4.4.1 release that fixes this.Jim--
--
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zodb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Jim, there's a junk file in BTrees-4.4.0 (also in 4.3.2, I see) which is causing me problems with Windows packaging: #BTreeTemplate.c#.
--
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zodb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.