Index failure after upgrade to ReviewBoard 2.5.16
77 views
Skip to first unread message
Rob Backhurst
Oct 3, 2017, 4:12:26 AM10/3/17
to reviewboard
Hi,
I upgraded our ReviewBoard system from 2.5.10 to 2.5.16 - since then, indexing doesn't seem to complete.
The index starts OK, but after a while stops with this error...
ERROR:root:Error updating reviews using default
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 189, in handle_label
self.update_backend(label, using)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 234, in update_backend
do_update(backend, index, qs, start, end, total, self.verbosity)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 89, in do_update
backend.update(index, current_qs)
File "/usr/lib/python2.7/site-packages/haystack/backends/whoosh_backend.py", line 191, in update
doc = index.full_prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 207, in full_prepare
self.prepared_data = self.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 198, in prepare
self.prepared_data[field.index_fieldname] = field.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 159, in prepare
return self.convert(super(CharField, self).prepare(obj))
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 87, in prepare
raise SearchFieldError("The model '%s' does not have a model_attr '%s'." % (repr(obj), attr))
SearchFieldError: The model '<ReviewRequest: <CUT_REVIEW_NAME>' does not have a model_attr 'commit'.
Traceback (most recent call last):
File "/usr/bin/rb-site", line 9, in <module>
load_entry_point('ReviewBoard==2.5.16', 'console_scripts', 'rb-site')()
File "/usr/lib/python2.7/site-packages/reviewboard/cmdline/rbsite.py", line 1964, in main
command.run()
File "/usr/lib/python2.7/site-packages/reviewboard/cmdline/rbsite.py", line 1884, in run
site.run_manage_command(args[0], args[1:])
File "/usr/lib/python2.7/site-packages/reviewboard/cmdline/rbsite.py", line 712, in run_manage_command
execute_from_command_line([__file__, cmd] + params)
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/usr/lib/python2.7/site-packages/reviewboard/reviews/management/commands/index.py", line 21, in handle
call_command('update_index')
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 159, in call_command
return klass.execute(*args, **defaults)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 184, in handle
return super(Command, self).handle(*items, **options)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 385, in handle
label_output = self.handle_label(label, **options)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 189, in handle_label
self.update_backend(label, using)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 234, in update_backend
do_update(backend, index, qs, start, end, total, self.verbosity)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 89, in do_update
backend.update(index, current_qs)
File "/usr/lib/python2.7/site-packages/haystack/backends/whoosh_backend.py", line 191, in update
doc = index.full_prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 207, in full_prepare
self.prepared_data = self.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 198, in prepare
self.prepared_data[field.index_fieldname] = field.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 159, in prepare
return self.convert(super(CharField, self).prepare(obj))
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 87, in prepare
raise SearchFieldError("The model '%s' does not have a model_attr '%s'." % (repr(obj), attr))
haystack.exceptions.SearchFieldError: The model '<ReviewRequest: CUT_REVIEW_NAME>' does not have a model_attr 'commit'.
Any ideas what could be causing this?
We are running this on CentOS Linux release 7.4.1708 (Core)
Thanks
Rob
Rob Backhurst
Oct 4, 2017, 5:16:13 AM10/4/17
to reviewboard
As this happens during the index, it is preventing us from having full search results - any help would be much appreciated as this is causing quite a few issues for our development team.
Please let me know if you need any more info.
Thanks
Rob
Christian Hammond
Oct 4, 2017, 6:05:33 AM10/4/17
to revie...@googlegroups.com
Hi Rob,
I haven't seen this before, and can't reproduce it. The error doesn't make sense, as review requests definitely have 'commit' as an attribute. Can you verify the version of haystack on the system?
Christian
--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Rob Backhurst
Oct 4, 2017, 6:40:12 AM10/4/17
to reviewboard
Hi Christian,
Thanks for getting back to me.
Is this what you're after?
python-django-haystack.noarch 2.3.1-1.el7
Thanksl
Rob
On Tuesday, 3 October 2017 09:12:26 UTC+1, Rob Backhurst wrote:
Christian Hammond
Oct 4, 2017, 7:17:44 PM10/4/17
to revie...@googlegroups.com
Thanks Rob. Let me have you run a test:
$ rb-site manage /path/to/sitedir shell
from reviewboard.reviews.models import ReviewRequest
r = ReviewRequest.objects.get(pk=1)
print hasattr(r, 'commit')
See what that says. It should be True. That's what Haystack is running to get its result.
Christian
--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Rob Backhurst
Oct 5, 2017, 3:11:48 AM10/5/17
to reviewboard
Hi Chris,
Yep, its True.
# rb-site manage /var/www/reviews shell
Python 2.7.5 (default, Aug 4 2017, 00:39:18)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from reviewboard.reviews.models import ReviewRequest
>>> r = ReviewRequest.objects.get(pk=1)
>>> print hasattr(r, 'commit')
True
>>>
Cheers
Rob
On Tuesday, 3 October 2017 09:12:26 UTC+1, Rob Backhurst wrote:
Christian Hammond
Oct 5, 2017, 3:16:38 AM10/5/17
to revie...@googlegroups.com
Strange.. Okay, I want to run something else.
$ rb-site manage /var/www/reviews shell
import reviewboard
print reviewboard.VERSION
print reviewboard.__file__
import haystack
print haystack.__version__
print haystack.__file__
That'll just help me verify that the versions and paths we're seeing are what we expect.
Christian
--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Rob Backhurst
Oct 5, 2017, 3:36:20 AM10/5/17
to reviewboard
Here you go...
>>> import reviewboard
>>> print reviewboard.VERSION
(2, 5, 16, 0, u'final', 0, True)
>>> print reviewboard.__file__
/usr/lib/python2.7/site-packages/reviewboard/__init__.pyc
>>> import haystack
>>> print haystack.__version__
(2, 1, 1, u'dev')
>>> print haystack.__file__
/usr/lib/python2.7/site-packages/haystack/__init__.pyc
On Tuesday, 3 October 2017 09:12:26 UTC+1, Rob Backhurst wrote:
Rob Backhurst
Oct 5, 2017, 8:28:23 AM10/5/17
to reviewboard
Perhaps we're able to add some kind of debugging to the indexing to provide more info?
Thanks
Rob
On Tuesday, 3 October 2017 09:12:26 UTC+1, Rob Backhurst wrote:
Christian Hammond
Oct 5, 2017, 1:39:33 PM10/5/17
to revie...@googlegroups.com
Hi Rob,
Looks like the version of Haystack being run is claiming to be 2.1.1 dev. I don't know what this version might be doing for the attribute lookup, but I'd start by trying to solve the version issue, make sure you're running a 2.3.x release.
Christian
Rob Backhurst
Oct 5, 2017, 3:27:25 PM10/5/17
to revie...@googlegroups.com
Hi Christian,
How strange! I’ll take a look...
Thanks
Rob
Sent from my iPhone
You received this message because you are subscribed to a topic in the Google Groups "reviewboard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reviewboard/52HnBiFJxp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to reviewboard...@googlegroups.com.
Rob Backhurst
Oct 5, 2017, 4:22:01 PM10/5/17
to reviewboard
Hi Christian,
We deinately only have version 2.3.1 installed.
python-django-haystack 2.3.1-1.el7
Why would it think we're using 2.1.1 dev? Can we force it to look in the correct place?
Thanks
Rob
On Tuesday, 3 October 2017 09:12:26 UTC+1, Rob Backhurst wrote:
On Tuesday, 3 October 2017 09:12:26 UTC+1, Rob Backhurst wrote:
Christian Hammond
Oct 5, 2017, 7:59:12 PM10/5/17
to revie...@googlegroups.com
Hi Rob,
That's the version listed in the source code for the version of Haystack being run. Not sure if that indicates a packaging problem or what, but it's very strange.
Could you show me that one line in fields.py?
Christian
--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Rob Backhurst
Oct 6, 2017, 3:25:17 AM10/6/17
to reviewboard
Sorry, which one line?
Christian Hammond
Oct 6, 2017, 1:32:05 PM10/6/17
to revie...@googlegroups.com
Hi Rob,
It was a couple e-mails ago, but can you actually just attach /usr/lib/python2.7/site-packages/haystack/fields.py? I'll see if that logic differs from what is in 2.3.1.
Christian
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
Stephen Gallagher
Oct 6, 2017, 2:11:10 PM10/6/17
to revie...@googlegroups.com
Rob, did you install ReviewBoard using pip or using the EPEL 7 RPM?
From above, it looks like you may have two copies of Haystack on your system, one installed via RPM and the other possibly installed by pip/easy_install. You will need to clear out the 2.1.1dev version.
From above, it looks like you may have two copies of Haystack on your system, one installed via RPM and the other possibly installed by pip/easy_install. You will need to clear out the 2.1.1dev version.
Rob Backhurst
Oct 6, 2017, 2:14:02 PM10/6/17
to revie...@googlegroups.com
Hi Christian, I’ll send the file to you asap.
Thanks
Rob
Sent from my iPhone
You received this message because you are subscribed to a topic in the Google Groups "reviewboard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reviewboard/52HnBiFJxp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to reviewboard...@googlegroups.com.
Rob Backhurst
Oct 6, 2017, 2:17:42 PM10/6/17
to revie...@googlegroups.com
Hi Stephen,
I did think that might be the case, but I have only installed using yum.
I have a test system setup in the same way (and showing the same behaviour) and checked to see if any packages had been installed using pip, but they have not...
I’m not not particularly savvy with pip/easy_install though so if you can suggest any tricks to show for sure, that would be great.
Thanks
Rob
Sent from my iPhone
You received this message because you are subscribed to a topic in the Google Groups "reviewboard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reviewboard/52HnBiFJxp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to reviewboard...@googlegroups.com.
Christian Hammond
Oct 6, 2017, 3:05:02 PM10/6/17
to revie...@googlegroups.com
Hi Rob,
Stephen and I dug into this a bit. Turns out the version discrepancy comes from an error in the upstream tarball's reported version. So you likely do have only one version of this installed.
I'm going to give you some additional code to place in that fields.py file, so we can get more information. On line 87, it's doing a:
Can you change that to:
raise SearchFieldError("The model '%s' ('%s' -- %s) does not have a model_attr '%s' (%s)." % (repr(obj), repr(current_object), current_object.__class__.__name__, attr, hasattr(current_object, attr)))
Then restart Apache and try again. See what information you get.
Christian
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to a topic in the Google Groups "reviewboard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reviewboard/52HnBiFJxp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to reviewboard+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
Rob Backhurst
Oct 9, 2017, 6:29:54 AM10/9/17
to reviewboard
Hi Christian,
See the new output below...
Removing all documents from your index because you said so.
All documents removed.
Indexing 558 users
Indexing 27878 review requests
ERROR:root:Error updating reviews using default
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 189, in handle_label
self.update_backend(label, using)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 234, in update_backend
do_update(backend, index, qs, start, end, total, self.verbosity)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 89, in do_update
backend.update(index, current_qs)
File "/usr/lib/python2.7/site-packages/haystack/backends/whoosh_backend.py", line 191, in update
doc = index.full_prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 207, in full_prepare
self.prepared_data = self.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 198, in prepare
self.prepared_data[field.index_fieldname] = field.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 160, in prepare
return self.convert(super(CharField, self).prepare(obj))
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 88, in prepare
raise SearchFieldError("The model '%s' ('%s' -- %s) does not have a model_attr '%s' (%s)." % (repr(obj), repr(current_object), current_object.__class__.__name__, attr, hasattr(current_object, attr)))
SearchFieldError: The model '<ReviewRequest: CUT_REVIEW_NAME>' ('<ReviewRequest: CUT_REVIEW_NAME>' -- ReviewRequest) does not have a model_attr 'commit' (True).
Traceback (most recent call last):
File "/usr/bin/rb-site", line 9, in <module>
load_entry_point('ReviewBoard==2.5.16', 'console_scripts', 'rb-site')()
File "/usr/lib/python2.7/site-packages/reviewboard/cmdline/rbsite.py", line 1964, in main
command.run()
File "/usr/lib/python2.7/site-packages/reviewboard/cmdline/rbsite.py", line 1884, in run
site.run_manage_command(args[0], args[1:])
File "/usr/lib/python2.7/site-packages/reviewboard/cmdline/rbsite.py", line 712, in run_manage_command
execute_from_command_line([__file__, cmd] + params)
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/usr/lib/python2.7/site-packages/reviewboard/reviews/management/commands/index.py", line 19, in handle
call_command('rebuild_index', interactive=False)
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 159, in call_command
return klass.execute(*args, **defaults)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/rebuild_index.py", line 16, in handle
call_command('update_index', **options)
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 159, in call_command
return klass.execute(*args, **defaults)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 184, in handle
return super(Command, self).handle(*items, **options)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 385, in handle
label_output = self.handle_label(label, **options)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 189, in handle_label
self.update_backend(label, using)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 234, in update_backend
do_update(backend, index, qs, start, end, total, self.verbosity)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 89, in do_update
backend.update(index, current_qs)
File "/usr/lib/python2.7/site-packages/haystack/backends/whoosh_backend.py", line 191, in update
doc = index.full_prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 207, in full_prepare
self.prepared_data = self.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 198, in prepare
self.prepared_data[field.index_fieldname] = field.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 160, in prepare
return self.convert(super(CharField, self).prepare(obj))
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 88, in prepare
raise SearchFieldError("The model '%s' ('%s' -- %s) does not have a model_attr '%s' (%s)." % (repr(obj), repr(current_object), current_object.__class__.__name__, attr, hasattr(current_object, attr)))
haystack.exceptions.SearchFieldError: The model '<ReviewRequest: CUT_REVIEW_NAME>' ('<ReviewRequest: CUT_REVIEW_NAME>' -- ReviewRequest) does not have a model_attr 'commit' (True).
Thanks
Rob
On Tuesday, 3 October 2017 09:12:26 UTC+1, Rob Backhurst wrote:
Christian Hammond
Oct 9, 2017, 12:22:42 PM10/9/17
to revie...@googlegroups.com
Can you send me the fields.py file as you now have it? That output contradicts the logic that should be in the code.
Christian
--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Rob Backhurst
Oct 10, 2017, 4:35:22 AM10/10/17
to reviewboard
Here you go...
# encoding: utf-8
from __future__ import unicode_literals
import re
from django.template import Context, loader
from django.utils import datetime_safe, six
from haystack.exceptions import SearchFieldError
from haystack.utils import get_model_ct_tuple
class NOT_PROVIDED:
pass
DATETIME_REGEX = re.compile('^(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})(T|\s+)(?P<hour>\d{2}):(?P<minute>\d{2}):(?P<second>\d{2}).*?$')
# All the SearchFields variants.
class SearchField(object):
"""The base implementation of a search field."""
field_type = None
def __init__(self, model_attr=None, use_template=False, template_name=None,
document=False, indexed=True, stored=True, faceted=False,
default=NOT_PROVIDED, null=False, index_fieldname=None,
facet_class=None, boost=1.0, weight=None):
# Track what the index thinks this field is called.
self.instance_name = None
self.model_attr = model_attr
self.use_template = use_template
self.template_name = template_name
self.document = document
self.indexed = indexed
self.stored = stored
self.faceted = faceted
self._default = default
self.null = null
self.index_fieldname = index_fieldname
self.boost = weight or boost
self.is_multivalued = False
# We supply the facet_class for making it easy to create a faceted
# field based off of this field.
self.facet_class = facet_class
if self.facet_class is None:
self.facet_class = FacetCharField
self.set_instance_name(None)
def set_instance_name(self, instance_name):
self.instance_name = instance_name
if self.index_fieldname is None:
self.index_fieldname = self.instance_name
def has_default(self):
"""Returns a boolean of whether this field has a default value."""
return self._default is not NOT_PROVIDED
@property
def default(self):
"""Returns the default value for the field."""
if callable(self._default):
return self._default()
return self._default
def prepare(self, obj):
"""
Takes data from the provided object and prepares it for storage in the
index.
"""
# Give priority to a template.
if self.use_template:
return self.prepare_template(obj)
elif self.model_attr is not None:
# Check for `__` in the field for looking through the relation.
attrs = self.model_attr.split('__')
current_object = obj
for attr in attrs:
if not hasattr(current_object, attr):
raise SearchFieldError("The model '%s' ('%s' -- %s) does not have a model_attr '%s' (%s)." % (repr(obj), repr(current_object), current_object.__class__.__name__, attr, hasattr(current_object, attr)))
current_object = getattr(current_object, attr, None)
if current_object is None:
if self.has_default():
current_object = self._default
# Fall out of the loop, given any further attempts at
# accesses will fail misreably.
break
elif self.null:
current_object = None
# Fall out of the loop, given any further attempts at
# accesses will fail misreably.
break
else:
raise SearchFieldError("The model '%s' has an empty model_attr '%s' and doesn't allow a default or null value." % (repr(obj), attr))
if callable(current_object):
return current_object()
return current_object
if self.has_default():
return self.default
else:
return None
def prepare_template(self, obj):
"""
Flattens an object for indexing.
This loads a template
(``search/indexes/{app_label}/{model_name}_{field_name}.txt``) and
returns the result of rendering that template. ``object`` will be in
its context.
"""
if self.instance_name is None and self.template_name is None:
raise SearchFieldError("This field requires either its instance_name variable to be populated or an explicit template_name in order to load the correct template.")
if self.template_name is not None:
template_names = self.template_name
if not isinstance(template_names, (list, tuple)):
template_names = [template_names]
else:
app_label, model_name = get_model_ct_tuple(obj)
template_names = ['search/indexes/%s/%s_%s.txt' % (app_label, model_name, self.instance_name)]
t = loader.select_template(template_names)
return t.render(Context({'object': obj}))
def convert(self, value):
"""
Handles conversion between the data found and the type of the field.
Extending classes should override this method and provide correct
data coercion.
"""
return value
class CharField(SearchField):
field_type = 'string'
def __init__(self, **kwargs):
if kwargs.get('facet_class') is None:
kwargs['facet_class'] = FacetCharField
super(CharField, self).__init__(**kwargs)
def prepare(self, obj):
return self.convert(super(CharField, self).prepare(obj))
def convert(self, value):
if value is None:
return None
return six.text_type(value)
class LocationField(SearchField):
field_type = 'location'
def prepare(self, obj):
from haystack.utils.geo import ensure_point
value = super(LocationField, self).prepare(obj)
if value is None:
return None
pnt = ensure_point(value)
pnt_lng, pnt_lat = pnt.get_coords()
return "%s,%s" % (pnt_lat, pnt_lng)
def convert(self, value):
from haystack.utils.geo import ensure_point, Point
if value is None:
return None
if hasattr(value, 'geom_type'):
value = ensure_point(value)
return value
if isinstance(value, six.string_types):
lat, lng = value.split(',')
elif isinstance(value, (list, tuple)):
# GeoJSON-alike
lat, lng = value[1], value[0]
elif isinstance(value, dict):
lat = value.get('lat', 0)
lng = value.get('lon', 0)
value = Point(float(lng), float(lat))
return value
class NgramField(CharField):
field_type = 'ngram'
def __init__(self, **kwargs):
if kwargs.get('faceted') is True:
raise SearchFieldError("%s can not be faceted." % self.__class__.__name__)
super(NgramField, self).__init__(**kwargs)
class EdgeNgramField(NgramField):
field_type = 'edge_ngram'
class IntegerField(SearchField):
field_type = 'integer'
def __init__(self, **kwargs):
if kwargs.get('facet_class') is None:
kwargs['facet_class'] = FacetIntegerField
super(IntegerField, self).__init__(**kwargs)
def prepare(self, obj):
return self.convert(super(IntegerField, self).prepare(obj))
def convert(self, value):
if value is None:
return None
return int(value)
class FloatField(SearchField):
field_type = 'float'
def __init__(self, **kwargs):
if kwargs.get('facet_class') is None:
kwargs['facet_class'] = FacetFloatField
super(FloatField, self).__init__(**kwargs)
def prepare(self, obj):
return self.convert(super(FloatField, self).prepare(obj))
def convert(self, value):
if value is None:
return None
return float(value)
class DecimalField(SearchField):
field_type = 'string'
def __init__(self, **kwargs):
if kwargs.get('facet_class') is None:
kwargs['facet_class'] = FacetDecimalField
super(DecimalField, self).__init__(**kwargs)
def prepare(self, obj):
return self.convert(super(DecimalField, self).prepare(obj))
def convert(self, value):
if value is None:
return None
return six.text_type(value)
class BooleanField(SearchField):
field_type = 'boolean'
def __init__(self, **kwargs):
if kwargs.get('facet_class') is None:
kwargs['facet_class'] = FacetBooleanField
super(BooleanField, self).__init__(**kwargs)
def prepare(self, obj):
return self.convert(super(BooleanField, self).prepare(obj))
def convert(self, value):
if value is None:
return None
return bool(value)
class DateField(SearchField):
field_type = 'date'
def __init__(self, **kwargs):
if kwargs.get('facet_class') is None:
kwargs['facet_class'] = FacetDateField
super(DateField, self).__init__(**kwargs)
def convert(self, value):
if value is None:
return None
if isinstance(value, six.string_types):
match = DATETIME_REGEX.search(value)
if match:
data = match.groupdict()
return datetime_safe.date(int(data['year']), int(data['month']), int(data['day']))
else:
raise SearchFieldError("Date provided to '%s' field doesn't appear to be a valid date string: '%s'" % (self.instance_name, value))
return value
class DateTimeField(SearchField):
field_type = 'datetime'
def __init__(self, **kwargs):
if kwargs.get('facet_class') is None:
kwargs['facet_class'] = FacetDateTimeField
super(DateTimeField, self).__init__(**kwargs)
def convert(self, value):
if value is None:
return None
if isinstance(value, six.string_types):
match = DATETIME_REGEX.search(value)
if match:
data = match.groupdict()
return datetime_safe.datetime(int(data['year']), int(data['month']), int(data['day']), int(data['hour']), int(data['minute']), int(data['second']))
else:
raise SearchFieldError("Datetime provided to '%s' field doesn't appear to be a valid datetime string: '%s'" % (self.instance_name, value))
return value
class MultiValueField(SearchField):
field_type = 'string'
def __init__(self, **kwargs):
if kwargs.get('facet_class') is None:
kwargs['facet_class'] = FacetMultiValueField
if kwargs.get('use_template') is True:
raise SearchFieldError("'%s' fields can not use templates to prepare their data." % self.__class__.__name__)
super(MultiValueField, self).__init__(**kwargs)
self.is_multivalued = True
def prepare(self, obj):
return self.convert(super(MultiValueField, self).prepare(obj))
def convert(self, value):
if value is None:
return None
return list(value)
class FacetField(SearchField):
"""
``FacetField`` is slightly different than the other fields because it can
work in conjunction with other fields as its data source.
Accepts an optional ``facet_for`` kwarg, which should be the field name
(not ``index_fieldname``) of the field it should pull data from.
"""
instance_name = None
def __init__(self, **kwargs):
handled_kwargs = self.handle_facet_parameters(kwargs)
super(FacetField, self).__init__(**handled_kwargs)
def handle_facet_parameters(self, kwargs):
if kwargs.get('faceted', False):
raise SearchFieldError("FacetField (%s) does not accept the 'faceted' argument." % self.instance_name)
if not kwargs.get('null', True):
raise SearchFieldError("FacetField (%s) does not accept False for the 'null' argument." % self.instance_name)
if not kwargs.get('indexed', True):
raise SearchFieldError("FacetField (%s) does not accept False for the 'indexed' argument." % self.instance_name)
if kwargs.get('facet_class'):
raise SearchFieldError("FacetField (%s) does not accept the 'facet_class' argument." % self.instance_name)
self.facet_for = None
self.facet_class = None
# Make sure the field is nullable.
kwargs['null'] = True
if 'facet_for' in kwargs:
self.facet_for = kwargs['facet_for']
del(kwargs['facet_for'])
return kwargs
def get_facet_for_name(self):
return self.facet_for or self.instance_name
class FacetCharField(FacetField, CharField):
pass
class FacetIntegerField(FacetField, IntegerField):
pass
class FacetFloatField(FacetField, FloatField):
pass
class FacetDecimalField(FacetField, DecimalField):
pass
class FacetBooleanField(FacetField, BooleanField):
pass
class FacetDateField(FacetField, DateField):
pass
class FacetDateTimeField(FacetField, DateTimeField):
pass
class FacetMultiValueField(FacetField, MultiValueField):
pass
Christian Hammond
Oct 10, 2017, 2:48:55 PM10/10/17
to revie...@googlegroups.com
Hmm, this is beyond bizarre. Note these lines:
if not hasattr(current_object, attr):
raise SearchFieldError("The model '%s' ('%s' -- %s) does not have a model_attr '%s' (%s)." % (repr(obj), repr(current_object), current_object.__class__.__name__, attr, hasattr(current_object, attr)))The relevant parts are the if statement and the final argument to SearchFieldError. Note how in the output, that final attribute resolves to True, but the if statement resolves to False. It's the same exact check for an attribute that is either there or not. It makes no sense for them to be different. The attribute can't be gone in one statement and there in another. So something truly bizarre appears to be happening, and I have no idea what could possibly cause that. If it wasn't so consistent, I'd blame the RAM on your system.
Just to gather more information, change the above code to;
print '%r (%s): %s' % (current_object, id(current_object), hasattr(current_object, attr))
if not hasattr(current_object, attr):
import inspect
raise SearchFieldError("The model '%s' ('%s' -- %s.%s at %s -- %s:%s) does not have a model_attr '%s' (%s)." % (repr(obj), repr(current_object), current_object.__class__.__module__, current_object.__class__.__name__, inspect.getfile(obj.__class__), id(obj), id(current_object), attr, hasattr(current_object, attr)))
We may be approaching a point in this where I won't be able to diagnose much else over e-mail, and am basically out of ideas. We'd need to either work directly on your system, set up a live session to diagnose it, or provide a series of custom builds and gather confidential information, and these require support contracts on our end. We'll see first if the above tells us anything interesting.
Christian
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
Rob Backhurst
Oct 11, 2017, 7:40:36 AM10/11/17
to reviewboard
Hi Christian,
Ok, lets hope this can tell you something useful...
ERROR:root:Error updating reviews using default
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 189, in handle_label
self.update_backend(label, using)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 234, in update_backend
do_update(backend, index, qs, start, end, total, self.verbosity)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 89, in do_update
backend.update(index, current_qs)
File "/usr/lib/python2.7/site-packages/haystack/backends/whoosh_backend.py", line 191, in update
doc = index.full_prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 207, in full_prepare
self.prepared_data = self.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 198, in prepare
self.prepared_data[field.index_fieldname] = field.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 161, in prepare
return self.convert(super(CharField, self).prepare(obj))
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 86, in prepare
print '%r (%s): %s' % (current_object, id(current_object), hasattr(current_object, attr))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 68: ordinal not in range(128)
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 161, in prepare
return self.convert(super(CharField, self).prepare(obj))
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 86, in prepare
print '%r (%s): %s' % (current_object, id(current_object), hasattr(current_object, attr))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 68: ordinal not in range(128)
Thanks
Rob
Stephen Gallagher
Oct 11, 2017, 2:21:22 PM10/11/17
to revie...@googlegroups.com
On Wed, Oct 11, 2017 at 7:40 AM 'Rob Backhurst' via reviewboard <revie...@googlegroups.com> wrote:
Hi Christian,
Ok, lets hope this can tell you something useful...
<snip>
Rob, I just packaged up Haystack 2.4.1 (and did rudimentary testing to see that ReviewBoard still works). Would you mind installing the RPM at https://bodhi.fedoraproject.org/updates/python-django-haystack-2.4.1-1.el7 , restarting httpd and seeing if that fixes things?
I'd like to see if this issue might just already be fixed in the newer release.
Rob Backhurst
Oct 12, 2017, 5:41:59 AM10/12/17
to reviewboard
Hi Stephen,
I've installed that on our test system and re-run the index - unfortunately the same problem.
Thanks
Rob
Rob Backhurst
Oct 12, 2017, 5:46:58 AM10/12/17
to reviewboard
Just to add, the haystack version shown in the reviewboard shell is looking better now though.
>>> import reviewboard
>>> print reviewboard.VERSION
(2, 5, 16, 0, u'final', 0, True)
>>> print reviewboard.__file__
/usr/lib/python2.7/site-packages/reviewboard/__init__.pyc
>>> import haystack
>>> print haystack.__version__
(2, 4, 1)
>>> print haystack.__file__
/usr/lib/python2.7/site-packages/haystack/__init__.pyc
Thanks
Rob
Stephen Gallagher
Oct 12, 2017, 2:58:13 PM10/12/17
to reviewboard
Ok, I just wanted to rule out an easy solution.
--
Rob Backhurst
Oct 13, 2017, 5:41:39 AM10/13/17
to reviewboard
No probs - where do we go from here?
Thanks
Rob
Christian Hammond
Oct 16, 2017, 1:31:25 AM10/16/17
to revie...@googlegroups.com
Hi Rob,
At this point, "Gremlins" appear to be the best answer from what I'm able to see right now. The output in that error is directly contradicting the conditional that allows the error to be shown, so something really bizarre is happening that absolutely should not be able to happen. It's not normal.
There's one last thing I want to see... I don't think it's going to give me any answers directly, but let's see where this goes.
Change the code to:
import inspect
has_attr = hasattr(current_object, attr)
print '%r (%s): %s' % (current_object.__class__, id(current_object), has_attr)
if not has_attr:
raise SearchFieldError("The model '%s' ('%s' -- %s.%s at %s -- %s:%s) does not have a model_attr '%s' (%s -- %s)." % (obj.__class__.__name__, current_object.pk, current_object.__class__.__module__, current_object.__class__.__name__, inspect.getfile(obj.__class__), id(obj), id(current_object), attr, hasattr(current_object, attr), has_attr))
Christian
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
Rob Backhurst
Oct 16, 2017, 5:09:05 AM10/16/17
to reviewboard
Hi Chris,
Certainly seems a bit weird - I appreciate all your efforts so far!.
See the out put from the latest index below. If this doesn't tell you anything useful, perhaps we can try (on our test system) removing the review/record that seems to be causing this and re-run the index?
ERROR:root:Error updating reviews using default
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 188, in handle_label
self.update_backend(label, using)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 233, in update_backend
do_update(backend, index, qs, start, end, total, verbosity=self.verbosity, commit=self.commit)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 96, in do_update
backend.update(index, current_qs, commit=commit)
File "/usr/lib/python2.7/site-packages/haystack/backends/whoosh_backend.py", line 196, in update
doc = index.full_prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 212, in full_prepare
self.prepared_data = self.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 203, in prepare
self.prepared_data[field.index_fieldname] = field.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 164, in prepare
return self.convert(super(CharField, self).prepare(obj))
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 92, in prepare
raise SearchFieldError("The model '%s' ('%s' -- %s.%s at %s -- %s:%s) does not have a model_attr '%s' (%s -- %s)." % (obj.__class__.__name__, current_object.pk, current_object.__class__.__module__, current_object.__class__.__name__, inspect.getfile(obj.__class__), id(obj), id(current_object), attr, hasattr(current_object, attr), has_attr))
SearchFieldError: The model 'ReviewRequest' ('27393' -- reviewboard.reviews.models.review_request.ReviewRequest at /usr/lib/python2.7/site-packages/reviewboard/reviews/models/review_request.pyc -- 231638480:231638480) does not have a model_attr 'commit' (True -- False).
Traceback (most recent call last):
File "/usr/bin/rb-site", line 9, in <module>
load_entry_point('ReviewBoard==2.5.16', 'console_scripts', 'rb-site')()
File "/usr/lib/python2.7/site-packages/reviewboard/cmdline/rbsite.py", line 1964, in main
command.run()
File "/usr/lib/python2.7/site-packages/reviewboard/cmdline/rbsite.py", line 1884, in run
site.run_manage_command(args[0], args[1:])
File "/usr/lib/python2.7/site-packages/reviewboard/cmdline/rbsite.py", line 712, in run_manage_command
execute_from_command_line([__file__, cmd] + params)
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/usr/lib/python2.7/site-packages/reviewboard/reviews/management/commands/index.py", line 19, in handle
call_command('rebuild_index', interactive=False)
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 159, in call_command
return klass.execute(*args, **defaults)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/rebuild_index.py", line 26, in handle
call_command('update_index', **options)
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 159, in call_command
return klass.execute(*args, **defaults)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 183, in handle
return super(Command, self).handle(*items, **options)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 385, in handle
label_output = self.handle_label(label, **options)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 188, in handle_label
self.update_backend(label, using)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 233, in update_backend
do_update(backend, index, qs, start, end, total, verbosity=self.verbosity, commit=self.commit)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 96, in do_update
backend.update(index, current_qs, commit=commit)
File "/usr/lib/python2.7/site-packages/haystack/backends/whoosh_backend.py", line 196, in update
doc = index.full_prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 212, in full_prepare
self.prepared_data = self.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 203, in prepare
self.prepared_data[field.index_fieldname] = field.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 164, in prepare
return self.convert(super(CharField, self).prepare(obj))
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 92, in prepare
raise SearchFieldError("The model '%s' ('%s' -- %s.%s at %s -- %s:%s) does not have a model_attr '%s' (%s -- %s)." % (obj.__class__.__name__, current_object.pk, current_object.__class__.__module__, current_object.__class__.__name__, inspect.getfile(obj.__class__), id(obj), id(current_object), attr, hasattr(current_object, attr), has_attr))
haystack.exceptions.SearchFieldError: The model 'ReviewRequest' ('27393' -- reviewboard.reviews.models.review_request.ReviewRequest at /usr/lib/python2.7/site-packages/reviewboard/reviews/models/review_request.pyc -- 231638480:231638480) does not have a model_attr 'commit' (True -- False).
Thaks
Rob
Christian Hammond
Oct 16, 2017, 5:38:52 AM10/16/17
to revie...@googlegroups.com
Hi Rob,
I think what's happening is that there's an exception being raised the first time this is accessed that is resulting in hasattr failing. One more check (I'm about to go to bed so I'll have to follow up in the morning).
Before the "has_attr = ..." line, add:
if attr == 'commit':
getattr(current_object, attr)
I imagine that's going to result in a new crash, which is good. Show me what that says.
Christian
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
Rob Backhurst
Oct 16, 2017, 10:46:43 AM10/16/17
to reviewboard
Hi Christian,
Here you go...
ERROR:root:Error updating reviews using default
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 188, in handle_label
self.update_backend(label, using)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 233, in update_backend
do_update(backend, index, qs, start, end, total, verbosity=self.verbosity, commit=self.commit)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 96, in do_update
backend.update(index, current_qs, commit=commit)
File "/usr/lib/python2.7/site-packages/haystack/backends/whoosh_backend.py", line 196, in update
doc = index.full_prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 212, in full_prepare
self.prepared_data = self.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 203, in prepare
self.prepared_data[field.index_fieldname] = field.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 166, in prepare
return self.convert(super(CharField, self).prepare(obj))
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 90, in prepare
getattr(current_object, attr)
File "/usr/lib/python2.7/site-packages/reviewboard/reviews/models/review_request.py", line 273, in get_commit
commit_id=six.text_type(self.changenum))
File "/usr/lib/python2.7/site-packages/django/db/models/query.py", line 493, in update
rows = query.get_compiler(self.db).execute_sql(None)
File "/usr/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 980, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File "/usr/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute
return self.cursor.execute(query, args)
File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
self.errorhandler(self, exc, value)
File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
IntegrityError: (1062, "Duplicate entry '1701871-1' for key 'reviews_reviewrequest_b8c24015'")
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 166, in prepare
return self.convert(super(CharField, self).prepare(obj))
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 90, in prepare
getattr(current_object, attr)
File "/usr/lib/python2.7/site-packages/reviewboard/reviews/models/review_request.py", line 273, in get_commit
commit_id=six.text_type(self.changenum))
File "/usr/lib/python2.7/site-packages/django/db/models/query.py", line 493, in update
rows = query.get_compiler(self.db).execute_sql(None)
File "/usr/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 980, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File "/usr/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute
return self.cursor.execute(query, args)
File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
self.errorhandler(self, exc, value)
File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
django.db.utils.IntegrityError: (1062, "Duplicate entry '1701871-1' for key 'reviews_reviewrequest_b8c24015'")
The error looks familiar - we had some issue sin the past with duplicate entries.
The index never used to fail for them though...
Thanks
Rob
Christian Hammond
Oct 16, 2017, 2:15:34 PM10/16/17
to revie...@googlegroups.com
Interesting. Okay, yeah, you'd need to remove the commit ID from one of them for now. I'll put a fix together for the next 2.5.x and schedule a release. Thanks for your patience on this!
Christian
Rob Backhurst
Oct 16, 2017, 2:48:53 PM10/16/17
to revie...@googlegroups.com
Thanks Christian, is that something we’ll need to do directly to the database? I don’t suppose you have any info on what needs to be done? Our DBA is on leave at the mo.
No probs, not always easy to get to the bottom of these things!
Thanks
Rob
Sent from my iPhone
You received this message because you are subscribed to a topic in the Google Groups "reviewboard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reviewboard/52HnBiFJxp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to reviewboard...@googlegroups.com.
Christian Hammond
Oct 16, 2017, 5:46:09 PM10/16/17
to revie...@googlegroups.com
Hi Rob,
Actually, this should function as a workaround for now. You can do this in that same Haystack file. Change the entirety of that previous code to:
hasattr(current_object, attr)
if hasattr(current_object, attr):
raise SearchFieldError("The model '%s' does not have a model_attr '%s'." % (repr(current_object), attr))
What will happen is the initial hasattr will trigger the crash that's resulting in the failure, but we're discarding the result of the initial one. A lucky (in this case) side-effect is that only the first call on a given review request will fail, and the second will succeed (internal state caching stuff). This should allow a full index to proceed.
It's a temporary fix until we get the next release out.
Christian
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
------
Christian Hammond
President/CEO of Beanbag
Makers of Review Board
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to a topic in the Google Groups "reviewboard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reviewboard/52HnBiFJxp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to reviewboard+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
Message has been deleted
Rob Backhurst
Oct 16, 2017, 9:01:07 PM10/16/17
to reviewboard
Hi Christian,
It crashes straight away with this error...
Removing all documents from your index because you said so.
All documents removed.
Indexing 558 users
ERROR:root:Error updating auth using default
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 188, in handle_label
self.update_backend(label, using)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 233, in update_backend
do_update(backend, index, qs, start, end, total, verbosity=self.verbosity, commit=self.commit)
File "/usr/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 96, in do_update
backend.update(index, current_qs, commit=commit)
File "/usr/lib/python2.7/site-packages/haystack/backends/whoosh_backend.py", line 196, in update
doc = index.full_prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 212, in full_prepare
self.prepared_data = self.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/indexes.py", line 203, in prepare
self.prepared_data[field.index_fieldname] = field.prepare(obj)
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 159, in prepare
return self.convert(super(CharField, self).prepare(obj))
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 87, in prepare
raise SearchFieldError("The model '%s' does not have a model_attr '%s'." % (repr(current_object), attr))
SearchFieldError: The model '<User: admin>' does not have a model_attr 'username'.
- show quoted text -
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 159, in prepare
return self.convert(super(CharField, self).prepare(obj))
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 87, in prepare
raise SearchFieldError("The model '%s' does not have a model_attr '%s'." % (repr(current_object), attr))
haystack.exceptions.SearchFieldError: The model '<User: admin>' does not have a model_attr 'username'.
Thanks
Rob
Happy user? Let us know! <a href="https://www.reviewboard.org/users/" r
Christian Hammond
Oct 17, 2017, 1:28:49 AM10/17/17
to revie...@googlegroups.com
Can you show me all the code within about 5 lines of your modification?
Christian
Rob Backhurst
Oct 17, 2017, 4:17:22 AM10/17/17
to reviewboard
77 # Give priority to a template.
78 if self.use_template:
79 return self.prepare_template(obj)
80 elif self.model_attr is not None:
81 # Check for `__` in the field for looking through the relation.
82 attrs = self.model_attr.split('__')
83 current_object = obj
84
85 for attr in attrs:
86 if hasattr(current_object, attr):
87 raise SearchFieldError("The model '%s' does not have a model_attr '%s'." % (repr(current_object), attr))
88
89 current_object = getattr(current_object, attr, None)
90
91 if current_object is None:
92 if self.has_default():
93 current_object = self._default
94 # Fall out of the loop, given any further attempts at
95 # accesses will fail misreably.
Thanks
Rob
Christian Hammond
Oct 18, 2017, 5:02:54 AM10/18/17
to revie...@googlegroups.com
Hi Rob,
That doesn't include the code I mentioned in my previous e-mail. Note the standalone 'hasattr' call on the line preceding the if statement. The workaround is to call that in a standalone way to prime a cache and avoid the error.
Christian
--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
Rob Backhurst
Oct 18, 2017, 6:43:58 AM10/18/17
to reviewboard
Hi Chris,
Ah yes sorry missed that...I have added it in but I get the same error though.
81 # Check for `__` in the field for looking through the relation.
82 attrs = self.model_attr.split('__')
83 current_object = obj
84
85 for attr in attrs:
86 hasattr(current_object, attr)
87
88 if hasattr(current_object, attr):
89 raise SearchFieldError("The model '%s' does not have a model_attr '%s'." % (repr(current_object), attr))
90
91 current_object = getattr(current_object, attr, None)
92
93 if current_object is None:
94 if self.has_default():
95 current_object = self._default
96 # Fall out of the loop, given any further attempts at
File "/usr/lib/python2.7/site-packages/haystack/fields.py", line 89, in prepare
raise SearchFieldError("The model '%s' does not have a model_attr '%s'." % (repr(current_object), attr))
haystack.exceptions.SearchFieldError: The model '<User: admin>' does not have a model_attr 'username'.
Thanks
Rob
Erik Johansson
Oct 18, 2017, 9:42:55 AM10/18/17
to revie...@googlegroups.com
I'm guessing the if statement should be "if not ..." (i.e. not is missing).
// Erik
--
Rob Backhurst
Oct 18, 2017, 9:49:41 AM10/18/17
to reviewboard
Yep thanks, index looks to be running now.
Hopefully it will get to the end this time - if so I'll make the same change on our live system.
Cheers
Rob
Rob Backhurst
Oct 19, 2017, 4:40:06 AM10/19/17
to reviewboard
Just to add, the index completed on our test system - thanks all for you help with this.
Christian, do you know when this fix will be in a released version?
Also, this may need a separate case so please let me know if you would rather I do that...
What is the intended behaviour of the search?
Should the Search API and/or the "quick search" include review requests marked as submitted?
If not, when was this changed and why?
Thanks
Rob
File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py&quo
Reply all
Reply to author
Forward
0 new messages