RB 5.0.4 HTTP 500 Internal Server Error

[Kerry Groeschel]

Occasionally clients to our RB 5.0.4/Ubuntu 22.04 get a HTTP 500 Internal Server Error when posting to RB.  It appears email notifications do not go out when this occurs but the posting to RB is successful.  Below is the RB log showing a traceback that seems to go back to a DNS timeout.  Anyone have any ideas on how to resolve this or troubleshoot it further?  

2023-05-16 13:28:53,171 - DEBUG -  - reviewboard.reviews.models.review_request - Calculated issue counts for review request ID 4045 across 1 review(s): Resulting counts = {'O': 1, 'R': 0, 'D': 0, 'A': 0, 'B': 0}; DB values = <QuerySet [{'comments__pk': None, 'comments__issue_opened': None, 'comments__issue_status': None, 'file_attachment_comments__pk': 10089, 'file_attachment_comments__issue_opened': True, 'file_attachment_comments__issue_status': 'O', 'general_comments__pk': None, 'general_comments__issue_opened': None, 'general_comments__issue_status': None, 'screenshot_comments__pk': None, 'screenshot_comments__issue_opened': None, 'screenshot_comments__issue_status': None}]>; Field IDs = {'comments': set(), 'file_attachment_comments': {10089}, 'general_comments': set(), 'screenshot_comments': set()}
2023-05-16 13:28:53,248 - DEBUG -  - djblets.cache.backend - Cache miss for key "<servername>:/:dmarc-record-<domain>.com"
2023-05-16 13:28:58,651 - ERROR - None - E1377138 - /api/review-requests/4045/reviews/31228/ - djblets.log.middleware - Exception thrown for user E1377138 at http://<servername>/api/review-requests/4045/reviews/31228/

The resolution lifetime expired after 5.402 seconds: Server 127.0.0.53 UDP port 53 answered The DNS operation timed out.; Server 127.0.0.53 UDP port 53 answered The DNS operation timed out.; Server 127.0.0.53 UDP port 53 answered The DNS operation timed out.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/django/views/decorators/vary.py", line 20, in inner_func
    response = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/resources/base.py", line 232, in __call__
    result = self.call_method_view(
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/webapi/base.py", line 348, in call_method_view
    return super(WebAPIResource, self).call_method_view(
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/resources/mixins/api_tokens.py", line 62, in call_method_view
    return super(ResourceAPITokenMixin, self).call_method_view(
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/resources/mixins/oauth2_tokens.py", line 99, in call_method_view
    return super(ResourceOAuth2TokenMixin, self).call_method_view(
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/resources/base.py", line 429, in call_method_view
    return view(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/resources/base.py", line 545, in put
    return self.update(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/decorators.py", line 156, in _call
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/util/decorators.py", line 47, in _call
    f = augmented_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/decorators.py", line 156, in _call
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/webapi/decorators.py", line 170, in _check
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/decorators.py", line 156, in _call
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/decorators.py", line 177, in _checklogin
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/decorators.py", line 156, in _call
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/decorators.py", line 156, in _call
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/decorators.py", line 322, in _validate
    return view_func(*args, **new_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/webapi/resources/base_review.py", line 290, in update
    return self.update_review(request, review, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/webapi/resources/review.py", line 121, in update_review
    return super(ReviewResource, self).update_review(
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/webapi/resources/base_review.py", line 389, in update_review
    review.publish(user=request.user,
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/reviews/models/review.py", line 420, in publish
    review_published.send(sender=self.__class__,
  File "/usr/local/lib/python3.10/dist-packages/django/dispatch/dispatcher.py", line 180, in send
    return [
  File "/usr/local/lib/python3.10/dist-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/notifications/email/signal_handlers.py", line 131, in send_review_published_mail
    message, sent = send_email(prepare_review_published_mail,
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/notifications/email/utils.py", line 290, in send_email
    message = email_builder(**kwargs)
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/notifications/email/message.py", line 405, in prepare_review_published_mail
    return prepare_base_review_request_mail(
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/notifications/email/message.py", line 237, in prepare_base_review_request_mail
    return EmailMessage(subject=subject.strip(),
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/notifications/email/message.py", line 55, in __init__
    super(EmailMessage, self).__init__(
  File "/usr/local/lib/python3.10/dist-packages/djblets/mail/message.py", line 234, in __init__
    not is_email_allowed_by_dmarc(parsed_from_email))):
  File "/usr/local/lib/python3.10/dist-packages/djblets/mail/dmarc.py", line 308, in is_email_allowed_by_dmarc
    dmarc_record = get_dmarc_record(hostname)
  File "/usr/local/lib/python3.10/dist-packages/djblets/mail/dmarc.py", line 255, in get_dmarc_record
    record = _fetch_dmarc_record(hostname=hostname,
  File "/usr/local/lib/python3.10/dist-packages/djblets/mail/dmarc.py", line 211, in _fetch_dmarc_record
    record_str = cache_memoize(
  File "/usr/local/lib/python3.10/dist-packages/djblets/cache/backend.py", line 965, in cache_memoize
    data = lookup_callable()
  File "/usr/local/lib/python3.10/dist-packages/djblets/mail/dmarc.py", line 213, in <lambda>
    lambda: _fetch_record(),
  File "/usr/local/lib/python3.10/dist-packages/djblets/mail/dmarc.py", line 202, in _fetch_record
    return dns.resolver.resolve('_dmarc.%s' % hostname,
  File "/usr/local/lib/python3.10/dist-packages/dns/resolver.py", line 1193, in resolve
    return get_default_resolver().resolve(qname, rdtype, rdclass, tcp, source,
  File "/usr/local/lib/python3.10/dist-packages/dns/resolver.py", line 1066, in resolve
    timeout = self._compute_timeout(start, lifetime,
  File "/usr/local/lib/python3.10/dist-packages/dns/resolver.py", line 879, in _compute_timeout
    raise LifetimeTimeout(timeout=duration, errors=errors)
dns.resolver.LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 127.0.0.53 UDP port 53 answered The DNS operation timed out.; Server 127.0.0.53 UDP port 53 answered The DNS operation timed out.; Server 127.0.0.53 UDP port 53 answered The DNS operation timed out.
2023-05-16 13:28:58,654 - ERROR - None - E1377138 - /api/review-requests/4045/reviews/31228/ - django.request - Internal Server Error: /api/review-requests/4045/reviews/31228/
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.10/dist-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/django/views/decorators/vary.py", line 20, in inner_func
    response = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/resources/base.py", line 232, in __call__
    result = self.call_method_view(
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/webapi/base.py", line 348, in call_method_view
    return super(WebAPIResource, self).call_method_view(
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/resources/mixins/api_tokens.py", line 62, in call_method_view
    return super(ResourceAPITokenMixin, self).call_method_view(
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/resources/mixins/oauth2_tokens.py", line 99, in call_method_view
    return super(ResourceOAuth2TokenMixin, self).call_method_view(
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/resources/base.py", line 429, in call_method_view
    return view(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/resources/base.py", line 545, in put
    return self.update(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/decorators.py", line 156, in _call
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/util/decorators.py", line 47, in _call
    f = augmented_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/decorators.py", line 156, in _call
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/webapi/decorators.py", line 170, in _check
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/decorators.py", line 156, in _call
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/decorators.py", line 177, in _checklogin
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/decorators.py", line 156, in _call
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/decorators.py", line 156, in _call
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/djblets/webapi/decorators.py", line 322, in _validate
    return view_func(*args, **new_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/webapi/resources/base_review.py", line 290, in update
    return self.update_review(request, review, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/webapi/resources/review.py", line 121, in update_review
    return super(ReviewResource, self).update_review(
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/webapi/resources/base_review.py", line 389, in update_review
    review.publish(user=request.user,
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/reviews/models/review.py", line 420, in publish
    review_published.send(sender=self.__class__,
  File "/usr/local/lib/python3.10/dist-packages/django/dispatch/dispatcher.py", line 180, in send
    return [
  File "/usr/local/lib/python3.10/dist-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/notifications/email/signal_handlers.py", line 131, in send_review_published_mail
    message, sent = send_email(prepare_review_published_mail,
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/notifications/email/utils.py", line 290, in send_email
    message = email_builder(**kwargs)
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/notifications/email/message.py", line 405, in prepare_review_published_mail
    return prepare_base_review_request_mail(
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/notifications/email/message.py", line 237, in prepare_base_review_request_mail
    return EmailMessage(subject=subject.strip(),
  File "/usr/local/lib/python3.10/dist-packages/reviewboard/notifications/email/message.py", line 55, in __init__
    super(EmailMessage, self).__init__(
  File "/usr/local/lib/python3.10/dist-packages/djblets/mail/message.py", line 234, in __init__
    not is_email_allowed_by_dmarc(parsed_from_email))):
  File "/usr/local/lib/python3.10/dist-packages/djblets/mail/dmarc.py", line 308, in is_email_allowed_by_dmarc
    dmarc_record = get_dmarc_record(hostname)
  File "/usr/local/lib/python3.10/dist-packages/djblets/mail/dmarc.py", line 255, in get_dmarc_record
    record = _fetch_dmarc_record(hostname=hostname,
  File "/usr/local/lib/python3.10/dist-packages/djblets/mail/dmarc.py", line 211, in _fetch_dmarc_record
    record_str = cache_memoize(
  File "/usr/local/lib/python3.10/dist-packages/djblets/cache/backend.py", line 965, in cache_memoize
    data = lookup_callable()
  File "/usr/local/lib/python3.10/dist-packages/djblets/mail/dmarc.py", line 213, in <lambda>
    lambda: _fetch_record(),
  File "/usr/local/lib/python3.10/dist-packages/djblets/mail/dmarc.py", line 202, in _fetch_record
    return dns.resolver.resolve('_dmarc.%s' % hostname,
  File "/usr/local/lib/python3.10/dist-packages/dns/resolver.py", line 1193, in resolve
    return get_default_resolver().resolve(qname, rdtype, rdclass, tcp, source,
  File "/usr/local/lib/python3.10/dist-packages/dns/resolver.py", line 1066, in resolve
    timeout = self._compute_timeout(start, lifetime,
  File "/usr/local/lib/python3.10/dist-packages/dns/resolver.py", line 879, in _compute_timeout
    raise LifetimeTimeout(timeout=duration, errors=errors)
dns.resolver.LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 127.0.0.53 UDP port 53 answered The DNS operation timed out.; Server 127.0.0.53 UDP port 53 answered The DNS operation timed out.; Server 127.0.0.53 UDP port 53 answered The DNS operation timed out.

[David Trowbridge]

Kerry,

The first thing to do is verify the network configuration on the Review Board server. It’s the server which is experiencing these timeouts.

If you don’t need DMARC for your emails, you could also change the settings for email delivery to avoid the DNS lookup here. That said, if DNS is unreliable, it’s hard to predict what other problems might arise.

David

--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/reviewboard/e9f68f8d-72c8-42da-beac-db61a15593aen%40googlegroups.com.

[Kerry Groeschel]

Thank you for the suggestion.  I set the User the user's From address to Always to stop the DMARC record check and have not seen the DNS time outs and no missing email notifications.