In search of help packaging ReviewBoard 3.0.14 for Gentoo Linux, problem when running server, probably something simple I overlooked.

[Eric Johnson]

Mostly, I've managed to get quite far with packaging ReviewBoard for Linux. Although I don't know much about the Gentoo packaging system, it has turned out to be easier than I feared.

We're currently running ReviewBoard 2.5.18, installed using the portage package manager. So we have done this packaging for 2.5.18. I tried this upgrade previously, but was stymied and gave up. This time I pushed ahead after understanding a better approach.

I've gotten *almost* all the way to a successful 3.0.14 install of ReviewBoard. I've got it up and running, but one thing is going wrong, and I haven't figured out why, just yet. When I say I've got it up and running, that means that I've been able to perform rb-site upgrade on all the instances, login works, and I can browse the reviews. I can access the administrative pages (since I'm an admin). However, when I click on an individual review, I get an error:

Traceback (most recent call last):

  File "/usr/lib64/python2.7/site-packages/django/core/handlers/base.py", line 137, in get_response
    response = response.render()
  File "/usr/lib64/python2.7/site-packages/django/template/response.py", line 105, in render
    self.content = self.rendered_content
  File "/usr/lib64/python2.7/site-packages/django/template/response.py", line 82, in rendered_content
    content = template.render(context)
  File "/usr/lib64/python2.7/site-packages/django/template/base.py", line 140, in render
    return self._render(context)
  File "/usr/lib64/python2.7/site-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)
  File "/usr/lib64/python2.7/site-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib64/python2.7/site-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib64/python2.7/site-packages/django/template/loader_tags.py", line 123, in render
    return compiled_parent._render(context)
  File "/usr/lib64/python2.7/site-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)
  File "/usr/lib64/python2.7/site-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib64/python2.7/site-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib64/python2.7/site-packages/django/template/loader_tags.py", line 123, in render
    return compiled_parent._render(context)
  File "/usr/lib64/python2.7/site-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)
  File "/usr/lib64/python2.7/site-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib64/python2.7/site-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib64/python2.7/site-packages/django/template/base.py", line 1125, in render
    return func(*resolved_args, **resolved_kwargs)
  File "/usr/lib64/python2.7/site-packages/djblets/extensions/templatetags/djblets_extensions.py", line 295, in load_extensions_css
    _render_css_bundle))
  File "/usr/lib64/python2.7/site-packages/djblets/extensions/templatetags/djblets_extensions.py", line 265, in _get_extension_bundles
    yield renderer(context, extension, bundle_name)
  File "/usr/lib64/python2.7/site-packages/djblets/extensions/templatetags/djblets_extensions.py", line 143, in _render_css_bundle
    name, 'CSS')
  File "/usr/lib64/python2.7/site-packages/djblets/extensions/templatetags/djblets_extensions.py", line 117, in _render_bundle
    return node_cls('"%s"' % extension.get_bundle_id(name)).render(context)
  File "/usr/lib64/python2.7/site-packages/pipeline/templatetags/pipeline.py", line 140, in render
    return self.render_compressed(package, package_name, 'css')
  File "/usr/lib64/python2.7/site-packages/djblets/extensions/templatetags/djblets_extensions.py", line 54, in render_compressed
    rendered = self.render_compressed_output(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/pipeline/templatetags/pipeline.py", line 82, in render_compressed_output
    return method(package, package.output_filename)
  File "/usr/lib64/python2.7/site-packages/pipeline/templatetags/pipeline.py", line 147, in render_css
    'url': mark_safe(staticfiles_storage.url(path))
  File "/usr/lib64/python2.7/site-packages/django/contrib/staticfiles/storage.py", line 139, in url
    hashed_name = self.hashed_name(clean_name).replace('\\', '/')
  File "/usr/lib64/python2.7/site-packages/django/contrib/staticfiles/storage.py", line 96, in hashed_name
    (clean_name, self))

ValueError: The file 'ext/rbintegrations.extension.RBIntegrationsExtension/css/fields.min.css' could not be found with <pipeline.storage.PipelineCachedStorage object at 0x7f8e1b6c9ad0>.

Note that this appears to be the same as:

https://groups.google.com/d/msg/reviewboard/FSzcHz-OaDg/89BXxRFDBAAJ

... but that thread in the mailing list didn't get a response.

---------------------

I'm guessing I've done something to package this up incorrectly, but I'm not quite sure where to look next. Hence, the request for help. Several context notes:

• I'm running using wsgi (v4.5.13) under Apache (v2.4.39)
• Gentoo installs from *sources*, so I'm not using the .egg or .whl distributions, but the tar.gz.
• I did have to patch here and there for the setup steps (more on that below)
• Excluding differences in *.pyc files, the results of the Gentoo package installs of ReviewBoard and rbintegrations that I built are the same as the result of performing pip install --local -U ReviewBoard

The last time I ran into a problem, I got stuck on the "setup" stage. Gentoo's infrastructure for installing Python packages involves calling the "setup.py" script. In the ReviewBoard and rbintegrations sources, the setup.py attempts to do a whole bunch of stuff that Gentoo doesn't allow in a package install step (installing node packages, in particular, as it breaks the sandbox for the package install). The solution ended up being very simple for the ReviewBoard package - in setup.py, change the "cmdclass=" line to "cmdclass_unused=", which disables all the extra steps in the install. For the rbintegrations package, I needed to go a little further, replacing the "from reviewboard.extensions.packaging import setup" with "from setuptools import setup", adding lines for "package_data" to ensure that the "setup.py install" step captures all of the relevant css, js, less, and png files, as well as installing a few images (which I copied from the results of the pip install).

Although I don't believe it is relevant, there are a definitely python packages that are different between my "portage" based install vs. the "pip" install. Specifically, some of the python packages already exist in the Gentoo packaging system, so I didn't make new versions of those:

asn1crypto-0.24.0.dist-info --> 0.22.0
certifi-2019.3.9.dist-info --> 2018.4.16
cffi-1.12.3.dist-info --> 1.11.4
python_dateutil-2.8.0.dist-info --> 2.7.2
django_evolution-0.7.8.dist-info --> 0.7.7
dnspython-1.16.0.dist-info --> 1.15.0
futures-3.2.0.dist-info --> 3.1.1
idna-2.8.dist-info --> 2.5
Markdown-2.6.11.dist-info --> 2.6.8
PyNaCl-1.3.0.dist-info --> 1.2.1
pillowfight-0.3-py2.7.egg-info --> 0.2
pyasn1-0.4.5.dist-info --> 0.4.2
pycparser-2.19-py2.7.egg-info --> 2.14
Pygments-2.4.2.dist-info --> 2.3.1
pytz-2019.1.dist-info --> 2018.9
requests-2.22.0.dist-info --> 2.21.0
urllib3-1.25.3.dist-info --> 1.24.2

As I said, the resulting installed image in /usr/lib64/python2.7/site-packages/reviewboard and /usr/lib64/python2.7/site-packages/rbintegrations are identical to the ones installed via "pip install", with the exception of the *.pyc files.

Any idea where I should look to fix this problem?

In case it is relevant:

# python --version
Python 2.7.15

[Christian Hammond]

Hi Eric,

Can you verify that rbintegrations was also packaged and that its extension is enabled?

We really don't advise installing from our tarballs, as we build the eggs and wheels with a particular setup to ensure they're packaged just right. It is of course possible to build with tarballs, but there are differences. The process of building eggs/wheels can result in changes to the required dependencies and to some of the build steps. It'd be best if the "source" could be the wheel, just extracted.

Christian

--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/reviewboard/CANu9%3DNdWnyWiQYjcZrqRG89ZrtdzRaxkvS7UV9GH0O0J7ryy_Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Christian Hammond

President/CEO of Beanbag

Makers of Review Board

[Eric Johnson]

Hi Christian,

Thanks so much for the response! As near as I can tell, ReviewBoard "just works" for the teams that use it here, so we're grateful for the quality work.

The issue appears to be solved at this point, and it looks like operator error. More details below.

On Friday, May 31, 2019 at 12:58:31 AM UTC-7, Christian Hammond wrote:

Hi Eric,

Can you verify that rbintegrations was also packaged and that its extension is enabled?

We really don't advise installing from our tarballs, as we build the eggs and wheels with a particular setup to ensure they're packaged just right. It is of course possible to build with tarballs, but there are differences. The process of building eggs/wheels can result in changes to the required dependencies and to some of the build steps. It'd be best if the "source" could be the wheel, just extracted.

Unfortunately, I don't see that installing from the "whl" files or the "egg" files is really used at all on Gentoo. It is fairly straightforward to make sure I get the dependencies right, since I had to encode the dependencies into the Gentoo package ebuild files. As for getting the files right, a recursive compare of the pip install vs. the Gentoo install shows that they end up the same with the packages I created.

The one challenge I observed with the ReviewBoard packages was that rbintegrations included a small handfule of binary files that I needed to copy over and add to the unpacked source.

As for why my changes were not working when I wrote you initially, it appears that I forgot to reset my virtual machine for testing purposes, and was instead dealing with incremental changes, and it didn't work. When I reset my VM, and applied the upgrade to the Gentoo packages on a "clean" install of the 2.5.18 version, the end-result worked just fine.

Thanks for your help. Your questions prodded me to keep looking, to make sure I could properly describe the problem, and that got me to find my mistake.

Eric.

 

Christian

To unsubscribe from this group and stop receiving emails from it, send an email to revie...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/reviewboard/CANu9%3DNdWnyWiQYjcZrqRG89ZrtdzRaxkvS7UV9GH0O0J7ryy_Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Stephen Gallagher]

On Tue, Jun 4, 2019 at 1:31 PM 'Eric Johnson' via Review Board
Community <revie...@googlegroups.com> wrote:
>
> Hi Christian,
>
> Thanks so much for the response! As near as I can tell, ReviewBoard "just works" for the teams that use it here, so we're grateful for the quality work.
>
> The issue appears to be solved at this point, and it looks like operator error. More details below.
>
> On Friday, May 31, 2019 at 12:58:31 AM UTC-7, Christian Hammond wrote:
>>
>> Hi Eric,
>>
>> Can you verify that rbintegrations was also packaged and that its extension is enabled?
>>
>>
>> We really don't advise installing from our tarballs, as we build the eggs and wheels with a particular setup to ensure they're packaged just right. It is of course possible to build with tarballs, but there are differences. The process of building eggs/wheels can result in changes to the required dependencies and to some of the build steps. It'd be best if the "source" could be the wheel, just extracted.
>
>
> Unfortunately, I don't see that installing from the "whl" files or the "egg" files is really used at all on Gentoo. It is fairly straightforward to make sure I get the dependencies right, since I had to encode the dependencies into the Gentoo package ebuild files. As for getting the files right, a recursive compare of the pip install vs. the Gentoo install shows that they end up the same with the packages I created.
>
> The one challenge I observed with the ReviewBoard packages was that rbintegrations included a small handfule of binary files that I needed to copy over and add to the unpacked source.
>
> As for why my changes were not working when I wrote you initially, it appears that I forgot to reset my virtual machine for testing purposes, and was instead dealing with incremental changes, and it didn't work. When I reset my VM, and applied the upgrade to the Gentoo packages on a "clean" install of the 2.5.18 version, the end-result worked just fine.
>
> Thanks for your help. Your questions prodded me to keep looking, to make sure I could properly describe the problem, and that got me to find my mistake.
>

For what it's worth, you might find my Fedora packaging to be useful
in guiding you here. Realistically, you probably want to do what I did
and just bundle the rbintegrations tarball with the ReviewBoard build.

The Fedora packaging is here:
https://src.fedoraproject.org/rpms/ReviewBoard/tree/3.0

[Eric Johnson]

Hi Stephen,

I almost had to go down that path. Fortunately, I was able to keep them separate. They have an oddly cyclical relationship. As near as I can tell, in the source, each depends on the existence of the other in setup.py . Fortunately, when packaging for Gentoo, I simply removed all the setup activities (making it equivalent to the whl files), which breaks the dependency cycle.

Eric.

[Christian Hammond]

Hi Eric,

Yeah, it's cyclical. rbintegrations needs the Review Board source to build (which is why Review Board does not depend on rbintegrations at `setup.py develop` time), but we want to auto-install rbintegrations when installing Review Board, which is why Review Board has a dependency on it. This is not great, but since most consume our packages from our own builds, most people don't have to worry about it. We opted to keep rbintegrations separate so that we could maintain it across Review Board releases, without forcing people to upgrade Review Board to get, say, the latest Slack fixes.

Christian

--

Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "Review Board Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/reviewboard/CANu9%3DNc-dqF66agep9dQgsqG3RH%3DYttkutKjwd6tnYPfXE_PRQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.