ISP system update broke Trac 1.0.3

[Rick Macdonald]

I started a thread a few minutes ago asking about the Trac version number shown as 1.2 when I did a fresh install of 1.4.2. I guess I should ask about the error that is causing me to do a fresh install in the first place.

We've been running Trac for maybe 9 years, currently at v1.0.3. Our ISP updated something that broke Trac, and doesn't seem to want to help us.

This is the error in the log:

[Mon Nov 09 18:40:29.152107 2020] [fcgid:warn] [pid 29341:tid 140584748545792] (104)Connection reset by peer: [client 24.64.162.143:54504] mod_fcgid: error reading data from FastCGI server
[Mon Nov 09 18:40:29.152157 2020] [core:error] [pid 29341:tid 140584748545792] [client 1.2.3.4:54504] End of script output before headers: index.fcgi

I don't have access to the system (such as apache config files). I don't know what they changed. I thought it was a change form PHP 5.6 to 7.3 that broke it, but reverting to v5.6 doesn't fix it.

It's probably time we upgraded to the latest anyway, but If there is something simple that we could try to get the old install working, that would be great.

[RjOllos]

Looks like it's an Apache configuration problem, and the breakage could be due to an Apache or mod_fcgid upgrade. If you have TRAC_ADMIN or CONFIG_ADMIN there may  some useful information in the System Information section on the /about page. Could you post a screen capture of that?

[Rick Macdonald]

Is there something I can get from the command line? I can't access the Trac web server at all. I get this 500 Error:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at webmaster to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

[Ryan Ollos]

On Mon, Nov 9, 2020 at 8:37 PM Rick Macdonald <sailo...@gmail.com> wrote:

Is there something I can get from the command line? I can't access the Trac web server at all. I get this 500 Error:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at webmaster to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

You could take a look in the environment logs/trac.log, but most likely the information we need is in the web server logs.

Is the issue that you have server access but no elevated privileges to inspect the Apache config and logs? If that's the case, and the server admin won't help you, you probably need to setup a new Trac instance and copy over the environment.

Ryan 

[Rick Macdonald]

Right. I have shell access to my user space on the server, so I can only see my own files. The trac/log file ends at 11 PM Oct 31, about when it got broken. The "mod_fcgi" error messages are from the apache log that appears as a link in my user space, but that's all it contains.

My original thought was "copy over the environment" as you say, but how is that done when I'm going from 1.03 to 1.4?

[Ryan Ollos]

On Mon, Nov 9, 2020 at 9:05 PM Rick Macdonald <sailo...@gmail.com> wrote:

Right. I have shell access to my user space on the server, so I can only see my own files. The trac/log file ends at 11 PM Oct 31, about when it got broken. The "mod_fcgi" error messages are from the apache log that appears as a link in my user space, but that's all it contains.

My original thought was "copy over the environment" as you say, but how is that done when I'm going from 1.03 to 1.4?

It's possible the issue could be fixed by a simple server restart, but doesn't sound like you have control over that.

Is there an .htaccess file in your user space that has Apache configuration?

To go from 1.0.3 to 1.4.2, you'd just need to run the trac-admin upgrade command. I recommend reading:

https://trac.edgewall.org/wiki/TracUpgrade

Ryan 

[Rick Macdonald]

Can an upgrade be done in one step, from 1.0 to 1.4 without doing 1.2 first?

I'm not sure where this came from originally. I think they had an automated initial Trac installer 9 years ago, but not now.

$ cat .htaccess
#<Files *.cgi>
#       SetHandler fastcgi-script
#</Files>

DirectoryIndex index.fcgi

# Make sure rewrites work
Options ExecCGI FollowSymLinks

<IfModule mod_rewrite.c>
        RewriteEngine On
        RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-d
        RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f
        RewriteCond $1 !^index.fcgi/(.*)
        # Keep the graphics and style sheet the way they are
        RewriteCond $1 !^htdocs(.*).css$
        RewriteCond $1 !^htdocs(.*).js$
        RewriteCond $1 !^htdocs(.*).gif$
        RewriteCond $1 !^htdocs(.*).jpg$
        RewriteCond $1 !^htdocs(.*).png$
        RewriteRule ^(.*)$ index.fcgi/$1 [L]

        RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-d
        RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f
        RewriteCond %{REQUEST_URI} attachment
        RewriteCond $1 !^index.fcgi/(.*)
        RewriteRule ^(.*)$ index.fcgi/$1 [L]
</IfModule>

[Rick Macdonald]

Looking at the broken installation closer, I found that a couple of years ago I created a script to start-tracd. I barely remember why though. I think it had to do with the ISP not having any interest in supporting our use of Trac. I can't believe this survived for almost 2 years.

nohup tracd -s --port 8000 --basic-auth="project,/home/user/example.com/trac/trac.passwd,Example Trac" /home/user/example.com/trac &

If I run this now (without the "-s") I get this error. Could this be the problem?

Traceback (most recent call last):
  File "/home/user/trac/python/lib/python2.7/site-packages/tracd", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/home/user/example.com/trac/python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2970, in <module>
    working_set = WorkingSet._build_master()
  File "/home/user/example.com/trac/python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 567, in _build_master
    ws.require(__requires__)
  File "/home/user/example.com/trac/python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 876, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/user/example.com/trac/python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 761, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: Genshi>=0.6

[RjOllos]

On Monday, November 9, 2020 at 9:40:56 PM UTC-8 sailo...@gmail.com wrote:

Can an upgrade be done in one step, from 1.0 to 1.4 without doing 1.2 first?

Yes, you can upgrade across any set of versions in one set.

I haven't worked with a .htaccess setup before. I assume there must be some mechanism to auto-reload the configuration, possibly in response to a "touch" on the file. 

[RjOllos]

On Monday, November 9, 2020 at 10:14:40 PM UTC-8 sailo...@gmail.com wrote:

Looking at the broken installation closer, I found that a couple of years ago I created a script to start-tracd. I barely remember why though. I think it had to do with the ISP not having any interest in supporting our use of Trac. I can't believe this survived for almost 2 years.

nohup tracd -s --port 8000 --basic-auth="project,/home/user/example.com/trac/trac.passwd,Example Trac" /home/user/example.com/trac &

If I run this now (without the "-s") I get this error. Could this be the problem?

Traceback (most recent call last):
  File "/home/user/trac/python/lib/python2.7/site-packages/tracd", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/home/user/example.com/trac/python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2970, in <module>
    working_set = WorkingSet._build_master()
  File "/home/user/example.com/trac/python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 567, in _build_master
    ws.require(__requires__)
  File "/home/user/example.com/trac/python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 876, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/user/example.com/trac/python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 761, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: Genshi>=0.6

I can't say how this relates to your other problems, but the issue here is that Genshi is not found. So you probably need to install Genshi into the Python install at "/home/user/trac/python".

$ /home/user/trac/python/bin/pip install Genshi

might accomplish that.

Bigger picture, you might consider moving to another hosting provider if they aren't willing to help you and yet your access is restricted on the system. You could get a virtual private server running Debian or Ubuntu on a hosting provider such as Digital Ocean. Bitnami and TurnKey Linux offer virtual appliances that would speed up the configuration:
https://trac.edgewall.org/wiki/CommercialServices#VirtualAppliances

Ryan

[Rick Macdonald]

It turns out our server is a VPS (at DreamHost), but we don't have root access.

I have had some success on my test subdomain (also at DreamHost, but regular shared hosting, not a VPS).

It turns out there is a system-wide Trac v1.2 installed, and that's what's getting run.

ls -l /usr/bin/tracd
-rwxr-xr-x 1 root root 362 Dec  3  2016 /usr/bin/tracd

$ cat /usr/bin/tracd
#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'Trac==1.2','console_scripts','tracd'
__requires__ = 'Trac==1.2'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('Trac==1.2', 'console_scripts', 'tracd')()
    )

I copied the trac directory from our production server and ran various update commands (that I'm trying to duplicate) and I successfully got my production Trac upgraded and it runs with the above v1.2. I don't know what to do on the production VPS server that has no system-wide Trac, just our v1.0.3. I'm afraid to mess with it in place.

I'm trying to find out what they updated 10 days ago that might have broken it.

[Rick Macdonald]

You suggested I try to re-install Genshi above. But when I tried using pip to re-install Genshi, I got an unresolved dynamic library:

~/trac/python/bin/pip --version
/home/user/trac/python/bin/python: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory

However, I found the missing Genshi error mentioned in some notes I wrote 6 years ago. At the time I used easy_install:

python ~/trac/python/bin/easy_install --install-dir=~/trac/python/lib/python2.7/site-packages Genshi

And this worked, so now the Trac instance runs again. I have no idea how Genshi got broken, especially since as far as I know the files are all contained within my user space.

Thank you so much for your time an effort! You really did get me looking in the right places.

So, someday we may have to upgrade from Trac 1.0.3. In your opinion, do you think it will work?  ;-) I've seen various methods:

python ~/trac/python/bin/easy_install --install-dir=~/trac/python/lib/python2.7/site-packages/ --upgrade Trac==1.x.x

or

pip install --upgrade Trac

And then dealing with any obsolete or change plugins.

[Ryan Ollos]

On Wed, Nov 11, 2020 at 7:15 PM Rick Macdonald <sailo...@gmail.com> wrote:

You suggested I try to re-install Genshi above. But when I tried using pip to re-install Genshi, I got an unresolved dynamic library:

~/trac/python/bin/pip --version
/home/user/trac/python/bin/python: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory

However, I found the missing Genshi error mentioned in some notes I wrote 6 years ago. At the time I used easy_install:

python ~/trac/python/bin/easy_install --install-dir=~/trac/python/lib/python2.7/site-packages Genshi

And this worked, so now the Trac instance runs again. I have no idea how Genshi got broken, especially since as far as I know the files are all contained within my user space.

Thank you so much for your time an effort! You really did get me looking in the right places.

So, someday we may have to upgrade from Trac 1.0.3. In your opinion, do you think it will work?  ;-) I've seen various methods:

python ~/trac/python/bin/easy_install --install-dir=~/trac/python/lib/python2.7/site-packages/ --upgrade Trac==1.x.x

or

pip install --upgrade Trac

And then dealing with any obsolete or change plugins.

pip is preferred since it will uninstall old packages, but either of the commands you showed should work.

You might consider taking a multistep approach to gain confidence and resolve plugin issues along the way.

1. Upgrade to 1.0.20. This should not break any plugins and will not require a db upgrade.

2. Upgrade to 1.2.7. This will require a db upgrade and plugins might fail if they don't support 1.2

3. Upgrade to 1.4.2. This will require a db upgrade and plugins might fail if they don't support 1.4

Which plugins are you using?

Ryan

[Rick Macdonald]

EditFilePlugin                 0.11.1.1        ./trac/plugins/EditFilePlugin-0.11.1.1-py2.7.egg
TracAccountManager     0.4.4             ./trac/plugins/TracAccountManager-0.4.4-py2.7.egg
TracAnnouncer               1.0dev-r0     ./trac/plugins/TracAnnouncer-1.0dev_r0-py2.7.egg
TracCustomFieldAdmin 0.2.8.post0   ./trac/plugins/TracCustomFieldAdmin-0.2.8.post0-py2.7.egg
TracWorkflowAdmin      0.12.0.2       ./trac/plugins/TracWorkflowAdmin-0.12.0.2-py2.7.egg
WorkflowEditorPlugin    1.2.0dev-r0  ./trac/plugins/WorkflowEditorPlugin-1.2.0dev_r0-py2.7.egg

I tried to check for newer versions, but it's not obvious. Does a tag of 1.4 for a plugin mean that it works with that version? For example, CustomFieldAdmin has this at the bottom of the trac-hacks page:

    Tags:
    0.11 0.12 1.0 1.2 1.4 babel bsd-license cbalan osimons plugin ticket

Rick

[Ryan Ollos]

On Wed, Nov 11, 2020 at 8:10 PM Rick Macdonald <sailo...@gmail.com> wrote:

EditFilePlugin                 0.11.1.1        ./trac/plugins/EditFilePlugin-0.11.1.1-py2.7.egg

I'm not familiar with this one. It doesn't seem to be on trac-hacks.org.

 

TracAccountManager     0.4.4             ./trac/plugins/TracAccountManager-0.4.4-py2.7.egg

You'll need to upgrade to at least 0.5, preferably 0.6dev.

 

TracAnnouncer               1.0dev-r0     ./trac/plugins/TracAnnouncer-1.0dev_r0-py2.7.egg

This is obsolete, and most features have been integrated into Trac. If you just want HTML emails, use:

https://trac-hacks.org/wiki/TracHtmlNotificationPlugin

 

TracCustomFieldAdmin 0.2.8.post0   ./trac/plugins/TracCustomFieldAdmin-0.2.8.post0-py2.7.egg

May need an upgrade, but should be fine.

 

TracWorkflowAdmin      0.12.0.2       ./trac/plugins/TracWorkflowAdmin-0.12.0.2-py2.7.egg

May need an upgrade, but should be fine.

 

WorkflowEditorPlugin    1.2.0dev-r0  ./trac/plugins/WorkflowEditorPlugin-1.2.0dev_r0-py2.7.egg

This isn't needed if you have TracWorkflowAdmin.

 

I tried to check for newer versions, but it's not obvious. Does a tag of 1.4 for a plugin mean that it works with that version?

Yes, exactly.

[Rick Macdonald]

Thank-you so much for all this help!

Rick

[Rick Macdonald]

Hello again Ryan,        [ resent due to smtp server config error ]

Some more information.

I ran tracd from the command line, it it also fails because it can't find the "six" package, the same as easy_install did trying to re-install Genshi. So maybe re-installing Genshi isn't even required this time and the "six" directory not being found is key to the "500 Internal server error" trying to access our Trac installation that has been running (sometimes limping) for over 10 years?

Traceback (most recent call last):

  File "/home/wsiuser/trac/python/lib/python2.7/site-packages/tracd", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/home/wsiuser/wellsight.com/trac/python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2970, in <module>
    working_set = WorkingSet._build_master()
  File "/home/wsiuser/wellsight.com/trac/python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 567, in _build_master
    ws.require(__requires__)
  File "/home/wsiuser/wellsight.com/trac/python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 876, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/wsiuser/wellsight.com/trac/python/lib/python2.7/site-packages/pkg_resources/__init__.py", line 761, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: six

Regards,
Rick

On 2023-03-16 22:05, Rick Macdonald wrote:

Hello Ryan,

Sorry to be replying to this 2-year-old thread, but I have the same error again. What worked to fix the problem last time (simply re-installing Genshi) doesn't work now.

Here is what I run and the error I get. I found a directory called "simple/six", but "pypi.python.org" redirects to "pypi.org". Could that redirection be why it can't find the package in /simple/six?

There are newer Genshi v0.7.xx, but not pre-built egg files that I could find.

$ python2.7 ~/trac/python/bin/easy_install --install-dir=~/trac/python/lib/python2.7/site-packages Genshi
Searching for Genshi
Best match: Genshi 0.7.4
Processing Genshi-0.7.4-py2.7-linux-x86_64.egg
Genshi 0.7.4 is already the active version in easy-install.pth

Using /home/xxxxxx/example.com/trac/python/lib/python2.7/site-packages/Genshi-0.7.4-py2.7-linux-x86_64.egg
Processing dependencies for Genshi
Searching for six
Reading https://pypi.python.org/simple/six/
Couldn't find index page for 'six' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
No local packages or download links found for six
error: Could not find suitable distribution for Requirement.parse('six')

Best regards,
Rick

--
You received this message because you are subscribed to a topic in the Google Groups "Trac Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/trac-users/sKX7GU0mwz4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to trac-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trac-users/d88a44ec-4cab-425e-98ed-9a5aa3ed71bfn%40googlegroups.com.