Multiple environment startup

[Tomasz Brzezina]

I have a problem with envirnoment startups during working on trac, I can see it in debug.file. Previously i thought, that this is XMLRPCPlugin connected problem 

https://trac-hacks.org/ticket/12977, but when I uninstall plugin, problem persists (less but still).

I can reproduce it but not everytime. If I go through the webpage sometimes I see environment startup in log. Of course this leads to performance degradation. (instead of 0.2 ms web page loads in 1300ms). This isn't connected to exact link, I can refresh link 40 times without startup, but after few seconds of doing nothing next refresh produces envirnonment startup.

rjollos said, that environment startup should be done only after apache restart, so i've got a problem ;D

My trac installation is debian style - apt-get install trac  - and I use apache2 with mod_wsgi  with multiple tracs, first Alias is a result of finding of solution, TRAC1 is the instalation I focused. If I solve one, I think I could solve others ;D

    Alias /trac/TRAC1/chrome /var/www/html/trac1/trac/htdocs/
    WSGIDaemonProcess trac
    WSGIScriptAlias /trac /var/lib/trac/apache/trac.wsgi
    <Directory /var/lib/trac/apache>
      WSGIApplicationGroup %{GLOBAL}
      Require all granted
    </Directory>
    <Location ~ "/(trac|svn)/TRAC1/login">
      AuthUserFile /etc/apache2/users.trac1
      Require valid-user
    </Location>
    <Location ~ "/(trac|svn)/TRAC2">
      AuthUserFile /etc/apache2/users.trac2
      Require user user1 user2
    </Location>
    <Location ~ "/(trac|svn)/TRAC3">
      AuthUserFile /etc/apache2/users.trac2
      Require user user1 user3
    </Location>
    <Location /svn/>
      DAV svn
      SVNParentPath /var/lib/svn/
      AuthType Basic
      AuthName "My's SVN"
      AuthzSVNAccessFile /etc/subversion/access
    </Location>

[RjOllos]

Did you find any other exception or error messages in your logs?

We discussed PKG_RESOURCE_ZIP_MANIFEST. Please confirm that you see the message before setting environment variable, and that the message is not seen after setting the environment variable:

"Warning: Detected setuptools version 5.5.1. The environment variable 'PKG_RESOURCES_CACHE_ZIP_MANIFESTS' must be set to avoid significant performance degradation."
 
The message will be seen when issuing any trac-admin command. See:
https://trac.edgewall.org/ticket/12085#comment:5

- Ryan

[RjOllos]

The setuptools issue was originally discussed in (1).

Which version of Debian are you using? I would be surprised to see the latest Debian packaging such an old version of Trac.

You could just install the latest Trac, and the latest setuptools as well while you are at it. If you are worried about polluting your system packages directory, an option would be to install Trac in a virtual environment. I'm going to modify the install steps soon (2) to describe how to do that. There are a few tricky parts, such as modifying your Apache config and getting the subversion bindings to import. I could write that up if you are willing to pursue that as an option.

The latest Trac is 1.2, and the latest on 1.0-stable is 1.0.13. A lot of stuff has been fixed since 1.0.2, and there are many improvements in 1.2.

If you'd like to go this route, let me know which version of Python you are running, and whether python-virtualenv and python-pip are packages available on your system (and what their versions are). If those packages are available, the steps are basically:

$ virtualenv pve
$ . pve/bin/activate
$ pip install -U setuptools wheel pip virtualenv
$ pip install trac == 1.0.13 # or 1.2
# Modify Apache to point to interpreter and lib directories in pve
# Make Subversion libraries visible to "pve"

I'll provide more detail if you have interest. I would need to pull those details from some of my installations, notes and scripts.

- Ryan

 (1) https://groups.google.com/d/msg/trac-users/gX5kYTUFXM4/dDbPFEojF9MJ
 (2) https://trac.edgewall.org/ticket/12288

[Tomasz Brzezina]

# trac-admin

Warning: Detected setuptools version 5.5.1. The environment variable 'PKG_RESOURCES_CACHE_ZIP_MANIFESTS' must be set to avoid significant performance degradation.

trac-admin - Konsola administracyjna Trac 1.0.2


Użycie: trac-admin </sciezka/do/srodowiska-projektu> [polecenie [podpolecenie] [opcja ...]]


Wywołanie trac-admin bez polecenia rozpoczyna tryb interaktywny.


help     Show documentation
initenv  Create and initialize a new environment


=============================================================

I thought that I set that variable.... I put it into /etc/profile.d/trac:

export PKG_RESOURCES_CACHE_ZIP_MANIFESTS=1

if I run in console

#export PKG_RESOURCES_CACHE_ZIP_MANIFESTS=1

# trac-admin
trac-admin - Konsola administracyjna Trac 1.0.2


Użycie: trac-admin </sciezka/do/srodowiska-projektu> [polecenie [podpolecenie] [opcja ...]]


Wywołanie trac-admin bez polecenia rozpoczyna tryb interaktywny.


help     Show documentation
initenv  Create and initialize a new environment

So trac env reacts to this parameter. But how to execute it system wide, not user env?

[Ryan Ollos]

Documentation states:

If running the Apache web server, Ubuntu/Debian users should add the export statement to /etc/apache2/envvars. RedHat/CentOS/Fedora should can add the export statement to /etc/sysconfig/httpd.

https://trac.edgewall.org/wiki/TracInstall#DeployingTrac 

- Ryan

[Tomasz Brzezina]

I forgot, that debian requires .sh extension - now my trac-admin shows:

# trac-admin
trac-admin - Konsola administracyjna Trac 1.0.2


Użycie: trac-admin </sciezka/do/srodowiska-projektu> [polecenie [podpolecenie] [opcja ...]]


Wywołanie trac-admin bez polecenia rozpoczyna tryb interaktywny.


help     Show documentation
initenv  Create and initialize a new environment

But still three environment startups in three xmlrpc lookups.

[Ryan Ollos]

While setting the environment may mute the warning message in the output of trac-admin, I'm not sure that it is effective for the Apache process. I think you have to set the export statement in envvars.

I think the warning message will go to the Apache logs. To be sure, you could check the logs before and after setting the environment variable.

https://trac.edgewall.org/browser/tags/trac-1.0.2/trac/web/main.py?marks=365#L355

- Ryan

[Ryan Ollos]

See also, previous discussion: https://groups.google.com/d/msg/trac-users/TOoDwxhoztw/nuIj-flxElMJ

- Ryan 

[Tomasz Brzezina]

Unfortunately, my /etc/apache2/envvars has this export since Aug 23 2015... (I even didn't know about it). So the problem is somewhere else.

[RjOllos]

On Sunday, December 4, 2016 at 1:31:39 PM UTC-8, Tomasz Brzezina wrote:

Unfortunately, my /etc/apache2/envvars has this export since Aug 23 2015... (I even didn't know about it). So the problem is somewhere else.

I'd still suggest checking the logs to be sure the environment variable is effective.

Lacking any finding there, I think your best option is to create a virtual environment and install Trac 1.0.13.

- Ryan

 

[Tomasz Brzezina]

W dniu sobota, 3 grudnia 2016 09:56:39 UTC+1 użytkownik RjOllos napisał:

The setuptools issue was originally discussed in (1).

Which version of Debian are you using? I would be surprised to see the latest Debian packaging such an old version of Trac.

Latest stable.

Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux

 

You could just install the latest Trac, and the latest setuptools as well while you are at it. If you are worried about polluting your system packages directory, an option would be to install Trac in a virtual environment. I'm going to modify the install steps soon (2) to describe how to do that. There are a few tricky parts, such as modifying your Apache config and getting the subversion bindings to import. I could write that up if you are willing to pursue that as an option.

I'd prefer stay at debian packages. But I can install it on other LXC container - for test purposes. I didn't use pip or easy_setup -which is better?

[Tomasz Brzezina]

$ virtualenv pve
$ . pve/bin/activate
$ pip install -U setuptools wheel pip virtualenv
$ pip install trac == 1.0.13 # or 1.2

pip installed 1.2 (i don't know how to force to install 1.0.x), == doesn't work

 

# Modify Apache to point to interpreter and lib directories in pve

I totally don't know how ;D I'm newbie in virtualenv. I understand, that after login i have to . pve/bin/activate ;D

[RjOllos]

On Sunday, December 4, 2016 at 1:37:56 PM UTC-8, Tomasz Brzezina wrote:

W dniu sobota, 3 grudnia 2016 09:56:39 UTC+1 użytkownik RjOllos napisał:

The setuptools issue was originally discussed in (1).

Which version of Debian are you using? I would be surprised to see the latest Debian packaging such an old version of Trac.

Latest stable.

Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux

I pinged the package maintainer about this:

https://groups.google.com/d/msg/trac-dev/xZuuBzzLFeE/0GCRz22jBAAJ

There are hundreds of fixes and minor improvements since 1.0.2.

https://trac.edgewall.org/wiki/TracChangeLog#a1.0.xReleases

 

You could just install the latest Trac, and the latest setuptools as well while you are at it. If you are worried about polluting your system packages directory, an option would be to install Trac in a virtual environment. I'm going to modify the install steps soon (2) to describe how to do that. There are a few tricky parts, such as modifying your Apache config and getting the subversion bindings to import. I could write that up if you are willing to pursue that as an option.

I'd prefer stay at debian packages. But I can install it on other LXC container - for test purposes. I didn't use pip or easy_setup -which is better?

Better to use pip. It ships with Python 2.7.9 and later.

I have a system with Debian 8 update 6, which I think is the latest.

# cat /etc/debian_version 

8.6

# python --version

Python 2.7.9

Looks like you'll need to install python-virtualenv, if not already installed.

I'll writeup the other steps involved in installing to a virtual environment.

- Ryan

[RjOllos]

On Sunday, December 4, 2016 at 1:45:52 PM UTC-8, Tomasz Brzezina wrote:

$ virtualenv pve
$ . pve/bin/activate
$ pip install -U setuptools wheel pip virtualenv
$ pip install trac == 1.0.13 # or 1.2

pip installed 1.2 (i don't know how to force to install 1.0.x), == doesn't work

You need to omit the spaces around "==".

$virtualenv pve

New python executable in /Users/rjollos/Documents/temp/pve/bin/python2.7

Also creating executable in /Users/rjollos/Documents/temp/pve/bin/python

Installing setuptools, pip, wheel.... pve/bindone.

$. pve/bin/activate

$pip install -U pip setuptools

Collecting pip

  Using cached pip-9.0.1-py2.py3-none-any.whl

Collecting setuptools

  Downloading setuptools-30.2.0-py2.py3-none-any.whl (472kB)

    100% |████████████████████████████████| 475kB 689kB/s 

Installing collected packages: pip, setuptools

  Found existing installation: pip 8.0.2

    Uninstalling pip-8.0.2:

      Successfully uninstalled pip-8.0.2

  Found existing installation: setuptools 19.6.2

    Uninstalling setuptools-19.6.2:

      Successfully uninstalled setuptools-19.6.2

Successfully installed pip-9.0.1 setuptools-30.2.0

$pip install trac==1.0.13

Collecting trac==1.0.13

  Downloading Trac-1.0.13-py2-none-any.whl (3.7MB)

    100% |████████████████████████████████| 3.8MB 258kB/s 

Requirement already satisfied: setuptools>=0.6b1 in ./pve/lib/python2.7/site-packages (from trac==1.0.13)

Collecting Genshi>=0.6 (from trac==1.0.13)

Installing collected packages: Genshi, trac

Successfully installed Genshi-0.7 trac-1.0.13

  

# Modify Apache to point to interpreter and lib directories in pve

I totally don't know how ;D I'm newbie in virtualenv. I understand, that after login i have to . pve/bin/activate ;D

Let me get back to you on that, shortly.

- Ryan

[RjOllos]

On Sunday, December 4, 2016 at 1:51:56 PM UTC-8, RjOllos wrote:

On Sunday, December 4, 2016 at 1:37:56 PM UTC-8, Tomasz Brzezina wrote:

W dniu sobota, 3 grudnia 2016 09:56:39 UTC+1 użytkownik RjOllos napisał:

The setuptools issue was originally discussed in (1).

Which version of Debian are you using? I would be surprised to see the latest Debian packaging such an old version of Trac.

Latest stable.

Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux

I pinged the package maintainer about this:

https://groups.google.com/d/msg/trac-dev/xZuuBzzLFeE/0GCRz22jBAAJ

There are hundreds of fixes and minor improvements since 1.0.2.

https://trac.edgewall.org/wiki/TracChangeLog#a1.0.xReleases

It looks like 1.0.13 is in stretch (testing):
https://packages.debian.org/search?keywords=trac

I'm not familiar with the Debian release process. Does that mean 1.0.13 will potentially be included in an update of Debian 8, such as update 7?

I see that Trac 1.2 is in sid (unstable). I assume that means we won't see Trac 1.2 in Debian until Debian 9.

- Ryan

[RjOllos]

On Sunday, December 4, 2016 at 1:56:23 PM UTC-8, RjOllos wrote: 

# Modify Apache to point to interpreter and lib directories in pve

I totally don't know how ;D I'm newbie in virtualenv. I understand, that after login i have to . pve/bin/activate ;D

Let me get back to you on that, shortly.

- Ryan

To make the Subversion bindings visible to a virtualenv, here is what I've done on OSX (Debian will be the same except paths will obviously be different).

$ cd pve/lib/python2.7/site-packages
# Create two symbolic links
$ls -al svn libsvn
lrwxr-xr-x  1 rjollos  staff  69 Dec  1 09:41 libsvn -> /usr/local/Cellar/subversion/1.9.5/lib/python2.7/site-packages/libsvn
lrwxr-xr-x  1 rjollos  staff  66 Dec  1 09:41 svn -> /usr/local/Cellar/subversion/1.9.5/lib/python2.7/site-packages/svn
# Create svn.path with a link
$cat svn.pth
/usr/local/Cellar/subversion/1.9.5/lib/python2.7/site-packages/libsvn

In your Apache config, you need to set python-path. Here is the configuration from one of my servers:

WSGIDaemonProcess $site python-path=$site/pve/lib/python2.7/site-packages:$site/pve/bin maximum-requests=1000 processes=2 threads=25 inactivity-timeout=120

You probably don't need to worry about maximum-requests and the options that follow it.

- Ryan

[RjOllos]

I also noticed a difference between my Apache configs and the one you showed in the first post. Below "WSGIApplicationGroup %{GLOBAL}" I have "WSGIProcessGroup trac". I am not sure whether that should make a difference, however you could try:

    <Directory /var/lib/trac/apache>
      WSGIApplicationGroup %{GLOBAL}

      WSGIProcessGroup trac
      Require all granted
    </Directory>

- Ryan

[Tomasz Brzezina]

W dniu niedziela, 4 grudnia 2016 22:37:20 UTC+1 użytkownik RjOllos napisał:

On Sunday, December 4, 2016 at 1:31:39 PM UTC-8, Tomasz Brzezina wrote:

Unfortunately, my /etc/apache2/envvars has this export since Aug 23 2015... (I even didn't know about it). So the problem is somewhere else.

I'd still suggest checking the logs to be sure the environment variable is effective.

Yes, when I remove variable and restarts apache, the warning appears in the log. If set back - warning not appears.

[Tomasz Brzezina]

W dniu niedziela, 4 grudnia 2016 22:51:56 UTC+1 użytkownik RjOllos napisał:

I have a system with Debian 8 update 6, which I think is the latest.

# cat /etc/debian_version 

8.6

# python --version

Python 2.7.9

The same ;D 

[Tomasz Brzezina]

W dniu niedziela, 4 grudnia 2016 23:49:46 UTC+1 użytkownik RjOllos napisał:

I also noticed a difference between my Apache configs and the one you showed in the first post. Below "WSGIApplicationGroup %{GLOBAL}" I have "WSGIProcessGroup trac". I am not sure whether that should make a difference, however you could try:

    <Directory /var/lib/trac/apache>
      WSGIApplicationGroup %{GLOBAL}
      WSGIProcessGroup trac
      Require all granted
    </Directory>

Good Heavens!!!

Now i have to keep my chair!!! This is too fast!!! That was the problem ;D

I'm only afraid if this not breaks the separation between tracs - but if not, my problem is gone :D

[Tomasz Brzezina]

According to this http://stackoverflow.com/a/6591232/354420 it looks that this HAVE to be set if you want separation :D Many thanks, I owe you a big beer ;D

I'm not sure if WSGIApplicationGroup %{GLOBAL} is necessary - i comment it out and still works :D 

[RjOllos]

Thanks, I learned something valuable!

- Ryan

 

[Tomasz Brzezina]

Is there any way to speed up FIRST run? 

After long pause (e.g. 24 hours) wsgi is unloaded from memory and first running takes lots of time - about 40 seconds - when it runs, everything runs very fast, but wakeing up is paniful.  Any ideas?

[RjOllos]

On Tuesday, December 6, 2016 at 3:15:22 PM UTC-8, Tomasz Brzezina wrote:

Is there any way to speed up FIRST run? 

After long pause (e.g. 24 hours) wsgi is unloaded from memory and first running takes lots of time - about 40 seconds - when it runs, everything runs very fast, but wakeing up is paniful.  Any ideas?

Do you have any hints from looking at the logs for Trac and Apache where the time is being spent? I'd be interested to know the time from when the request hits the web server to when the //Environment Startup// message appears in the Trac log.

- Ryan 

[Tomasz Brzezina]

Do you have any hints from looking at the logs for Trac and Apache where the time is being spent? I'd be interested to know the time from when the request hits the web server to when the //Environment Startup// message appears in the Trac log.

Nope, I run it once a day - so every 24 hours it starts. Maybe in production time I will found more info (I'm thininking about running trac  in cron ;D

[Tomasz Brzezina]

I think, that "problem" is with logrotate - it restarts apache2, so wsgi isn't loaded - first time next day wsgi needs to start...

[RjOllos]

On Friday, December 16, 2016 at 5:36:37 AM UTC-8, Tomasz Brzezina wrote:

I think, that "problem" is with logrotate - it restarts apache2, so wsgi isn't loaded - first time next day wsgi needs to start...

Are you using an explicit "restart" command? If so, you could try "reload" or "graceful".

When I was porting the trac-hacks.org server to a new host I remember reviewing the logrotate configuration. There was a post-rotate command to restart the server. My memory is fuzzy on it, but I think I read that "copytruncate" can be used instead of restarting the server before/after rotating the logs. The commands we use now are:

        daily
        copytruncate
        missingok
        rotate 7
        compress
        delaycompress
        notifempty

This looks like the guidance I followed, best I can remember:
http://unix.stackexchange.com/a/47694

- Ryan