Poudriere Timeout

[Cy Schubert]

Hi,


Has anyone seen this before?

print/texlive-texmf texlive-texmf-20140525_4 package/timeout runaway_process


/usr/bin/touch /wrkdirs/usr/ports/print/texlive-texmf/work/stage/usr/local/s
hare
/texmf-dist/doc/.keep_me
/bin/mkdir -p /wrkdirs/usr/ports/print/texlive-texmf/work/stage/usr/local/sh
are/
texmf-dist/source
/usr/bin/touch /wrkdirs/usr/ports/print/texlive-texmf/work/stage/usr/local/s
hare
/texmf-dist/source/.keep_me
====> Compressing man pages (compress-man)
===========================================================================
=======================<phase: package >============================
===> Building package for texlive-texmf-20140525_4
====>> Killing timed out build after 3600 seconds
====>> Cleaning up wrkdir
===> Cleaning for texlive-texmf-20140525_4
build of print/texlive-texmf ended at Fri Jan 16 23:49:09 PST 2015
build time: 02:41:43
!!! build failure encountered !!!

Building the port by hand, make package, took just over 19 minutes (1150
seconds). Would increasing the timeout from 3600 to something larger (like
7200 -- I know this will require hacking the code) address the issue?


--
Cheers,
Cy Schubert <Cy.Sc...@komquats.com> or <Cy.Sc...@cschubert.com>
FreeBSD UNIX: <c...@FreeBSD.org> Web: http://www.FreeBSD.org

The need of the many outweighs the greed of the few.


_______________________________________________
freebs...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-port...@freebsd.org"

[Matthias Apitz]

El día Sunday, January 18, 2015 a las 05:45:19PM -0800, Cy Schubert escribió:

> ====> Compressing man pages (compress-man)
> ===========================================================================
> =======================<phase: package >============================
> ===> Building package for texlive-texmf-20140525_4
> ====>> Killing timed out build after 3600 seconds
> ====>> Cleaning up wrkdir
> ===> Cleaning for texlive-texmf-20140525_4
> build of print/texlive-texmf ended at Fri Jan 16 23:49:09 PST 2015
> build time: 02:41:43
> !!! build failure encountered !!!
>
> Building the port by hand, make package, took just over 19 minutes (1150
> seconds). Would increasing the timeout from 3600 to something larger (like
> 7200 -- I know this will require hacking the code) address the issue?

I don't think that this would help. Something is waiting indefenitely
for some resource or even input from terminal.

matthias

--
Matthias Apitz, gu...@unixarea.de, http://www.unixarea.de/ +49-170-4527211
1989-2014: The Wall was torn down so that we go to war together again.
El Muro ha sido derribado para que nos unimos en ir a la guerra otra vez.
Diese Grenze wurde aufgehoben damit wir gemeinsam wieder in den Krieg ziehen.

[Matthew Seaman]

On 2015/01/19 01:45, Cy Schubert wrote:
> Hi,
>
>
> Has anyone seen this before?
>
> print/texlive-texmf texlive-texmf-20140525_4 package/timeout runaway_process
>
>
> /usr/bin/touch /wrkdirs/usr/ports/print/texlive-texmf/work/stage/usr/local/s
> hare
> /texmf-dist/doc/.keep_me
> /bin/mkdir -p /wrkdirs/usr/ports/print/texlive-texmf/work/stage/usr/local/sh
> are/
> texmf-dist/source
> /usr/bin/touch /wrkdirs/usr/ports/print/texlive-texmf/work/stage/usr/local/s
> hare
> /texmf-dist/source/.keep_me

> ====> Compressing man pages (compress-man)
> ===========================================================================
> =======================<phase: package >============================
> ===> Building package for texlive-texmf-20140525_4
> ====>> Killing timed out build after 3600 seconds
> ====>> Cleaning up wrkdir
> ===> Cleaning for texlive-texmf-20140525_4
> build of print/texlive-texmf ended at Fri Jan 16 23:49:09 PST 2015
> build time: 02:41:43
> !!! build failure encountered !!!
>
> Building the port by hand, make package, took just over 19 minutes (1150
> seconds). Would increasing the timeout from 3600 to something larger (like
> 7200 -- I know this will require hacking the code) address the issue?

Yeah. I've been seeing that exact problem. Seems that 'pkg create' is
taking an inordinately long time. truss shows that it is still
processing files, but very slowly. Not sure why it hits this port
specifically -- possibly just the number of files included in the
package. Printing something occasionally while processing the files in
the package would probably allow the process to complete instead of
poudriere killing it because it had apparently hung up, but doesn't
address the underlying reason for why it is so slow in the first place.

Cheers,

Matthew

[Torsten Zuehlsdorff]

Hi,

> Has anyone seen this before?
>
> print/texlive-texmf texlive-texmf-20140525_4 package/timeout runaway_process

Yes, i have. I've solved this problem by moving the build-jails of
poudriere to an memory disk. This make poudriere no longer io-bund and
incredibly fast. And solve this issue ;)

Greetings,
Torsten

[Torsten Zuehlsdorff]

Hi,

>>> Has anyone seen this before?
>
>>> print/texlive-texmf texlive-texmf-20140525_4 package/timeout runaway_process
>>
>> Yes, i have. I've solved this problem by moving the build-jails of
>> poudriere to an memory disk. This make poudriere no longer io-bund and
>> incredibly fast. And solve this issue ;)
>

> How did you do this ? I want to try this myself 8-}

I've hacked poudriere to run within a jail. Therefore its slightly
modified and you must checked the paths for your configuration.

When you run "poudriere bulk" you can view with "jls" the path were it
creates the build-jails. Stop it right there.

In my case the path is:
/usr/local/jail/poudriere/poudriere/data/.m

(It is possible that you must create the path)

I do before execution:

# mdmfs -s 20gb -S -o async md1 /usr/local/jail/poudriere/poudriere/data/.m

This creates an memory disk of 20 GB and mounts it to the given path.
Please be careful that the size is not greater than you RAM + free SWAP.
Otherwise you could kill some programs.

After this, launch poudriere. Invoke it with -J and choose a small
number for testing. With 20 GB memory disk it works best for me with "-J
6". With less RAM use less jails. After building finished unmount the
path and free the ram:

# umount /usr/local/jail/poudriere/poudriere/data/.m
# mdconfig -d -u 1

Everything can be scripted easily. In my case this speeds up the
complete build for the packages of my Laptop (around 800 packages,
involving X, Firefox, Thunderbird, LibreOffice) from 20 hours to 4,5 hour.

Memory disk is a very good choice, because most of the IO - creating
jail, building dependencies, compiling, etc. - is thrown away at the
end. At least we just want the final package.

I can write a more detailed description if needed.

Greetings from Germany,

[Kurt Jaeger]

Hi!

> >> Yes, i have. I've solved this problem by moving the build-jails of
> >> poudriere to an memory disk. This make poudriere no longer io-bund and
> >> incredibly fast. And solve this issue ;)

> > How did you do this ? I want to try this myself 8-}

> I've hacked poudriere to run within a jail.

Aha, the .m mountpoint. My test host has 32 GB, so 20 GB should not be
a problem.

Testport: www/p5-Selenium-Remote-Driver on 10.1-amd64, 9.3-amd64 and 8.4-i386.

Results:

old: 00:05:43
new: 00:05:11

old: 00:01:56
new: 00:00:12

old: 00:02:11
new: 00:00:14

Nice!

--
p...@opsec.eu +49 171 3101372 5 years to go !

[olli hauer]

On 2015-01-19 20:18, Kurt Jaeger wrote:
> Hi!
>
>>>> Yes, i have. I've solved this problem by moving the build-jails of
>>>> poudriere to an memory disk. This make poudriere no longer io-bund and
>>>> incredibly fast. And solve this issue ;)
>
>>> How did you do this ? I want to try this myself 8-}
>
>> I've hacked poudriere to run within a jail.
>
> Aha, the .m mountpoint. My test host has 32 GB, so 20 GB should not be
> a problem.
>
> Testport: www/p5-Selenium-Remote-Driver on 10.1-amd64, 9.3-amd64 and 8.4-i386.
>
> Results:
>
> old: 00:05:43
> new: 00:05:11
>
> old: 00:01:56
> new: 00:00:12
>
> old: 00:02:11
> new: 00:00:14
>
> Nice!
>

Hi Kurt,

are you running PD also in a jail?

If not PD can be tuned by setting MFSSIZE *or* USE_TMPFS in poudriere.conf.

On my system I have good results with 8 concurrent builds and MFSSIZE=6G or 'USE_TMPFS=all'.
Fine tuning can be done with an additional SSD (look at `systat -iostat' during a build)

poudriere.conf:

# When building packages, a memory device can be used to speedup the build.
# Only one of MFSSIZE or USE_TMPFS is supported. TMPFS is generally faster
# and will expand to the needed amount of RAM. MFS is a bit slower, but is
# more mature and can have its memory usage capped.

# If set WRKDIRPREFIX will be mdmfs of the given size (mM or gG)
#MFSSIZE=4G

# Use tmpfs(5)
...
# all - Run the entire build in memory, including builder jails.
USE_TMPFS=all

--
olli

[Torsten Zuehlsdorff]

Hi,

>>>> Yes, i have. I've solved this problem by moving the build-jails of
>>>> poudriere to an memory disk. This make poudriere no longer io-bund and
>>>> incredibly fast. And solve this issue ;)
>
>>> How did you do this ? I want to try this myself 8-}
>
>> I've hacked poudriere to run within a jail.
>
> Aha, the .m mountpoint. My test host has 32 GB, so 20 GB should not be
> a problem.
>
> Testport: www/p5-Selenium-Remote-Driver on 10.1-amd64, 9.3-amd64 and 8.4-i386.
>
> Results:
>
> old: 00:05:43
> new: 00:05:11

I suppose the fetch has to be done?

> old: 00:01:56
> new: 00:00:12
>
> old: 00:02:11
> new: 00:00:14
>
> Nice!

You are welcome :) There are a bunch of speed-optimizations possible for
poudriere, but this will defeat its general purpose. Using a memory disk
is the easiest and fastest way for a big improvement.

Greetings,
Torsten

[Cy Schubert]

In message <54BCAF05...@FreeBSD.org>, Matthew Seaman writes:
> On 2015/01/19 01:45, Cy Schubert wrote:

> > Hi,
> >
> >
> > Has anyone seen this before?
> >
> > print/texlive-texmf texlive-texmf-20140525_4 package/timeout runaway_proces
> s
> >
> >

> > /usr/bin/touch /wrkdirs/usr/ports/print/texlive-texmf/work/stage/usr/local/
> s
> > hare
> > /texmf-dist/doc/.keep_me
> > /bin/mkdir -p /wrkdirs/usr/ports/print/texlive-texmf/work/stage/usr/local/s
> h
> > are/
> > texmf-dist/source
> > /usr/bin/touch /wrkdirs/usr/ports/print/texlive-texmf/work/stage/usr/local/

> s
> > hare
> > /texmf-dist/source/.keep_me
> > ====> Compressing man pages (compress-man)
> > ===========================================================================
> > =======================<phase: package >============================
> > ===> Building package for texlive-texmf-20140525_4
> > ====>> Killing timed out build after 3600 seconds
> > ====>> Cleaning up wrkdir
> > ===> Cleaning for texlive-texmf-20140525_4
> > build of print/texlive-texmf ended at Fri Jan 16 23:49:09 PST 2015
> > build time: 02:41:43
> > !!! build failure encountered !!!
> >
> > Building the port by hand, make package, took just over 19 minutes (1150
> > seconds). Would increasing the timeout from 3600 to something larger (like
> > 7200 -- I know this will require hacking the code) address the issue?
>
> Yeah. I've been seeing that exact problem. Seems that 'pkg create' is
> taking an inordinately long time. truss shows that it is still
> processing files, but very slowly. Not sure why it hits this port
> specifically -- possibly just the number of files included in the
> package. Printing something occasionally while processing the files in
> the package would probably allow the process to complete instead of
> poudriere killing it because it had apparently hung up, but doesn't
> address the underlying reason for why it is so slow in the first place.

Depending on the server it does. I've been building custom packages on a
couple of machines in my basement, an AMD X2 5000+ system with 8 GB and an
X2 4600+ system with 5.5 GB. The 5000+ builds amd64 packages while the
4600+ system builds my i386 packages. The problem does not exhibit itself
on the 5000+ but does on the 4600+. Increasing the package build timeout in
common.sh from 3600 to 7200 resolved the issue -- the package builds takes
approximately an hour and 20 minutes. Maybe a poudriere knob to allow the
timeouts to be tuned may address this.

--
Cheers,
Cy Schubert <Cy.Sc...@komquats.com> or <Cy.Sc...@cschubert.com>
FreeBSD UNIX: <c...@FreeBSD.org> Web: http://www.FreeBSD.org

The need of the many outweighs the greed of the few.

[Bryan Drewery]

On 1/18/2015 7:45 PM, Cy Schubert wrote:
> Hi,
>
>
> Has anyone seen this before?
>

> print/texlive-texmf texlive-texmf-20140525_4 package/timeout runaway_process
>
>
> /usr/bin/touch /wrkdirs/usr/ports/print/texlive-texmf/work/stage/usr/local/s
> hare
> /texmf-dist/doc/.keep_me
> /bin/mkdir -p /wrkdirs/usr/ports/print/texlive-texmf/work/stage/usr/local/sh
> are/
> texmf-dist/source
> /usr/bin/touch /wrkdirs/usr/ports/print/texlive-texmf/work/stage/usr/local/s

> hare
> /texmf-dist/source/.keep_me
> ====> Compressing man pages (compress-man)
> ===========================================================================
> =======================<phase: package >============================
> ===> Building package for texlive-texmf-20140525_4
> ====>> Killing timed out build after 3600 seconds
> ====>> Cleaning up wrkdir
> ===> Cleaning for texlive-texmf-20140525_4
> build of print/texlive-texmf ended at Fri Jan 16 23:49:09 PST 2015
> build time: 02:41:43
> !!! build failure encountered !!!
>
> Building the port by hand, make package, took just over 19 minutes (1150
> seconds). Would increasing the timeout from 3600 to something larger (like
> 7200 -- I know this will require hacking the code) address the issue?
>
>

I'm curious why building outside of poudriere took 19 minutes, but
inside took over 60 minutes. Did you have MFS or TMPFS enabled at the
time of 60+ minutes?

Is pkg the same version in the jail as your host? It may be a recent pkg
create regression.

--
Regards,
Bryan Drewery

[Cy Schubert]

In message <54C18193...@FreeBSD.org>, Bryan Drewery writes:
> This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
> --NvIRPgp2pVcRmjhwtM9TwAvH1900GK90a
> Content-Type: text/plain; charset=windows-1252
> Content-Transfer-Encoding: quoted-printable

>
> On 1/18/2015 7:45 PM, Cy Schubert wrote:
> > Hi,

> >=20
> >=20
> > Has anyone seen this before?=20
> >=20
> > print/texlive-texmf texlive-texmf-20140525_4 package/timeout runaway_pr=
> ocess
> >=20
> >=20
> > /usr/bin/touch /wrkdirs/usr/ports/print/texlive-texmf/work/stage/usr/lo=
> cal/s
> > hare
> > /texmf-dist/doc/.keep_me
> > /bin/mkdir -p /wrkdirs/usr/ports/print/texlive-texmf/work/stage/usr/loc=
> al/sh
> > are/
> > texmf-dist/source
> > /usr/bin/touch /wrkdirs/usr/ports/print/texlive-texmf/work/stage/usr/lo=
> cal/s
> > hare
> > /texmf-dist/source/.keep_me
> > =3D=3D=3D=3D> Compressing man pages (compress-man)
> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
> =3D
> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<p=
> hase: package >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > =3D=3D=3D> Building package for texlive-texmf-20140525_4
> > =3D=3D=3D=3D>> Killing timed out build after 3600 seconds
> > =3D=3D=3D=3D>> Cleaning up wrkdir
> > =3D=3D=3D> Cleaning for texlive-texmf-20140525_4

> > build of print/texlive-texmf ended at Fri Jan 16 23:49:09 PST 2015
> > build time: 02:41:43
> > !!! build failure encountered !!!

> >=20
> > Building the port by hand, make package, took just over 19 minutes (115=
> 0=20
> > seconds). Would increasing the timeout from 3600 to something larger (l=
> ike=20

> > 7200 -- I know this will require hacking the code) address the issue?

> >=20
> >=20

>
> I'm curious why building outside of poudriere took 19 minutes, but
> inside took over 60 minutes. Did you have MFS or TMPFS enabled at the
> time of 60+ minutes?

No. I don't have enough memory to support mfs or tmpfs except for /tmp.
Having said that, my poudriere jails don't have tmpfs /tmp while all my
servers (and laptop) do.

>
> Is pkg the same version in the jail as your host? It may be a recent pkg
> create regression.

All the software is the same. The difference is that the failed build was
on an AMD 64 X2 4600+ (dual core with 5.5 GB) whereas the successful build
was on an Intel Core I3 dual core with four threads with 6 GB. All
filesystems are ZFS.

My guess is that the speed of the processors may have had something to do
with it. I don't think the extra 500 MB ram mattered much. All my build
systems are amd64 architecture.

[Matthew Seaman]

I've been seeing the same sort of problem with texlive-texmf. I only
build a relatively small number of packages (1500) and I usually do a
complete rebuild on Saturdays, and daily updates otherwise. I also
enable package jobs everywhere as this does in general lead to faster
compilation times despite the system load thrashing around at about
60-ish. This has the result that at the weekend, the single-threaded
texlive-texmf 'pkg create' process is competing against many other
processes for CPU time, and can overshoot the 3600 second timeout, hence
resulting in the process being killed. During the week, generally there
are only a few ports needing rebuild and texlive-texmf usually ends up
as the only thing building, in which case it finishes just fine.

I've a pretty simplistic work-around to this I intend to commit to pkg
over the weekend: just add a counter to 'pkg create' so it puts out some
output while processing the package's files.

Cheers,

Matthew

--
Dr Matthew J Seaman MA, D.Phil.

PGP: http://www.infracaninophile.co.uk/pgpkey
JID: mat...@infracaninophile.co.uk

[Bryan Drewery]

My guess is that MAKE_JOBS was disabled for the build inside of
Poudriere. By default (for bulk) it will parallelizing building multiple
packages at once, and each port build gets only -j1. This is something I
would like to improve with dynamic cpusets and/or racct type things. I
just haven't gotten there yet.

testport defaults to using MAKE_JOBS though.

--
Regards,
Bryan Drewery