Arsenal Build Issues

[Nathan Rusch]

Hey everyone,

I've been trying to build burner for the better part of a day now (starting small), but I've run into quite a few issues, the latest of which has me stalled.

The first set of issues had to do with the configuration files. I noticed that quite a few of the include paths in several of the .pro files seemed to be incorrect or missing (this affected stone, stonegui, classmaker, and classes). I'm curious how these files are generated, and if they've been validated or tested recently. After some poking and prodding, I've mostly been able to get this side of things worked out, but I think it's worth mentioning.

Secondly, one of the lib dependencies is trying to use a Qt class method that wasn't added until Qt 6.2+. This had me stumped for a little while until I manually checked the method list for that particular class in the Qt docs (I think the class was QRegExp) and found that the method that was being called didn't exist in Qt 4.5.3. I mention this because it might be helpful for others going through this in the future to bring some of the documented reference versions up to date.

Finally, I'm curious if there is any support/documentation/anecdotal advice for building against already-installed versions of some of the dependencies (specifically, PyQt, and SIP). I've been hacking around in the config files and environment variables trying to get things to link up properly, and for the most part it's been working, but now I'm pretty stuck on getting "pyclasses" built. Basically, it keeps hitting a wall looking for "blur/blurqt.sip," and I'm not sure how to go about pointing that level of the build process in the right direction. So the last two questions I have are:

1) Is there a way to increase the verbosity level of the build process to more easily track down these issues?

2) Are there any flags that can be passed to the top-level build.py script to set up things like SIP include directories?

I'd really appreciate any advice or thoughts on any of these.

Thanks,

Nathan

[Nathan Rusch]

To follow up on this topic, I've gotten to the point where I've
decided it's going to be easier to build each dependency manually than
to try and coerce the build system into working.

With that in mind, I'm curious if anyone can provide a dependency
hierarchy of all the apps and libs, so i can go about building them in
the correct order.

Thanks for any help,

Nathan

[Todd Daugherty]

Do you have anything to help Nathan?

Todd

[Zsolt Czigléczki]

Sure.

I'm still on rev 102, which is 1 year old, so I'm not sure what
changed regarding hierarchy.
This is for assburner (changed to burner, I think):

Building: sipstatic /Users/spirit/src/AS/cpp/lib/sip
Running Command: python configure.py -k
Running Command: make
Building: sip /Users/spirit/src/AS/cpp/lib/sip
Running Command: python configure.py
Running Command: make
Building: pyqt /Users/spirit/src/AS/cpp/lib/PyQt4
Running Command: python configure.py
Running Command: make
Building: stone /Users/spirit/src/AS/cpp/lib/stone
Running Command: qmake CONFIG-=debug stone.pro
Running Command: make
Building: pystone /Users/spirit/src/AS/cpp/lib/stone
Running Command: python configure.py
Running Command: make
Building: stonegui /Users/spirit/src/AS/cpp/lib/stonegui
Running Command: qmake CONFIG-=debug stonegui.pro
Running Command: make
Building: classmakerini /Users/spirit/src/AS/cpp/apps/classmaker
Building: classmaker /Users/spirit/src/AS/cpp/apps/classmaker
Running Command: qmake CONFIG-=debug classmaker.pro
Running Command: make
Building: classgen /Users/spirit/src/AS/cpp/lib/classes
Running Command: cd ../../apps/classmaker && ./classmaker -s
/Users/spirit/src/AS/cpp/lib/classes/schema.xml -o
/Users/spirit/src/AS/cpp/lib/classes
Building: classes /Users/spirit/src/AS/cpp/lib/classes
Running Command: qmake CONFIG-=debug classes.pro
Running Command: make
Building: pyclasses /Users/spirit/src/AS/cpp/lib/classes
Running Command: python configure.py
Running Command: make
Building: pyassburner_1_3
/Users/spirit/src/AS/cpp/apps/assburner_1_3
Running Command: python configure.py -k
Running Command: make
Building: assburner_1_3ini
/Users/spirit/src/AS/cpp/apps/assburner_1_3
Building: assburner /Users/spirit/src/AS/cpp/apps/assburner_1_3
Running Command: qmake CONFIG-=debug assburner.pro
Running Command: make

I hope this will help.

Regards,
Zsolt

__________ Information from ESET Mail Security, version of virus signature database 6278 (20110708) __________

The message was checked by ESET Mail Security.
http://www.eset.com

[Barry Robison]

Hello again,

Right now we're building against Qt 4.6.2, the build docs probably need to be updated.

Unfortunately you do need to build and use the included sip and PyQt libraries. There are some differences in them that we have tried to get merged upstream.

1) Is there a way to increase the verbosity level of the build process to more easily track down these issues?

build.py clean release verbose install freezer burner pyabsubmit

should do it.

2) Are there any flags that can be passed to the top-level build.py script to set up things like SIP include directories?

Because you need to use the included versions, the sipconfig.py it installs "should" point to the correct place, but this may not be true depending on your distro.

What distro are you using?

[Nathan Rusch]

Thanks for the responses everyone.

Barry, I'm building on Fedora 10 at the moment at work, but we're
almost entirely migrated to 14 and will be jumping to 15 eventually,
so right now I'm just trying to be proactive (I'm one of the last TDs
still on 10). We've got another big show coming up, and it would be
great to at least have a sandbox farm running by the time we start
getting deep into that.

I did eventually figure out that the included SIP and PyQt need to be
built, but I keep running into what I can only assume are yacc errors
when trying to build SIP on FC10 (I don't get these errors on FC14,
but I've run into other issues on that machine that are preventing me
from continuing that build). I do have yacc and lex installed, but the
errors are a little obtuse for me to track down, so I'm not sure where
to go from here:
-----------------------------------------------------------------------------
...
g++ -o -pipe -O2 -w -DNDEBUG -I. -o lexer.o lexer.c
parser.o: In function `newQualifier':
parser.c:(.text+0x471): undefined reference to `yyerror'
...
-----------------------------------------------------------------------------
and then a whole bunch more of the same "undefined reference to
yyerror/yylex" errors.

If I can make it over this last hill, I think the rest of the manual
builds should be pretty straightforward (thanks Zsolt for the
dependency list).

Thanks again guys,

-Nathan

[Matthew Newell]

It's odd that g++ is being used to compile those files.  In my sources the Makefile is not generated and it specifies gcc to be used to compile C files.  Also you shouldn't need yacc/lex to compile sip(or any other software in the suite). You might try to revert that entire directory to a pristin checkout, then if it insists on reprocessing the grammer files you may need to modify the Makefile, or touch the .c files to ensure they are newer than the grammer files(.y and .l).   If the .y or .l files are newer than the .c files, make may try to regenerate the .c files, which is odd in itself as there are no such rules in the Makefile, but make has builtin rules and I don't know all the details of how they work as I have always avoided working with make directly:)

Matt

[Nathan Rusch]

Thanks Matt. I'll try poking the FC10 build some more tomorrow (with a
fresh repo clone).

In the meantime, I've resurrected the FC14 machine, but it's stuck on
the PyQt configuration step. This definitely IS a pristine hg clone,
and this error occurs whether I run the Blur build system or try to
build it manually. SIP and SIP-static have both already been built and
installed (from the arsenal suite source).

Here's the verbose output:

"/usr/local/bin/sip" -k -o -P -x VendorID -t WS_X11 -x
PyQt_NoPrintRangeBug -t Qt_4_6_1 -x Py_v3 -g -a QtCore.api -c QtCore -
b QtCore/QtCore.sbf -I /home/Nathan/devroot/hg_untracked/arsenalsuite/
cpp/lib/PyQt4/sip /home/Nathan/devroot/hg_untracked/arsenalsuite/cpp/
lib/PyQt4/sip/QtCore/QtCoremod.sip
sip: /home/Nathan/devroot/hg_untracked/arsenalsuite/cpp/lib/PyQt4/sip/
QtCore/QtCoremod.sip:33: syntax error
Error: Unable to create the C++ code.

Pretty unrevealing, and peeking at the file in question (not that I
know anything about SIP syntax), it appears to match the syntax of
some of the other modules, so I'm really not sure what's going on. One
thought was that maybe the PyQt source that ships with arsenal is out
of date (i.e. still targeted to Qt 4.5.3 and some older SIP build). Is
this any kind of a possibility? The apple seems to always be just out
of reach...

Thanks for any advice on this one.

-Nathan

[Barry Robison]

I just downloaded a FC14 ISO, I'll give this shot tomorrow.

[Matthew Newell]

One problem that is likely unrelated but will surely be an issue is that it appears to being building against Python version 3.  None of the python code has been tested against(ported to...) v3 yet, so I think you'll need to figure out a way to put python v2 in the path.  I don't have an arsenalsuite checkout handy to check what's in QtCore/QtCoremod.sip because I'm use Blur's svn repo, which is somewhat diverged from arsenal, but should be identical wrt sip/pyqt.  I'll wait until barry gives it a try before I try to dig any further.

Matt

[Nathan Rusch]

Matthew: It's actually using python2.6.6. From a cursory glance at the
SIP flags, it looks like -x is used to specify features to exclude, so
it seems that usage would make sense in the case of Python 3. I don't
actually have Python 3 on any of my machines at this point either, so
I'm not sure why it would be inferring something like that.

Barry: Thanks for taking the time to dig into it. I look forward to
hearing your results.

-Nathan

[Will Zhou]

Hi Nathan,

Me too. Here is my output:

will@ubuntu:~/arsenalsuite/cpp/lib/PyQt4$ python configure.py 
Determining the layout of your Qt installation...
Checking to see if the QtGui module should be built...
Checking to see if the QtHelp module should be built...
Checking to see if the QtMultimedia module should be built...
Checking to see if the QtNetwork module should be built...
Checking to see if the QtOpenGL module should be built...
Checking to see if the QtScript module should be built...
Checking to see if the QtScriptTools module should be built...
Checking to see if the QtSql module should be built...
Checking to see if the QtSvg module should be built...
Checking to see if the QtTest module should be built...
Checking to see if the QtWebKit module should be built...
Checking to see if the QtXml module should be built...
Checking to see if the QtXmlPatterns module should be built...
Checking to see if the phonon module should be built...
Checking to see if the QtAssistant module should be built...
Checking to see if the QtDesigner module should be built...
Checking to see if the QAxContainer module should be built...
Checking to see if the dbus support module should be built...
DBus v1 does not seem to be installed.
Qt v4.6.2 free edition is being used.
SIP 4.10.2 is being used.
The Qt header files are in /usr/include/qt4.
The shared Qt libraries are in /usr/lib.
The Qt binaries are in /usr/bin.
The Qt mkspecs directory is in /usr/share/qt4.
These PyQt modules will be built: QtCore, QtGui, QtHelp, QtMultimedia,
QtNetwork, QtOpenGL, QtScript, QtScriptTools, QtSql, QtSvg, QtTest, QtWebKit,
QtXml, QtXmlPatterns, QtAssistant, QtDesigner.
The PyQt Python package will be installed in /usr/lib/python2.6/dist-packages.
PyQt is being built with generated docstrings.
PyQt is being built with 'protected' redefined as 'public'.
The Designer plugin will be installed in /usr/lib/qt4/plugins/designer.
The PyQt .sip files will be installed in /usr/share/sip/PyQt4.
pyuic4, pyrcc4 and pylupdate4 will be installed in /usr/bin.
Generating the C++ source for the QtCore module...
sip: /home/will/arsenalsuite/cpp/lib/PyQt4/sip/QtCore/QtCoremod.sip:33: syntax error

Error: Unable to create the C++ code.

will@ubuntu:~/arsenalsuite/cpp/lib/PyQt4$

My OS is Ubuntu 10.04 x64 with gcc 4.4.3.

Thanks,
Will

2011/7/12 Nathan Rusch <natha...@gmail.com>

[Ben De Luca]

I wonder if thats the version sip that goes with that version of
PyQt4, I haven't played with arsenal for a bit, but I seem to remember
in my adventures with PyQT4 some thing similar.

You may need to find sip.so on the pythonpath and kill it.

[Barry Robison]

Ok so on FC14, after installing the following

yum install mercurial gcc gcc-c++ byacc python-devel postgresql-devel libXext-devel

SIP installs with a simple

python build.py clean release install sip

and Qt 4.6.2 configures with

./configure -no-qt3support -qt-sql-psql -ptmalloc -fast -opensource -L /usr/lib/pgsql/ -I /usr/include/pgsql/


Unfortanately it's still building so will need to wait until tomorrow to continue!

[Barry Robison]

Solution is almost there.

There is defintely something polluting SIP in the repo.

I have everything built now, just trying to narrow down precisely what is breaking it.


regards,
-barry

[Nathan Rusch]

Good to hear Barry.

I'm curious, though: are you not running into any issues building
PyQt, or are you saying the issues with the SIP source are what's
preventing PyQt from building?

-Nathan

[Nathan Rusch]

Alright, so I finally got everything up to burner to build, using a
lot of manual Makefile editing (there are still a lot of bad paths and
missing libraries getting hard-coded into the base Makefiles).

However, when I try to run it, I get this:
---------------------------------------------------------------------
Opened temp log file at /tmp/26639.log
Mon Jul 18 13:52:47 2011:26639 src/main.cpp:286 int main(int,
char**):Assburner version 1.5.11, build starting
Mon Jul 18 13:52:47 2011:26639 src/connection.cpp:48 static
Stone::Connection* Stone::Connection::create(const QString&):Unable to
create connection for dbType:
QObject::connect: Cannot connect (null)::connectionLost2() to
ConnectionWatcher::connectionLost()
Traceback (most recent call last):
File "<string>", line 9, in <module>
File "<string>", line 5, in __init__
ImportError: No module named blur.Stone
Segmentation fault (core dumped)
---------------------------------------------------------------------

Obviously there's at least one missing piece somewhere, but I'm not
sure if this is a missing build dependency, a missing config file, a
missing Manager/farm queue instance, or something else. The web of
dependencies seems to keep growing...

Any advice? Thanks,

-Nathan

[Nathan Rusch]

I should add that I've shortened the error list a bit. Now I just get
this:

Opened temp log file at /tmp/4262.log
Mon Jul 18 16:12:27 2011:4262 src/main.cpp:286 int main(int,

char**):Assburner version 1.5.11, build starting

Mon Jul 18 16:12:27 2011:4262 src/connection.cpp:48 static

Stone::Connection* Stone::Connection::create(const QString&):Unable to
create connection for dbType:
QObject::connect: Cannot connect (null)::connectionLost2() to
ConnectionWatcher::connectionLost()

Segmentation fault (core dumped)

The log file isn't really any more informative.

[Viet Nguyen]

I've seen this before either when the burner.ini can't be found/has bad info, or when the qt-psql plugin isn't built/installed.

[Nathan Rusch]

Hmm. Well Qt was definitely built with the psql plugin. As for a
burner.ini file, I have no idea where one should be or what should be
in it... but I'd love any more information anyone has about how to
actually get any of this set up and working.

Thanks,

-Nathan

[Zsolt Czigléczki]

Hello,

You can find some information on the ini file in the BuildHOWTO.

I start burner with 'burner -config ab.ini -auto-register -nogui' in
the directory where my ab.ini is.

Regards,
Zsolt

> __________ Information from ESET Mail Security, version of virus signature database 6308 (20110719) __________

>
> The message was checked by ESET Mail Security.
> http://www.eset.com
>
>
>

__________ Information from ESET Mail Security, version of virus signature database 6308 (20110719) __________

[Barry Robison]

Hi guys,

One of the new developers on our team spent some time on the SIP problems on Friday.

He committed changes which hopefully fix the whackiness in building sip and PyQt.

If you want to try a fresh build, I've just done one on FC14 and it all built fine on the first go.


cheers,
-baz

[Nathan Rusch]

Thanks for the update Barry,

After trying a fresh build from a new repo clone, using the automated
build system, things seem to be going much more smoothly. However,
there are still a couple of issues.

First, the Qt project file for classmaker needs to have -lutil added
as one of its linked libraries; if it isn't, the build process will
die there with undefined linker references to 'forkpty' and
'openpty' (referenced by libstone.so).


Secondly, the absubmit build is dying with the following output:

----------------------------------------
g++ -m64 -Wl,-O1 -Wl,-rpath,/usr/local/Qt/4.6.2/lib -o absubmit .out/
main.o -L/usr/local/Qt/4.6.2/lib -L../../lib/classes -lclasses -L../../
lib/stone -L../../lib/absubmit -labsubmit -lstone -lstonegui -Wl,-
rpath . -lQtSql -L/usr/local/pgsql/lib -L/usr/local/Qt/4.6.2/lib -lpq -
lQtXml -lQtGui -L/usr/X11R6/lib64 -lQtNetwork -lQtCore -lpthread
../../lib/classes/libclasses.so: undefined reference to
`Stone::RecordImp::key() const'
----------------------------------------

If I manually add $DESTDIR/usr/local/lib to LD_LIBRARY_PATH and do an
ldd on libclasses.so, it resolves libstone.so.1 correctly, but adding
that same path to my build script's LD_LIBRARY_PATH doesn't help. I've
also tried appending -lstone to the LIBS var in the absubmit Qt
project file in a few places, but that doesn't resolve it either.

I feel like I'm pretty close now. Just a couple more kinks to work
out.

Thanks for your patience on thi,

-Nathan

[Barry Robison]

On Tue, Jul 26, 2011 at 4:22 PM, Nathan Rusch <natha...@gmail.com> wrote:

First, the Qt project file for classmaker needs to have -lutil added
as one of its linked libraries; if it isn't, the build process will
die there with undefined linker references to 'forkpty' and
'openpty' (referenced by libstone.so).

Cool, let me see if adding that has any problems for me!
 

Secondly, the absubmit build is dying with the following output:

----------------------------------------
g++ -m64 -Wl,-O1 -Wl,-rpath,/usr/local/Qt/4.6.2/lib -o absubmit .out/
main.o -L/usr/local/Qt/4.6.2/lib -L../../lib/classes -lclasses -L../../
lib/stone -L../../lib/absubmit -labsubmit -lstone -lstonegui -Wl,-
rpath . -lQtSql -L/usr/local/pgsql/lib -L/usr/local/Qt/4.6.2/lib -lpq -
lQtXml -lQtGui -L/usr/X11R6/lib64 -lQtNetwork -lQtCore -lpthread
../../lib/classes/libclasses.so: undefined reference to
`Stone::RecordImp::key() const'
----------------------------------------

Hmm my guess is something is not getting rebuilt cleanly, I changed key() to no longer be const in a recent patch.

[Nathan Rusch]

Another (hopefully final) progress report:

Using fresh updated source code, things seem to be working much
better. I've finally gotten the whole thing to build automagically (at
least with freezer, burner, and pyabsubmit as the targets), but there
are still some quirks remaining in the build process that are probably
worth correcting.

1) The Qt project files for absubmit, freezer, and abpsmon all need to
have 'lutil' added as a library dependency as well (since they all
link against libstone).

2) I'm not sure if something has changed in the burner build files,
but it's now requiring me to add 'ldl' and 'lutil' to its dependencies
to get it to build (as a dependency of both libstone and
libpython2.6.a).

3) The configure.py for freezer needs to have this added on line 107
to prevent a link error in sipFreezer when trying to find libabsubmit:
makefile.extra_lib_dirs.append( "../../absubmit" );

4) On Fedora 14, I had to yum install ImageMagick-c++-devel (might be
worth mentioning that as a dependency)

As a side note, one thing about the build process that I think might
be worth revisiting is the fact that even though $DESTDIR gets
prepended onto all the build output paths, build steps that are
looking for sip files are still hard-coded to search /usr/local/share/
sip/blur instead of taking the $DESTDIR into account.

Anyway, looks like it's time to start testing some of these out.

Thanks for your help,

-Nathan


On Jul 26, 5:34 am, Barry Robison <barry.robi...@gmail.com> wrote: