How to prevent DB to grow too quickly
Emmanuel Blot
I'm using Bitten for about one month to build (not even test) a
C-based project, with a SVN repository.
The DB has grown from 15MB up to 350+MB because of Bitten, since it
keeps (stores) the list of all successully checked out files in the DB
for every single build.
It seems that this is a bit overkill, as the list of files can be
retrieved from each SVN changeset, and in case of a successful check
out, there's no reason why the build working copy would not contain a
replica of the matching SVN server changeset.
Is it possible to tell Bitten to keep the list of checked out files in
the event of an error - only - ?
This would really help keeping the DB within manageable size ranges.
Thanks in advance,
Manu
Walter Bell
We're not doing that here (we literally have 5600 builds of a 1gig svn
repository), so maybe it's something with your recipe? Can you post a
recipe that we could take a look at?
-w
Emmanuel Blot
When doing a "svn checkout", every single file that is checked out end
up in the build log which is stored in the DB.
It's about 7000 lines a checkout. The compilation itself is about 700
lines - I guess I could tweak the eCos build system to be a bit less
verbose. Anyway there are 10 times more lines for checking out the
files as for building them.
Double these numbers with DEBUG + RELEASE builds. Triple them with
three different slaves (Windows, Linux, Mac). That gives 7000*2*3 =
42000 lines for each revision in the trunk, for checkout information
only.
This is really huge, and those lines carry a little information.
Some statistics:
grep '"bitten_build"' project.sql | wc -l gives 621 entries
grep bitten_log_message sdk.sql | wc -l gives 2421000 entries
> We're not doing that here (we literally have 5600 builds of a 1gig svn
> repository), so maybe it's something with your recipe?
The recipe looks like the following:
<build xmlns:python="http://bitten.cmlenz.net/tools/python"
xmlns:svn="http://bitten.cmlenz.net/tools/svn"
xmlns:sh="http://bitten.cmlenz.net/tools/sh">
<step id="checkout" description="Checkout source from repository">
<svn:checkout url="http://server/svn/project"
path="${path}" revision="${revision}" />
</step>
<step id="build-ecos" description="Build eCos">
<sh:exec executable="/bin/sh" file="host/bin/build.sh"
args="-c -i -d -p ecos" />
</step>
<step id="build-sdk" description="Build SDK">
<sh:exec executable="/bin/sh" file="host/bin/build.sh"
args="-c -i -d -p sdk" />
</step>
<step id="build-sdktests" description="Build SDK tests">
<sh:exec executable="/bin/sh" file="host/bin/build.sh"
args="-c -i -d -p sdktests" />
</step>
<step id="build-flasher" description="Build Flasher">
<sh:exec executable="/bin/sh" file="host/bin/build.sh"
args="-c -i -d -p flasher" />
</step>
<step id="build-bootloader" description="Build Loader">
<sh:exec executable="/bin/sh" file="host/bin/build.sh"
args="-c -i -d -p bootloader" />
</step>
</build>
The "culprit" is the very first rule: svn checkout
I did not find an option to tell the SVN module not to be that
verbose, but to only report errors.
Any piece of advice?
Thanks,
Manu
Leandro Conde
Emmanuel Blot wrote:
>> Can you say more about storing the entire svn output to the database?
>
> When doing a "svn checkout", every single file that is checked out end
> up in the build log which is stored in the DB.
I would also be interested to know any ways to change this, since the
same issue is happening (I should say, *will happen*, technically, since
still we have only a few builds configured,) to us here at work.
Just a way to silence down the output of svn checkouts, would do a big
difference, IMHO.
> /snip/
>
> Thanks,
> Manu
Best regards,
leandro
Ole Trenner
I haven't tried it yet, but since the svn recipe commands are just calls
to the svn command line client it might suffice to provide the "-q"
(quiet) argument. You could try something like this:
--- bitten/build/svntools.py (revision 546)
+++ bitten/build/svntools.py (working copy)
@@ -27,7 +27,7 @@
:param revision: the revision to check out
:param dir_: the name of a local subdirectory to check out into
"""
- args = ['checkout']
+ args = ['checkout', '-q']
if revision:
args += ['-r', revision]
if path:
The "-q" option prevents svn from printing the list of files to stdout.
Cheers, Ole.
>
>> /snip/
>>
>> Thanks,
>> Manu
>
> Best regards,
> leandro
>
> >
--
Ole Trenner
<o...@jayotee.de>
Emmanuel Blot
> same issue is happening (I should say, *will happen*, technically, since
> still we have only a few builds configured,) to us here at work.
>
> Just a way to silence down the output of svn checkouts, would do a big
> difference, IMHO.
Have a look at ticket #284 (I think), I've attached a patch that makes
the checkout report far less verbose.
Cheers,
Manu
Leandro Conde
Cheers,
leandro
Walter Bell
the delay.
-w
Emmanuel Blot
> the delay.
No problem. I'm not sure the patch was good enough though ;-)
Thanks,
Manu
rdebo...@xpandata.com
completely wrong - but looking over this post it seems to me that the
problem is that you are using svn:checkout.
> <svn:checkout url="http://server/svn/project"
> path="${path}" revision="${revision}" />
> </step>
from the subversion repository (hence resulting in 7000 output
lines)... I would suggest you try and replace the svn:checkout with an
svn:update, so that only the updates since the last build are
retrieved - i.e. the items in the changeset that is triggering the
build.
Hope that helps.
Ray
Emmanuel Blot
> from the subversion repository (hence resulting in 7000 output
> lines)... I would suggest you try and replace the svn:checkout with an
> svn:update, so that only the updates since the last build are
> retrieved - i.e. the items in the changeset that is triggering the
> build.
That would not be valid: the build should start up from a clean, fresh
copy of the files.
Thanks anyway
Manu