Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Recommend release repository software.
8 views
Skip to first unread message
Oleksandr Gavenko (aka gavenkoa)
Oct 6, 2011, 7:19:13 PM10/6/11
to
Currently we use FTP server to store build artifacts (to manage
dependencies between modules and 3rd party libraries).
Our build script contain HOST/LOGIN/PASS info for FTP read/write
access so no any additional configuration was needed to release build
artifacts to FTP.
But this cause storing authentication info for write operation under
build scripts so any developer can damage any package.
As we use basic setup and I have no any experience in building and
maintaining release repositories I look for recommendation how to get
release repository with write access management.
Note that anonymous read access MUST be easy accessibility from GNU
Make, Apache ANT project and does not be toolset specific (as Apache
Ive or Maven).
May be you recommend schema of using FTP/scp/sftp/nfs/WebDAV to
achieve this goal?
dependencies between modules and 3rd party libraries).
Our build script contain HOST/LOGIN/PASS info for FTP read/write
access so no any additional configuration was needed to release build
artifacts to FTP.
But this cause storing authentication info for write operation under
build scripts so any developer can damage any package.
As we use basic setup and I have no any experience in building and
maintaining release repositories I look for recommendation how to get
release repository with write access management.
Note that anonymous read access MUST be easy accessibility from GNU
Make, Apache ANT project and does not be toolset specific (as Apache
Ive or Maven).
May be you recommend schema of using FTP/scp/sftp/nfs/WebDAV to
achieve this goal?
Jorgen Grahn
Oct 6, 2011, 8:02:44 PM10/6/11
to
On Thu, 2011-10-06, Oleksandr Gavenko (aka gavenkoa) wrote:
> Currently we use FTP server to store build artifacts (to manage
> dependencies between modules and 3rd party libraries).
I don't understand this -- can you be more concrete? I would have
> Currently we use FTP server to store build artifacts (to manage
> dependencies between modules and 3rd party libraries).
thought doing this would make it *harder* to manage the dependencies.
> Our build script contain HOST/LOGIN/PASS info for FTP read/write
> access so no any additional configuration was needed to release build
> artifacts to FTP.
>
> But this cause storing authentication info for write operation under
> build scripts so any developer can damage any package.
>
> As we use basic setup and I have no any experience in building and
> maintaining release repositories I look for recommendation how to get
> release repository with write access management.
>
> Note that anonymous read access MUST be easy accessibility from GNU
> Make, Apache ANT project and does not be toolset specific (as Apache
> Ive or Maven).
>
> May be you recommend schema of using FTP/scp/sftp/nfs/WebDAV to
> achieve this goal?
already using it?
With ssh, learn about public key authentication and ssh-agent(1).
You can (and should!) avoid storing passwords in the build system.
If you're going the ssh route, I recommend rsync over ssh instead of
scp or sftp. Extremely flexible, and extremely fast if you have part
or all of the data already.
Other than that, I don't understand your problem well enough to comment.
/Jorgen
--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
Oleksandr Gavenko (aka gavenkoa)
Oct 7, 2011, 10:19:41 AM10/7/11
to
On 7 окт, 03:02, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
> On Thu, 2011-10-06, Oleksandr Gavenko (aka gavenkoa) wrote:
>
> > Currently we use FTP server to store build artifacts (to manage
> > dependencies between modules and 3rd party libraries).
>
> I don't understand this -- can you be more concrete? I would have
> thought doing this would make it *harder* to manage the dependencies.
>
Sorry for ambiguity...
> On Thu, 2011-10-06, Oleksandr Gavenko (aka gavenkoa) wrote:
>
> > Currently we use FTP server to store build artifacts (to manage
> > dependencies between modules and 3rd party libraries).
>
> I don't understand this -- can you be more concrete? I would have
> thought doing this would make it *harder* to manage the dependencies.
>
The idea behind of using FTP as build artefact repository is for
splitting project to subprojects
for better module reuse and maintaining more clean interfaces.
Also this prevent from storing 3rd parties modules under Version
Control System (it is not
not good practice to store binary modules under and we omit storing
same binary under
different projects).
> > Our build script contain HOST/LOGIN/PASS info for FTP read/write
> > access so no any additional configuration was needed to release build
> > artifacts to FTP.
>
> > But this cause storing authentication info for write operation under
> > build scripts so any developer can damage any package.
>
> > As we use basic setup and I have no any experience in building and
> > maintaining release repositories I look for recommendation how to get
> > release repository with write access management.
>
> > Note that anonymous read access MUST be easy accessibility from GNU
> > Make, Apache ANT project and does not be toolset specific (as Apache
> > Ive or Maven).
>
> > May be you recommend schema of using FTP/scp/sftp/nfs/WebDAV to
> > achieve this goal?
>
> nfs == networked file system in general. Is there a reason you're not
> already using it?
>
> With ssh, learn about public key authentication and ssh-agent(1).
> You can (and should!) avoid storing passwords in the build system.
>
$PROJ for this project
and add release developers to this group?
So release developer can do *scp* or *sftp* and point FTP/HTTP server
to /srv/projs/ for anonymous read for regular developers?
> If you're going the ssh route, I recommend rsync over ssh instead of
> scp or sftp. Extremely flexible, and extremely fast if you have part
> or all of the data already.
>
$ rsync -rv --progress -e ssh $HOST:/path/to/files lib
> Other than that, I don't understand your problem well enough to comment.
>
separate mails.
Jorgen Grahn
Oct 7, 2011, 11:57:06 AM10/7/11
to
On Fri, 2011-10-07, Oleksandr Gavenko (aka gavenkoa) wrote:
and where building subsystem FOO caused the results (a binary, or a
library plus C header files) to be stored in some central location.
It was hell. We never got the dependency graph complete, so if you
made a change in one subsystem, you never knew which others had to be
rebuilt, and in which order. The job which make or cmake or whatever
would have done automatically, ended up as manual detective work. And
we could never do correct incremental builds, so we could only build
once a day, rebuilding the same things over and over again.
Frequently, that build failed because someone updated FOO without
realizing that BAR needed to be adjusted too.
Perhaps the central storage approach works, if the interfaces between
subprojects are stable and never change, so you can treat FOO as if it
was a well-designed third-party library.
But I'd much rather have a single project with a single, non-broken
build system, and build it all myself. Google for the paper "recursive
make considered harmful" for more thoughts on this.
It may be easier if everyone just uses rsync to keep a local,
up-to-date copy on local disk.
But I spoke more generally; you can have password-less access to a
machine and an account if you have a ssh public key which either has
no passphrase (not recommended) or has been unlocked using ssh-agent
and ssh-add. There can be one account per user, or I suppose they
could all login as a "deliverables" user.
I've never set up such a thing with strict rules for who can read or
write.
>> If you're going the ssh route, I recommend rsync over ssh instead of
>> scp or sftp. Extremely flexible, and extremely fast if you have part
>> or all of the data already.
>>
> Thanks for tips:
>
> $ rsync -rv --progress -e ssh $HOST:/path/to/files lib
>
>> Other than that, I don't understand your problem well enough to comment.
>>
> Sorry for ambiguity... I will ask another more concret questions in
> separate mails.
Please don't send mail, not to me anyway. Post here instead.
> On 7 ???, 03:02, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
>> On Thu, 2011-10-06, Oleksandr Gavenko (aka gavenkoa) wrote:
>>
>> > Currently we use FTP server to store build artifacts (to manage
>> > dependencies between modules and 3rd party libraries).
>>
>> I don't understand this -- can you be more concrete? I would have
>> thought doing this would make it *harder* to manage the dependencies.
>>
> Sorry for ambiguity...
>
> The idea behind of using FTP as build artefact repository is for
> splitting project to subprojects
> for better module reuse and maintaining more clean interfaces.
I have worked with a few products which were split up into subsystems,
>> On Thu, 2011-10-06, Oleksandr Gavenko (aka gavenkoa) wrote:
>>
>> > Currently we use FTP server to store build artifacts (to manage
>> > dependencies between modules and 3rd party libraries).
>>
>> I don't understand this -- can you be more concrete? I would have
>> thought doing this would make it *harder* to manage the dependencies.
>>
> Sorry for ambiguity...
>
> The idea behind of using FTP as build artefact repository is for
> splitting project to subprojects
> for better module reuse and maintaining more clean interfaces.
and where building subsystem FOO caused the results (a binary, or a
library plus C header files) to be stored in some central location.
It was hell. We never got the dependency graph complete, so if you
made a change in one subsystem, you never knew which others had to be
rebuilt, and in which order. The job which make or cmake or whatever
would have done automatically, ended up as manual detective work. And
we could never do correct incremental builds, so we could only build
once a day, rebuilding the same things over and over again.
Frequently, that build failed because someone updated FOO without
realizing that BAR needed to be adjusted too.
Perhaps the central storage approach works, if the interfaces between
subprojects are stable and never change, so you can treat FOO as if it
was a well-designed third-party library.
But I'd much rather have a single project with a single, non-broken
build system, and build it all myself. Google for the paper "recursive
make considered harmful" for more thoughts on this.
up-to-date copy on local disk.
But I spoke more generally; you can have password-less access to a
machine and an account if you have a ssh public key which either has
no passphrase (not recommended) or has been unlocked using ssh-agent
and ssh-add. There can be one account per user, or I suppose they
could all login as a "deliverables" user.
I've never set up such a thing with strict rules for who can read or
write.
>> If you're going the ssh route, I recommend rsync over ssh instead of
>> scp or sftp. Extremely flexible, and extremely fast if you have part
>> or all of the data already.
>>
> Thanks for tips:
>
> $ rsync -rv --progress -e ssh $HOST:/path/to/files lib
>
>> Other than that, I don't understand your problem well enough to comment.
>>
> Sorry for ambiguity... I will ask another more concret questions in
> separate mails.
Oleksandr Gavenko (aka gavenkoa)
Oct 9, 2011, 5:36:26 AM10/9/11
to
On 7 окт, 18:57, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
> On Fri, 2011-10-07, Oleksandr Gavenko (aka gavenkoa) wrote:
> > On 7 ???, 03:02, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
> > The idea behind of using FTP as build artefact repository is for
> > splitting project to subprojects
> > for better module reuse and maintaining more clean interfaces.
>
> I have worked with a few products which were split up into subsystems,
> and where building subsystem FOO caused the results (a binary, or a
> library plus C header files) to be stored in some central location.
>
> It was hell. We never got the dependency graph complete, so if you
> made a change in one subsystem, you never knew which others had to be
> rebuilt, and in which order. The job which make or cmake or whatever
> would have done automatically, ended up as manual detective work. And
> we could never do correct incremental builds, so we could only build
> once a day, rebuilding the same things over and over again.
> Frequently, that build failed because someone updated FOO without
> realizing that BAR needed to be adjusted too.
>
Good point about compatibilities was written in libtool documentation.> On Fri, 2011-10-07, Oleksandr Gavenko (aka gavenkoa) wrote:
> > On 7 ???, 03:02, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
> > The idea behind of using FTP as build artefact repository is for
> > splitting project to subprojects
> > for better module reuse and maintaining more clean interfaces.
>
> I have worked with a few products which were split up into subsystems,
> and where building subsystem FOO caused the results (a binary, or a
> library plus C header files) to be stored in some central location.
>
> It was hell. We never got the dependency graph complete, so if you
> made a change in one subsystem, you never knew which others had to be
> rebuilt, and in which order. The job which make or cmake or whatever
> would have done automatically, ended up as manual detective work. And
> we could never do correct incremental builds, so we could only build
> once a day, rebuilding the same things over and over again.
> Frequently, that build failed because someone updated FOO without
> realizing that BAR needed to be adjusted too.
>
Every incompatible change require increment of [MAJOR] version
component (so if you have 'libfoo32.so' you get 'libfoo33.so', all
build scripts will link to old library until you edit build script to
point to new library version). This old article:
http://www106.pair.com/rhp/parallel.html
say that every [MAJOR] increment require new path/file names to avoid
linking with wrong version and allow simultaneously installation of
different version on single host.
> Perhaps the central storage approach works, if the interfaces between
> subprojects are stable and never change, so you can treat FOO as if it
> was a well-designed third-party library.
>
If follow instruction about that I wrote above anyone can avoid
problems.
I become to understand that compatibility MUST be verifies and
accepted EXPLICITLY.
If you want use new library version you update its name in build
scripts (version suffix) and commit changes so it shared among team.
By commit you mark that project is compatible with new library
version. You can wrote about this in CHANGES file (I think changes in
external dependencies is user visible changes).
> It may be easier if everyone just uses rsync to keep a local,
> up-to-date copy on local disk.
>
> But I spoke more generally; you can have password-less access to a
> machine and an account if you have a ssh public key which either has
> no passphrase (not recommended) or has been unlocked using ssh-agent
> and ssh-add. There can be one account per user, or I suppose they
> could all login as a "deliverables" user.
>
Good tip.
I think from Makefile you can require availability of any tools like
*ftp*, *scp*, *stfp*, *cadaver*, etc.
But for Java project this is not true. Many Java-team argue if I
prompt them install additional software besides SVN/Java/IDE/Ant.
I look Apache ANT documentation. Ant support *ftp* and *scp* task.
*ftp* is good to get anonymous downloads and *scp* is good to get
authenticated uploads (with ssh pub key).
But ANT does not support *sftp* which allow *symlink* command.
I need this command for purpose linking revision version to minor
version:
symlink /path/to/proj/1.3.2 /path/to/proj/1.3
so any dependent project download library with latest bug fixes.
So build scripts must remove old minor version directory and repeat
coping command as for revision version.
> Please don't send mail, not to me anyway. Post here instead.
>
I use Google Groups. This gateway sometimes being too smart and to
something unexpected... If this happen again I switch to
news:nntp.aioe.org server...
Jorgen Grahn
Oct 11, 2011, 6:25:49 PM10/11/11
to
On Sun, 2011-10-09, Oleksandr Gavenko (aka gavenkoa) wrote:
> On 7 ??????, 18:57, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
...
> can avoid problems.
It's one way of managing it, but the cost is still infinitely larger
than /doing nothing at all/.
But I don't know about your project, so I can't tell what you gain.
> On 7 ??????, 18:57, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
...
>> Perhaps the central storage approach works, if the interfaces between
>> subprojects are stable and never change, so you can treat FOO as if it
>> was a well-designed third-party library.
>>
> If follow instruction about that I wrote above [libtool] anyone
>> subprojects are stable and never change, so you can treat FOO as if it
>> was a well-designed third-party library.
>>
> can avoid problems.
It's one way of managing it, but the cost is still infinitely larger
than /doing nothing at all/.
But I don't know about your project, so I can't tell what you gain.
Oleksandr Gavenko (aka gavenkoa)
Oct 12, 2011, 11:10:36 AM10/12/11
to
On 12 окт, 01:25, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
> On Sun, 2011-10-09, Oleksandr Gavenko (aka gavenkoa) wrote:
>
> > On 7 ??????, 18:57, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
> ...
> >> Perhaps the central storage approach works, if the interfaces between
> >> subprojects are stable and never change, so you can treat FOO as if it
> >> was a well-designed third-party library.
>
> > If follow instruction about that I wrote above [libtool] anyone
> > can avoid problems.
>
> It's one way of managing it, but the cost is still infinitely larger
> than /doing nothing at all/.
>
> But I don't know about your project, so I can't tell what you gain.
>
After reading and thinking on project versioning I get filling
> On Sun, 2011-10-09, Oleksandr Gavenko (aka gavenkoa) wrote:
>
> > On 7 ??????, 18:57, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
> ...
> >> Perhaps the central storage approach works, if the interfaces between
> >> subprojects are stable and never change, so you can treat FOO as if it
> >> was a well-designed third-party library.
>
> > If follow instruction about that I wrote above [libtool] anyone
> > can avoid problems.
>
> It's one way of managing it, but the cost is still infinitely larger
> than /doing nothing at all/.
>
> But I don't know about your project, so I can't tell what you gain.
>
that libtool semantic is ultimate right for managing API dependency.
Can you point to another direction or problems with this approach?
Jorgen Grahn
Oct 16, 2011, 12:00:52 PM10/16/11
to
On Wed, 2011-10-12, Oleksandr Gavenko (aka gavenkoa) wrote:
Sorry for the late answer. But I was really asking *you* what you
*gain* by doing it this way. If you develop one product, why would
you want separate versioning of its internal parts?
*gain* by doing it this way. If you develop one product, why would
you want separate versioning of its internal parts?
Oleksandr Gavenko (aka gavenkoa)
Oct 16, 2011, 2:13:52 PM10/16/11
to
parts.
For example Java web service + desktop security tokens that require
native drivers.
Web service version (as synonym of feature set) can grow while driver
installer stay in own version.
Or support for new platform can be added to device, which require
driver installer version bumping.
As you can see build toolsets are differ and build machine for Java
app very differ from build machine for driver. So releases must be
stored somewhere.
I think it is good practice create some service to share release
builds. And such service must provide easy access to builds and have
good maintainability. Currently we stuck with FTP.
Also we have several custom product builds and different version
coexist in production. It is good to have access to all versions of
builds in single place to quick find released product and deploy
product as it was deployed in customers.
Also several libraries was shared between some projects. Instead of
storing copy of source in several project (with single place it is
easy maintain bug fixes) we try maintain binary builds on release
server.
Jorgen Grahn
Oct 16, 2011, 2:27:28 PM10/16/11
to
On Sun, 2011-10-16, Oleksandr Gavenko (aka gavenkoa) wrote:
> On Oct 16, 7:00 pm, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
>> On Wed, 2011-10-12, Oleksandr Gavenko (aka gavenkoa) wrote:
>> > Can you point to another direction or problems with this approach?
>>
>> Sorry for the late answer. But I was really asking *you* what you
>> *gain* by doing it this way. If you develop one product, why would
>> you want separate versioning of its internal parts?
>>
> We develop several products and product can contain mix of different
> parts.
Ah, ok. Then I agree, you should do something like this. I have no
> On Oct 16, 7:00 pm, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
>> On Wed, 2011-10-12, Oleksandr Gavenko (aka gavenkoa) wrote:
>> > Can you point to another direction or problems with this approach?
>>
>> Sorry for the late answer. But I was really asking *you* what you
>> *gain* by doing it this way. If you develop one product, why would
>> you want separate versioning of its internal parts?
>>
> We develop several products and product can contain mix of different
> parts.
direct experience which may help though. Sorry.
I only asked those questions because it is common (in my experience) to
do such splits *without* concrete reasons. I was afraid your product was
in that category.
0 new messages