What to do about hard-coded path....

[Chris Barker]

I'm working on making some conda packages for standard ./configure && make && make install type builds. (on OS-X at the moment).

I get this note after building:

Detected hard-coded path in text file bin/....

for a numb erof top level scripts. And indeed, the hardd-coded path to my miniconda installation is in there. They seem to work fine on my machine, but of course, these packages are not relocatable.

What can I do about this?

PS: I'm right now doing the gnu autotools toolchain (autoconf, automake, libtool), as some packages are not shipping pre-build configure scripts (darn!), and the OS-X compilers don't have autotools. I could use homebrew, or ???, but I kind of like that Anaconda has allowed me not to depend on homebrew at all. Maybe continuum would like to put these in the official channels???

-Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Bradley Kreider]

Hey Chris:

I get this note after building:

Detected hard-coded path in text file bin/....

for a numb erof top level scripts. And indeed, the hardd-coded path to my miniconda installation is in there. They seem to work fine on my machine, but of course, these packages are not relocatable.

Have you tried "has_prefix_files?  There is also a form for binary files as well.  I've built a variety of different kinds of packages.  Here is the link to the reference: http://conda.pydata.org/docs/build.html#making-packages-relocatable

  has_prefix_files:
    - bin/file1
    - lib/file2

Good Luck!

Bradley Kreider

[Chris Barker]

that looks like what I need to do -- thuogh those docs have me stunped as to the details -- can you point me to a sample that sets these?

Also, I see:

"""
 Note that due to the way the binary replacement works, the placeholder prefix must be longer than the install prefix.
"""

but what's been put into the binaries has been decided by the build scripts for the project I'm building -- how can I control how big that is??? (maybe make sure that I've got conda installed into a really long path -- but that sure seems like a hack!

Thanks,

    -Chris

 

Good Luck!

Bradley Kreider

--
You received this message because you are subscribed to the Google Groups "conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to conda+un...@continuum.io.
To post to this group, send email to co...@continuum.io.
Visit this group at http://groups.google.com/a/continuum.io/group/conda/.

[Ilan Schnell]

conda-build itself makes sure a really long build prefix is used

[Aaron Meurer]

On Wed, Mar 4, 2015 at 1:21 PM, Chris Barker <chris....@noaa.gov> wrote:
> I'm working on making some conda packages for standard ./configure && make
> && make install type builds. (on OS-X at the moment).
>
> I get this note after building:
>
> Detected hard-coded path in text file bin/....
>
> for a numb erof top level scripts. And indeed, the hardd-coded path to my
> miniconda installation is in there. They seem to work fine on my machine,
> but of course, these packages are not relocatable.

Yes they are.

I guess the message printed to the terminal here is confusing. The
hard-coded paths in text files are managed automatically by conda
build. You don't need to worry about them at all. Conda build replaces
the hard-coded path with a placeholder, which gets replaced by the
real install prefix by conda when it is installed.

Unless you have hard-coded paths in binary files, you don't need to
worry. If you do have hard-coded paths in binary files, you have to
enable the binary prefix replacement manually (typically
"detect_binary_files_with_prefix: true" in the build section is
enough).

>
> What can I do about this?
>
> PS: I'm right now doing the gnu autotools toolchain (autoconf, automake,
> libtool), as some packages are not shipping pre-build configure scripts
> (darn!), and the OS-X compilers don't have autotools. I could use homebrew,
> or ???, but I kind of like that Anaconda has allowed me not to depend on
> homebrew at all. Maybe continuum would like to put these in the official
> channels???

I've got a gcc package, and there is a pretty good chance we will move
it to the official channels. We could also build automake, etc. as
well I suppose. They aren't as important because nothing dynamically
links against them, so there is no problem using a system version when
building a package.

Aaron Meurer

>
> -Chris
>
>
>
>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
>
> Chris....@noaa.gov
>

[Filipe Pires Alvarenga Fernandes]

I've got a gcc package, and there is a pretty good chance we will move
it to the official channels. We could also build automake, etc. as
well I suppose. They aren't as important because nothing dynamically
links against them, so there is no problem using a system version when
building a package.

If you think only about dynamical links, yes .  But for the sake of "re-producible packages" I would love to have those tools as conda packages.

[Chris Barker]

On Wed, Mar 4, 2015 at 1:36 PM, Aaron Meurer <aaron....@continuum.io> wrote:

On Wed, Mar 4, 2015 at 1:21 PM, Chris Barker <chris....@noaa.gov> wrote:
> I'm working on making some conda packages for standard ./configure && make
> && make install type builds. (on OS-X at the moment).
>
> I get this note after building:
>
> Detected hard-coded path in text file bin/....

 

> but of course, these packages are not relocatable.

Yes they are.

Cool -- great to know! 

I looked at the installed files, and they did indeed have hard-code paths, but I take these were put in there by conda on installation.

I guess the message printed to the terminal here is confusing.

yup -- maybe "Detected and repaired hard-coded path in text file bin/"

would be better.

 

Unless you have hard-coded paths in binary files, you don't need to
worry. If you do have hard-coded paths in binary files, you have to
enable the binary prefix replacement manually (typically
"detect_binary_files_with_prefix: true" in the build section is
enough).

I've got some of these too -- but I take it the message:

Detected hard-coded path in binary file bin/udunits2

means that they've been fixed.

I've got a gcc package, and there is a pretty good chance we will move
it to the official channels. We could also build automake, etc. as
well I suppose.

It seems to be pretty straightforward so far.

 

They aren't as important because nothing dynamically
links against them, so there is no problem using a system version when
building a package.

True -- but while Anaconda started out primarily as a way for folks to get a run-time environment set up, with some kind of  work, the difference between run-time and build time is blurry. 

And there are a number of us trying to build packages so that our users can have a easy to get run-time environment, but we could use some help with the build-time environment, too. Adding the gcc package and autotools has so far allowed me to do build a full set of packages without needing homebrew or any of that -- nice!

It'll also make it easier to set up CI systems (though Travis CI already does give you autotools, via homebrew by default...)

-Chris

[Chris Barker]

On Wed, Mar 4, 2015 at 2:04 PM, Chris Barker <chris....@noaa.gov> wrote:

Unless you have hard-coded paths in binary files, you don't need to
worry. If you do have hard-coded paths in binary files, you have to
enable the binary prefix replacement manually (typically
"detect_binary_files_with_prefix: true" in the build section is
enough).

I've got some of these too -- but I take it the message:

Detected hard-coded path in binary file bin/udunits2

means that they've been fixed.

Excep tthis doesn't seem right -- granted, probably the best thing to do is test on another box but I"m not easily set up for that. So:

Building the "udunits library", from this recipe:

https://github.com/ioos/conda-recipes/tree/master/udunits2

which has:

build:
	detect_binary_files_with_prefix: true

in meta.yaml.

But when built, I get:

 build_prefix.a: /Users/chris.barker/miniconda/envs/_build_placehold_placehold_placehold_placehol

which looks nice and long.

But the libs themselves have:

$ otool -L libudunits2.dylib 

libudunits2.dylib:

libudunits2.0.dylib (compatibility version 2.0.0, current version 2.0.0)

/usr/lib/libexpat.1.dylib (compatibility version 7.0.0, current version 7.2.0)

/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1669.0.0)

/Users/chris.barker/miniconda/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

Note the "Users/chris.barker..." dylib.

Which, now I look at it is the libgcc_s that presumable came with the anaconda gcc package (I got from Aaron's channel). But it actually duplicates the system libgcc. So not sure what will happen there....

OR is this all nothing to worry about?

Note that the libudunits2.la file has:

# Directory that this library needs to be installed in:

libdir='/opt/anaconda1anaconda2anaconda3/lib'

which looks like a placeholder to me.

-Chris

[Aaron Meurer]

Only paths from the build prefix are going to be replaced. This suggests that you have gcc installed in your root environment but not in your build environment (not in requirements/build), but the package is linking against libgcc_s.1.dylib, using the one from the gcc that it was compiled with.

Also, you don't have to worry about absolute paths for linking, because install_name_tool on OS X or patchelf on Linux are used to change those to relative paths. 

 

Which, now I look at it is the libgcc_s that presumable came with the anaconda gcc package (I got from Aaron's channel). But it actually duplicates the system libgcc. So not sure what will happen there....

OR is this all nothing to worry about?

Note that the libudunits2.la file has:

# Directory that this library needs to be installed in:

libdir='/opt/anaconda1anaconda2anaconda3/lib'

which looks like a placeholder to me  

.

Yep, that's the placeholder.

Aaron Meurer

[Chris Barker]

On Wed, Mar 4, 2015 at 3:29 PM, Aaron Meurer <aaron....@continuum.io> wrote:

$ otool -L libudunits2.dylib 

libudunits2.dylib:

libudunits2.0.dylib (compatibility version 2.0.0, current version 2.0.0)

/usr/lib/libexpat.1.dylib (compatibility version 7.0.0, current version 7.2.0)

/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1669.0.0)

/Users/chris.barker/miniconda/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

Note the "Users/chris.barker..." dylib.

Only paths from the build prefix are going to be replaced. This suggests that you have gcc installed in your root environment but not in your build environment (not in requirements/build), but the package is linking against libgcc_s.1.dylib, using the one from the gcc that it was compiled with.

indeed -- that's the case. I did put gcc in the requirements/build, but then conda build couldn't find it.

I installed it with conda install and specifying your channel -- I didn't want to add your channel globally, as I don't want to accidentally pick things up there.

conda list shows it as installed:

# packages in environment at /Users/chris.barker/miniconda:

...

gcc                       4.8.2                         5  

...

but if I do conda install, it doesn't find it:

$ conda install gcc

Fetching package metadata: ......

Error: No packages found in current osx-64 channels matching: gcc

similarly,l if I add it as a build requirement, I get:

$ conda build conda-recipes/udunits2/

BUILD START: udunits2-2.2.17-0

Fetching package metadata: ........

Error: No packages found in current osx-64 channels matching: gcc

I'm not sure why it can't find it in the root environment - isn't it supposed to?

But I don't want to specify it as a build requirement anyway -- other folks building this package may want to use whatever gcc they have installed.

Also, you don't have to worry about absolute paths for linking, because install_name_tool on OS X or patchelf on Linux are used to change those to relative paths.

OK -- this looks like it will work fine?

-Chris

[Aaron Meurer]

It's only in my channel. But we are in the process of moving it to the
official channels now.

>
> But I don't want to specify it as a build requirement anyway -- other folks
> building this package may want to use whatever gcc they have installed.

But your library is linking to libgcc. So you may want to see if you
can get it to link statically. Otherwise, it's a good idea to have it
link to the same library it was built with.

Aaron Meurer

[Chris Barker]

On Wed, Mar 4, 2015 at 3:56 PM, Aaron Meurer <aaron....@continuum.io> wrote:

> conda list shows it as installed:
>
> # packages in environment at /Users/chris.barker/miniconda:
> ...
> gcc                       4.8.2                         5

 

...

 

> similarly,l if I add it as a build requirement, I get:

>
> $ conda build conda-recipes/udunits2/
> BUILD START: udunits2-2.2.17-0
> Fetching package metadata: ........
> Error: No packages found in current osx-64 channels matching: gcc
>
> I'm not sure why it can't find it in the root environment - isn't it
> supposed to?

It's only in my channel. But we are in the process of moving it to the
official channels now.

got that -- but I don't get why it can't find a package that's installed in root.

In fact, I have the same problem with packages I build and install myself, on my machine -- I can't use them as a dependency in new builds.

Is there a way to set up a "channel" that's just a directory on my disk? That would do it.

> But I don't want to specify it as a build requirement anyway -- other folks
> building this package may want to use whatever gcc they have installed.

But your library is linking to libgcc. So you may want to see if you
can get it to link statically. Otherwise, it's a good idea to have it
link to the same library it was built with.

hmm -- tricky on here then -- we want the system libs if we can.

But maybe the way to go is not use your gcc until/unless it's officially available in Anaconda.

(but I wanted it for Fortran -- darn!)

-Chris

[Aaron Meurer]

On Wed, Mar 4, 2015 at 6:13 PM, Chris Barker <chris....@noaa.gov> wrote:
> On Wed, Mar 4, 2015 at 3:56 PM, Aaron Meurer <aaron....@continuum.io>
> wrote:
>>
>> > conda list shows it as installed:
>> >
>> > # packages in environment at /Users/chris.barker/miniconda:
>> > ...
>> > gcc 4.8.2 5
>
>
> ...
>
>>
>> > similarly,l if I add it as a build requirement, I get:
>> >
>> > $ conda build conda-recipes/udunits2/
>> > BUILD START: udunits2-2.2.17-0
>> > Fetching package metadata: ........
>> > Error: No packages found in current osx-64 channels matching: gcc
>> >
>> > I'm not sure why it can't find it in the root environment - isn't it
>> > supposed to?
>>
>> It's only in my channel. But we are in the process of moving it to the
>> official channels now.
>
>
> got that -- but I don't get why it can't find a package that's installed in
> root.

If you used the -c flag to install it from my channel, it won't know
where to find it thereafter unless you use -c again, or add the
channel to your condarc.

>
> In fact, I have the same problem with packages I build and install myself,
> on my machine -- I can't use them as a dependency in new builds.

You should be able to if they are in the conda-bld directory.

>
> Is there a way to set up a "channel" that's just a directory on my disk?
> That would do it.
>
>> > But I don't want to specify it as a build requirement anyway -- other
>> > folks
>> > building this package may want to use whatever gcc they have installed.
>>
>> But your library is linking to libgcc. So you may want to see if you
>> can get it to link statically. Otherwise, it's a good idea to have it
>> link to the same library it was built with.
>
>
> hmm -- tricky on here then -- we want the system libs if we can.
>
> But maybe the way to go is not use your gcc until/unless it's officially
> available in Anaconda.
>
> (but I wanted it for Fortran -- darn!)
>
> -Chris

We will put it in the Continuum repos, but the next step will be to
split out a libgcc for linking, since the gcc packages are quite
large.

Aaron Meurer

[Chris Barker]

On Wed, Mar 4, 2015 at 4:21 PM, Aaron Meurer <aaron....@continuum.io> wrote:

> got that -- but I don't get why it can't find a package that's installed in
> root.

If you used the -c flag to install it from my channel, it won't know
where to find it thereafter unless you use -c again, or add the
channel to your condarc.

right, but I don't want to do that, as I don't want it to find anything else there... that's the trick. For now,  I've removed the gcc package, so it'll just build with my system gcc -- working fine so far -- I'll cross the Fortran bridge later!

> In fact, I have the same problem with packages I build and install myself,
> on my machine -- I can't use them as a dependency in new builds.

You should be able to if they are in the conda-bld directory.

OK -- odd -- maybe I did something else weird -- I'll try again.

Thanks for all your help!

 > But maybe the way to go is not use your gcc until/unless it's officially

> available in Anaconda.

 

We will put it in the Continuum repos, but the next step will be to

split out a libgcc for linking, since the gcc packages are quite
large.

Agreed -- that would be better. And maybe spli tout fortran, etc? if it's not too much of a mess to do so.

-Chris

[Chris Barker]

On Wed, Mar 4, 2015 at 4:21 PM, Aaron Meurer <aaron....@continuum.io> wrote:

On Wed, Mar 4, 2015 at 6:13 PM, Chris Barker <chris....@noaa.gov> wrote:
We will put it in the Continuum repos, but the next step will be to
split out a libgcc for linking, since the gcc packages are quite
large.

If you want to do autotools (and I hope you do), I've put recipes up here:

https://github.com/ChrisBarker-NOAA/autotools_conda_recipes

they are trivial, but might be faster to copy that re-write them...

-Chris