docker opencog build fails with "ccache: FATAL: x_realloc: Could not allocate 17179869184 bytes"

[Apil Tamang]

Hey all, I finally got the docker-compose command to work, thus getting the opencog-dev:cli image up and running (see OpenCog's Readme). However, encountered another glitch while building the actual opencog system. I believe it was towards the end that I got a fatal error which got the command "/tmp/octool -bi" to exit with the following:

[octool] Finished building source

[octool] Source Directory is set to /opencog

[octool] Build Directory is set to /opencog/build

[octool] Starting installation

[  1%] Built target blopex

[  2%] Built target serial_driver

[  4%] Built target dimensionalembedding

[  4%] Built target statistics

[  4%] Built target embodiment_atom_types

[  5%] Built target spacetime_atom_types

[  9%] Built target PatternMiner

[  9%] Built target DistributedPatternMinerClient

[ 11%] Built target DistributedPatternMinerServer

[ 11%] Built target TestPatternMinerAgent

[ 13%] Built target cogita

[ 15%] Built target nlp_atom_types

[ 16%] Built target nlp-types

[ 26%] Built target wsd

[ 27%] Built target nlpfz

[ 29%] Built target lg-dict

[ 30%] Built target sureal

[ 30%] Built target viterbi_atom_types

[ 31%] Built target viterbi

[ 33%] Built target py-shell

[ 34%] Built target PersistModule

[ 40%] Built target server

[ 43%] Built target scheme-shell

[ 48%] Built target builtinreqs

[ 48%] Built target cogserver

[ 50%] Built target guile-cogserver

[ 51%] Built target LGDictModule

[ 51%] Built target PersistZmqModule

[ 51%] Built target QueryModule

[ 52%] Built target RuleEngineModule

[ 52%] Built target SuRealModule

[ 52%] Built target attention_atom_types

[ 54%] Built target benchmark

[ 55%] Built target agent_finder

[ 56%] Built target PythonModule

[ 56%] Built target attention-types

[ 65%] Built target attention

[ 65%] Built target spacetime-types

[ 77%] Built target SpaceMap

[ 84%] Built target spacetime

[ 86%] Built target AStar

[ 87%] Built target HPASearch

[ 87%] Built target SpaceMapUtil

[ 87%] Built target PathFinder3D

[ 88%] Built target TangentBug

[ 88%] Built target astartest

[ 90%] Built target tangentbug_test

[ 91%] Built target time_octomap

[ 91%] Built target point_memory

[ 93%] Built target cogserver_cython

[ 94%] Built target embodiment-types

[ 95%] Built target cogserver_type_constructors

[ 97%] Built target spacetime_cython

[ 98%] Built target spatial_cython

[100%] Built target statistics_cython

[100%] Built target dotty

[100%] Built target adaptor-tulip

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libdimensionalembedding.so

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libstatistics.so

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libnlp-types.so

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libwsd.so

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libnlpfz.so

Linking CXX shared library CMakeFiles/CMakeRelink.dir/liblg-dict.so

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libsureal.so

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libviterbi.so

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libpy-shell.so

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libscheme-shell.so

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libbuiltinreqs.so

Linking CXX executable CMakeFiles/CMakeRelink.dir/cogserver

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libguile-cogserver.so

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libserver.so

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libLGDictModule.so

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libPersistModule.so

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libPersistZmqModule.so

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libQueryModule.so

Linking CXX shared library CMakeFiles/CMakeRelink.dir/libRuleEngineModule.so

ccache: FATAL: x_realloc: Could not allocate 17179869184 bytes

Error running link command: Segmentation fault

make[2]: *** [opencog/cogserver/modules/CMakeFiles/CMakeRelink.dir/libRuleEngineModule.so] Error 1

make[1]: *** [opencog/cogserver/modules/CMakeFiles/RuleEngineModule.dir/preinstall] Error 2

make: *** [preinstall] Error 2

If I'm calculating it right, this is ~17 GBs of data, no !! Did this error happen because the docker container could not access all this space to store the ccache output? OR, is it because I didn't have enough RAM to handle this. I highly doubt the latter, but I couldn't be sure.

I haven't exactly delved into the depth of how docker containers get assigned volume spaces, but it seems like the output of CCACHE is stored in a shared volume (i.e. shared space on the host OS), thus having no restrictions. Might this issue have a different cause to it? If someone found an elegant way to get around this, please consider letting know.

Thanks..

[Linas Vepstas]

Yes, very strange. There's nothing in OC that is that big. Normally, ccache limits itself to  some finite size.  Perhaps octool should be changed to fix a size, e.g. 4GB by saying `ccache --max-size=4GB` 

In my experience ccache does not make that much of a difference, for opencog: I get hits about 1/4th of the time.

--linas

--
You received this message because you are subscribed to the Google Groups "opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opencog+unsubscribe@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/b0c2d023-6441-4de5-b06f-b893e7605c7b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Apil Tamang]

All,

I repeated the build once again, and did not encounter this problem. I was able to successfully build all of opencog and also telnet to the cogserver. So, looks like the docker images are good-to-go.