Python 3 Scikitlearn

[mjde...@gmail.com]

Hi,

I tried to use the OSv app for python 3 support. I built the image with the system installed python 3 version. I also installed scikit learn and added that to the modules to keep from the site packages. Scikit also relies on numpy and scipy. However, I got it to import and build an image. But when I tried to run it, failed with an unresolved symbol. I have a centos system and was building within the docker container fedora 29 template. Any suggestions or thoughts on why this would fail?

Thanks,

Mike

[Waldek Kozaczuk]

Hi,

Can you provide exact build command you used to build the image? Also, can you provide the exact stack trace so we know which symbol is was unresolved?

Thanks,

Waldek

[Michael De Lucia]

I Waldek,

I will get the exact command and stack trace to you on Friday.

Thanks,

Mike

Sent from my iPhone

--
You received this message because you are subscribed to a topic in the Google Groups "OSv Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/osv-dev/V9fch2WFX7o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/61cb76c8-63f4-490c-b040-5634148c3930%40googlegroups.com.

[Waldek Kozaczuk]

Is it possibly the same issue as described here - https://github.com/cloudius-systems/osv/issues/1025?

On Wednesday, August 21, 2019 at 10:08:15 PM UTC-4, Michael De Lucia wrote:

I Waldek,

I will get the exact command and stack trace to you on Friday.

Thanks,

Mike

Sent from my iPhone

On Aug 21, 2019, at 9:39 PM, Waldek Kozaczuk <jwkoz...@gmail.com> wrote:

Hi,

Can you provide exact build command you used to build the image? Also, can you provide the exact stack trace so we know which symbol is was unresolved?

Thanks,

Waldek

On Wednesday, August 21, 2019 at 9:19:36 PM UTC-4, mjde...@gmail.com wrote:

Hi,

I tried to use the OSv app for python 3 support. I built the image with the system installed python 3 version. I also installed scikit learn and added that to the modules to keep from the site packages. Scikit also relies on numpy and scipy. However, I got it to import and build an image. But when I tried to run it, failed with an unresolved symbol. I have a centos system and was building within the docker container fedora 29 template. Any suggestions or thoughts on why this would fail?

Thanks,

Mike

--
You received this message because you are subscribed to a topic in the Google Groups "OSv Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/osv-dev/V9fch2WFX7o/unsubscribe.

To unsubscribe from this group and all its topics, send an email to osv...@googlegroups.com.

[Michael De Lucia]

Hi Waldek,

 

No that does not seem to be the problem. I started with just trying to see with numpy first, since that is a dependency. Here is what I did. I used your docker image build on Fedora29, which does a git and pulls down the OSv repo. Then I built the python3x osv-app in osv-apps/python3x/ on the github repo.

 

I have python 3.7 installed on the fedora29 image and did pip3 install numpy to get that installed in my python 3.7 environment. I modified the GET file under the python3x directory to include the numpy module.

 

Next I ran ./scripts/build image=python3x and the build completed successfully.

 

Then I ran ./scripts/run.py –api -e “/python3 -c \”import numpy; print(numpy.__version__)\””

 

That is when it failed in an unresolved symbol for blas_memory_alloc

 

[root@8ec8090052ab osv]# ./scripts/run.py --api -e "/python3 -c \"import numpy; print(numpy.__version__)\""

OSv v0.53.0-74-gef56fde7

eth0: 192.168.122.15

Booted up in 611.56 ms

/lib/python3.7/numpy/.libs/libopenblasp-r0-2ecf47d5.3.7.dev.so: failed looking up symbol blas_memory_alloc

 

[backtrace]

0x0000000040356209 <elf::object::symbol(unsigned int, bool)+969>

0x00000000403562cf <elf::object::resolve_pltgot(unsigned int)+127>

0x0000000040356494 <elf_resolve_pltgot+52>

0x000000004039b82f <???+1077524527>

0x000020000077ff6f <???+7864175>

0x0000000000000003 <???+3>

 

Any ideas on how to resolve this?

 

Thanks,

 

Mike

To unsubscribe from this group and all its topics, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/dba8eead-7d65-4f86-bef0-98eb80925d6d%40googlegroups.com.

[Waldek Kozaczuk]

It looks like you have one of the libraries, libopenblasp-r0-2ecf47d5.3.7.dev.so depends on, missing.

You can use:

./scripts/manifest_from_host.sh -l /lib/python3.7/numpy/.libs/libopenblasp-r0-2ecf47d5.3.7.dev.so

to see which libraries have to be added.

To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/dba8eead-7d65-4f86-bef0-98eb80925d6d%40googlegroups.com.

[Michael De Lucia]

Waldek,

 

The openblas was in a different directory, here is the location and command, and output. It looks like there is no dependences, but it is a position depended executable. Solutions?

 

[root@f955a42a9e41 osv]# ./scripts/manifest_from_host.sh -l /usr/local/lib64/python3.7/site-packages/numpy/.libs/libopenblasp-r0-2ecf47d5.3.7.dev.so

# Position Dependent Executable

/libopenblasp-r0-2ecf47d5.3.7.dev.so: /usr/local/lib64/python3.7/site-packages/numpy/.libs/libopenblasp-r0-2ecf47d5.3.7.dev.so

# --------------------

# Dependencies

# --------------------

# --------------------

To unsubscribe from this group and all its topics, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/e1bdaf15-e259-4bad-8beb-15f81c4506f8%40googlegroups.com.

[Michael De Lucia]

Waldek,

I used your Ubuntu docker builder and   numpy worked. I just had to also include the unit test module. It seemed to work. Not sure the differences between Ubuntu and Fedora build.

Thanks,

Mike

Sent from my iPhone

To unsubscribe from this group and all its topics, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/e1bdaf15-e259-4bad-8beb-15f81c4506f8%40googlegroups.com.

[Michael De Lucia]

Waldek,

 

I finally got sklearn installed and got to the same issue you pointed out described here - https://github.com/cloudius-systems/osv/issues/1025

 

Has this been implemented yet?

 

Thanks,

 

Mike

 

From: osv...@googlegroups.com <osv...@googlegroups.com> On Behalf Of Waldek Kozaczuk

Sent: Tuesday, August 27, 2019 11:40 AM

To unsubscribe from this group and all its topics, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/e1bdaf15-e259-4bad-8beb-15f81c4506f8%40googlegroups.com.

[Waldek Kozaczuk]

Hi,

Given you are the second person asking about it, I started implementing it last night given it is not that difficult. But it would still take a bit of time to make it correct and covered with unit tests.

However, I peeked more at the details of https://github.com/cloudius-systems/osv/issues/1025 and realized scikit-learn uses the multiprocessing module (https://docs.python.org/3.4/library/multiprocessing.html?highlight=process) which allows multiprocessing based on forking new processes vs multithreading. OSv supports pthreads very well even on multiple cores (SMP) but being unikernel it does NOT and never intended to support fork()/spawn(). That being said I suspect that even if implemented name semaphores you will hit the fork() obstacle down the road.

Based on this doc - https://scipy.github.io/old-wiki/pages/ParallelProgramming, and https://scipy.github.io/old-wiki/pages/Cookbook/Multithreading.html you can use SciPy with multithreading which should work on OSv.

There is also GraalVM Python which is supposed to run python much faster and is focused on SciPy but it is still experimental - https://www.graalvm.org/docs/reference-manual/languages/python/. I have had a lot of luck running native images (JVM code AOT-compiled to machine code) on OSv.

Regards,

Waldek

To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/e1bdaf15-e259-4bad-8beb-15f81c4506f8%40googlegroups.com.

[Michael De Lucia]

Thanks Waldek! You can compile sklearn without openmp, which I believe would get rid of the multiprocessing part.

As an alternative I guess I could try and use the weka machine learning library which is in Java.

Mike

Sent from my iPhone

To unsubscribe from this group and all its topics, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/d7902b92-c058-4fcb-8db9-bcdaf0152b8a%40googlegroups.com.

[Henrique Fingler]

 Hi Waldek,

 Any update on this issue? I implemented a extremely horrible sem_open and stubbed close and unlink, but I'm getting a mmap fault somewhere in LogisticRegression fit (https://github.com/scikit-learn/scikit-learn/blob/1495f6924/sklearn/linear_model/logistic.py#L1466).

0x000000004046537b <osv::generate_signal(siginfo&, exception_frame*)+59>

0x00000000404653ea <osv::handle_mmap_fault(unsigned long, int, exception_frame*)+26>

0x000000004032f509 <mmu::vm_fault(unsigned long, exception_frame*)+185>

0x0000000040393a36 <page_fault+166>

0x0000000040392876 <???+1077487734>

0x0000100000c11e5f <???+12656223>

[Henrique Fingler]

 Welp, spoke too soon. I was able to make it run by recompiling sklearn without openmp using the SKLEARN_NO_OPENMP flag.   (https://scikit-learn.org/dev/developers/advanced_installation.html)
 If I knew c++ better I'd implement a real sem_open, although I'm almost sure stubbing it is enough for sklearn.

[Geraldo Netto]

Dear Henrique/Friends,

Maybe we could "reshape" freebsd[1]/musl libc implementation [2]
Of course, it might take a while but it's doable

[1] https://github.com/freebsd/freebsd/search?q=sem_open&unscoped_q=sem_open
[2] https://git.musl-libc.org/cgit/musl/tree/src/thread


Keep Rocking,

Geraldo Netto
Sapere Aude => Non dvcor, dvco
site: http://exdev.sf.net/
github: https://github.com/geraldo-netto
linkedin: https://www.linkedin.com/in/geraldonetto/
facebook: https://web.facebook.com/geraldo.netto.161

> You received this message because you are subscribed to the Google Groups "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/e8ecc4f5-ccac-4f5d-84a1-6f50e4586f10%40googlegroups.com.

[Waldek Kozaczuk]

Hi,

I have a WIP branch with named semaphores implementation - https://github.com/wkozaczuk/osv/commit/157657cd1add282d7987f9040f0e71a579c59ef6 - not finished though. 

But I need to better understand your motivation - are you trying to implement sem_open() to be able to run SciPy which relies on multiprocessing unit? If that is the case, then as you read one of my previous replies in this thread - https://groups.google.com/d/msg/osv-dev/V9fch2WFX7o/PWS_WoNRBgAJ - OSv does not support fork()/execve() so it would never work. Or do you need named semaphores support (sem_open()) for other reasons?