How to port external library into pNaCl Toolchain

[Anoop Singh]

Hi Group Members,

I wanted to know the clear steps to include an external library into the pNaCl toolchain.

When i checked into webports some packages are already ported. But I don't know how it was ported and what code modifications were done to get this porting done.

I am seeing under any package in webports there is build.sh file alsong with other files. How does it work? 

Let us consider a package which has some CC files and header files along with Makefile, how can i port it to pNaCl toolchain  or How can i make a build to get the .pexe output of this package?

Please provide me the detail steps means where to copy the source code and what modifications are required and what command needs to run to make build as I am very new to this work.

Thanks & regards,

Anoop.

[Sam Clegg]

On Mon, Jan 25, 2016 at 12:06 AM, Anoop Singh <aks....@gmail.com> wrote:
> Hi Group Members,
>
> I wanted to know the clear steps to include an external library into the
> pNaCl toolchain.
> When i checked into webports some packages are already ported. But I don't
> know how it was ported and what code modifications were done to get this
> porting done.
>
> I am seeing under any package in webports there is build.sh file alsong with
> other files. How does it work?

You probably want to start by taking a look at the docs:
https://chromium.googlesource.com/webports/+/master/README.md

Most ports don't require patching. For those that need a patch you
can see the changes in the "nacl.patch" file.

>
> Let us consider a package which has some CC files and header files along
> with Makefile, how can i port it to pNaCl toolchain or How can i make a
> build to get the .pexe output of this package?

As long as the Makefile honors the standard CC/CXX/CFLAGS/CXXFLAGS
environment variables then the wepports build system should take care
of all this for you.

>
> Please provide me the detail steps means where to copy the source code and
> what modifications are required and what command needs to run to make build
> as I am very new to this work.
>
> Thanks & regards,
> Anoop.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Native-Client-Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to native-client-di...@googlegroups.com.
> To post to this group, send email to native-cli...@googlegroups.com.
> Visit this group at https://groups.google.com/group/native-client-discuss.
> For more options, visit https://groups.google.com/d/optout.

[Anoop Singh]

Hi Sam,

Thanks a lot for replying. 

I have gone through the webports documentation and I was able to build imagemagick which was there already(ported in webports) using command "webports install imagemagick".

But still I have some confusion. Let us say a code base which is not part of webports how to build it as part of pnacl toolchain.

I have this confusion as in webports, we don't have any code. Ports have just a folder and inside the folder we have build.sh file and nacl.patch file. When we run command "webports install <package>" it builds and create .pexe but this way I can not use for my code building.

Assume i have a code base which has autogen.sh, makefile.am, configure.ac and src folder. Under src folder I have cpp and .h files.

Now i want to make a build as part of pNacl toolchain. How can I do it? I want to build on linux 86_64 and get .pexe out of it which is platform independent and runs on chrome browser. 

Do I need to write build.sh file? Where to put my source code and what commands need to be used to build it?

One more thing, if I have precompiled binaries and libraries created on linux build, can I include them directly into pNacl toolchain(??) and build them to get corresponding .pexe?

I read that  a file can be build using command " pnacl-clang++ hello.cc -o hello.pexe" ? I tried and its working for me. Can I extend my full code base to be built using Makefile and pnacl-clang++?

Please help me to overcome these confusions as i am new to pnacl-library. Thanks a lot in advance.

Thanks & regards,

Anoop. 

[Sam Clegg]

On Tue, Jan 26, 2016 at 1:04 AM, Anoop Singh <aks....@gmail.com> wrote:
> Hi Sam,
>
> Thanks a lot for replying.
>
> I have gone through the webports documentation and I was able to build
> imagemagick which was there already(ported in webports) using command
> "webports install imagemagick".
>
> But still I have some confusion. Let us say a code base which is not part of
> webports how to build it as part of pnacl toolchain.
>
> I have this confusion as in webports, we don't have any code. Ports have
> just a folder and inside the folder we have build.sh file and nacl.patch
> file. When we run command "webports install <package>" it builds and create
> .pexe but this way I can not use for my code building.
>
> Assume i have a code base which has autogen.sh, makefile.am, configure.ac
> and src folder. Under src folder I have cpp and .h files.
> Now i want to make a build as part of pNacl toolchain. How can I do it? I
> want to build on linux 86_64 and get .pexe out of it which is platform
> independent and runs on chrome browser.
> Do I need to write build.sh file? Where to put my source code and what
> commands need to be used to build it?

If you have code which is based on autoconf, automake (or any build
system that honors the same environment variables such as CC/CXX/etc),
then all you really need to do is set your environment up correctly
when you run configure, and then simply type 'make'. This is the same
for any cross compile target.

There are some helper scripts in webports that make this easier. See
build_tools/nacl-configure.sh and build_tools/nacl-make.sh. Bare in
mind that these wrappers don't have any test coverage yet, but I'd be
happy to add some if they prove useful to you.

> One more thing, if I have precompiled binaries and libraries created on
> linux build, can I include them directly into pNacl toolchain(??) and build
> them to get corresponding .pexe?

If you mean can you use libraries compiled with your host linux
toolchain, then no. All librares that are linked into pnacl
executables must be generated by the pnacl compiler (or some other
tool that knows how to produce compatible bitcode files).

[Anoop Singh]

Hi Sam,

Thanks a lot for clearing some of my confusions and pointing towards the script nacl-configure.sh and nacl-make.sh which may help me to build the package.

I will try it today and will ask if i face any issue. Are these scripts help in making pnacl build too?

Thanks a lot once again.

With best regards,
Anoop.

[Sam Clegg]

On Wed, Jan 27, 2016 at 9:30 AM, Anoop Singh <aks....@gmail.com> wrote:
> Hi Sam,
>

> Thanks a lot for clearing some of my confusions and pointing towards the
> script nacl-configure.sh and nacl-make.sh which may help me to build the
> package.
>
> I will try it today and will ask if i face any issue. Are these scripts help
> in making pnacl build too?

Yes, they honor the TOOLCHAIN environment variable, just like webports
itself. So you can do something like: TOOLCHAIN=pnacl
path/to/nacl-configure.sh ./configure.

[Anoop Singh]

Hi Sam,

 

Thanks for the guidance.

 

I copied my code into the /webports/src/ports/<code-root-directory> and using the command TOOLCHAIN=pnacl ../../build_tools/nacl-configure.sh ./configure, I tried to configure.

I got the error "configure script not found: ./configure "

Then I thought of trying the command in this way "TOOLCHAIN=pnacl ../../build_tools/nacl-configure.sh ./autogen.sh" as my code contains autogen.sh file but I got this

 

errorFailed to find patch 0 location in configure script: ./autogen.sh

Failed to find patch 1 location in configure script: ./autogen.sh

Failed to find patch 2 location in configure script: ./autogen.sh

Failed to find config.sub (./config.sub).

Please specify using $CONFIG_SUB.

 

Because of this, I tried to run only "sh autogen.sh" in my code which went through and generated configure file and then I ran this command again

TOOLCHAIN=pnacl ../../build_tools/nacl-configure.sh ./configure 

Actually autogen file script installs autoreconf and does the configure operation.

 

This worked and it created Makefile with correct CC,CFFLAGS as per the pnacl architecture. Is this the way to do? or I missed something? 

My next step is to run make as Makefile is proper as per pnacl architecture now.

 

I ran "make" command under my code root directory it shows me these error:

 

error: variable has incomplete type 'struct timeval'

error: use of undeclared identifier 'MSG_NOSIGNAL'

 

These errors are from my code only. These two errors are due to the reason MSG_NOSIGNAL is defined in #include <sys/socket.h> and struct timeval is defined in #include <sys/time.h>.

As these two header files are linux supported, I think pnacl is not able to get them. Any guess how to fix this issue? Do we need to get pnacl equivalent of these? Or do you think the generated Makefile is not having the correct values?

 

When I make gcc build on my ubuntu machine the same code goes through without any error which is obvious.

 

Could you please also let me know how to use build_tools/nacl-make.sh file to make a build correctly? like the command you suggested for configure as "TOOLCHAIN=pnacl ../../build_tools/nacl-configure.sh ./configure".

 

Thanks & regards,

Anoop.

[Anoop Singh]

Hi Sam,

This is in continuation of my previous mail. I am giving you the env values set in my make file.

CC =nacl_sdk/pepper_47/toolchain/linux_pnacl/bin/pnacl-clang

CCDEPMODE = depmode=gcc3

CFLAGS = -g -O2

CPP =nacl_sdk/pepper_47/toolchain/linux_pnacl/bin/pnacl-clang -E

CPPFLAGS =  -I/nacl_sdk/pepper_47/include

CXX = nacl_sdk/pepper_47/toolchain/linux_pnacl/bin/pnacl-clang++

CXXCPP = nacl_sdk/pepper_47/toolchain/linux_pnacl/bin/pnacl-clang++ -E

CXXDEPMODE = depmode=gcc3

CXXFLAGS = -g -O2

CYGPATH_W = echo

LDFLAGS =  -L/nacl_sdk/pepper_47/lib/pnacl/Release

build = x86_64-unknown-linux-gnu

build_alias =

build_cpu = x86_64

build_os = linux-gnu

install_sh = ${SHELL} /home/anoop/webports/src/ports/pdl_converter/install-sh

Please try to answer my queries posed in previous mail by lloking these env if required.

Many many thanks in advance. 

Thanks & regards,

Anoop.

[Sam Clegg]

On Thu, Jan 28, 2016 at 3:53 AM, Anoop Singh <aks....@gmail.com> wrote:
> Hi Sam,
>
>
>

> Thanks for the guidance.
>
>
>
> I copied my code into the /webports/src/ports/<code-root-directory> and
> using the command TOOLCHAIN=pnacl ../../build_tools/nacl-configure.sh
> ./configure, I tried to configure.
>
> I got the error "configure script not found: ./configure "
>
> Then I thought of trying the command in this way "TOOLCHAIN=pnacl
> ../../build_tools/nacl-configure.sh ./autogen.sh" as my code contains
> autogen.sh file but I got this
>
>
>
> errorFailed to find patch 0 location in configure script: ./autogen.sh
>
> Failed to find patch 1 location in configure script: ./autogen.sh
>
> Failed to find patch 2 location in configure script: ./autogen.sh
>
> Failed to find config.sub (./config.sub).
>
> Please specify using $CONFIG_SUB.
>
>
>
> Because of this, I tried to run only "sh autogen.sh" in my code which went
> through and generated configure file and then I ran this command again
>
> TOOLCHAIN=pnacl ../../build_tools/nacl-configure.sh ./configure
>
> Actually autogen file script installs autoreconf and does the configure
> operation.
>
>
>
> This worked and it created Makefile with correct CC,CFFLAGS as per the pnacl
> architecture. Is this the way to do? or I missed something?

This seems correct, although you don't need to copy your source into
the webports tree. You can just run nacl-configure.sh from wherever
your configure script is.

>
> My next step is to run make as Makefile is proper as per pnacl architecture
> now.
>
>
>
> I ran "make" command under my code root directory it shows me these error:
>
>
>
> error: variable has incomplete type 'struct timeval'
>
> error: use of undeclared identifier 'MSG_NOSIGNAL'
>
>
>
> These errors are from my code only. These two errors are due to the reason
> MSG_NOSIGNAL is defined in #include <sys/socket.h> and struct timeval is
> defined in #include <sys/time.h>.
>
> As these two header files are linux supported, I think pnacl is not able to
> get them. Any guess how to fix this issue? Do we need to get pnacl
> equivalent of these? Or do you think the generated Makefile is not having
> the correct values?

You are doing the right thing here. NaCl does provide a lot of the
POSIX interfaces such as these (most via the nacl_io library). In
this case MSG_NOSIGNAL not part of the sys/socket.h we provide so you
will want to guard its use in your code (the simplest hack is to use
"ifndef __native_client__").

For 'struct timeval', this should be present in the sys/time.h. Are
you sure you are including there?

cheers,
sam

[Anoop Singh]

Hi Sam,

Thanks for the clarifications.

To guard MSG_NOSIGNAL, I will go as suggested by you. I will protect the code where MSG_NOSIGNAL is used in this way. 

#ifndef _native_client_

 if(size < chunkSize) chunkSize = size;

        while ((chunkSize > 0) && (chunkSize <= size )) {

                if(((iBytesSent = send(transporter, data, (int) chunkSize, MSG_NOSIGNAL)) < 0) &&

                        (errno != EINTR))

                {

#if DEBUG

                        cout<<"Connection closed!! errno = "<< errno << endl;

#endif

                        close(transporter);

                        return 0;

                }

#endif

Is it correct? In this case do I need to add some more chunk of code too? I am a bit confused here. 

Regarding 'struct timeval' I added the file and build goes through. No error. I am wondering how gcc build was going through without this included?

Thanks a lot for your help.

With best regards,

Anoop.

[Anoop Singh]

Hi Sam,

I don't think the way I fixed MSG_NOSIGNAL issue is correct as I told in my previous mail. 

Please let me know the correct way of doing it. For temporary I replaced MSG_NOSIGNAL with value 0 and build goes through. 

One more thing, when I make a gcc build for my package, I get convert.a, convert.la, convert.so, convert.so.0.0.0 in the specified lib folder.

What output I should get for the pnacl build? I see only convert.a and convert.la but no convert.so or convert.so.0.0.0 in the specified folder. Please let me know how to get it and what is the correct lib format for pnacl?

For one another library when i run command

"TOOLCHAIN=pnacl build_tools/nacl-configure.sh ./configure" after running my autogen.sh script I get error like this:

Failed to find patch 1 location in configure script: ./configure

Failed to find config.sub (./config.sub).

Please specify using $CONFIG_SUB.

configure file is already generated but still above error. Could you please let me know what could be the issue? But this same error I don't see for my previous package.

Thanks a lot for your kind help. 

Thanks & regards,

Anoop.

[Anoop Singh]

Hi Sam,

I don't think the way I fixed MSG_NOSIGNAL issue is correct as I told in my previous mail. 

Please let me know the correct way of doing it. For temporary I replaced MSG_NOSIGNAL with value 0 and build goes through. 

One more thing, when I make a gcc build for my package, I get convert.a, convert.la, convert.so, convert.so.0.0.0 in the specified lib folder.

What output I should get for the pnacl build? I see only convert.a and convert.la but no convert.so or convert.so.0.0.0 in the specified folder. Please let me know how to get it and what is the correct lib format for pnacl? Actually i wanted to get .pexe out of the build so that i can use it with chrome browser using manifest.nmf file. How is this possible? Can i use the library only to communicate with the chrome browser, if yes, then how?

For one another library when i run command

"TOOLCHAIN=pnacl build_tools/nacl-configure.sh ./configure" after running my autogen.sh script I get error like this:

Failed to find patch 1 location in configure script: ./configure

Failed to find config.sub (./config.sub).

Please specify using $CONFIG_SUB.

configure file is already generated but still above error. Could you please let me know what could be the issue? But this same error I don't see for my previous package.

Thanks a lot for your kind help and great guidance.

Thanks & regards,

Anoop.

[Sam Clegg]

On Fri, Jan 29, 2016 at 8:23 AM, Anoop Singh <aks....@gmail.com> wrote:
> Hi Sam,
>

> I don't think the way I fixed MSG_NOSIGNAL issue is correct as I told in my
> previous mail.
> Please let me know the correct way of doing it. For temporary I replaced
> MSG_NOSIGNAL with value 0 and build goes through.

That seems like a reasonable hack.

>
> One more thing, when I make a gcc build for my package, I get convert.a,
> convert.la, convert.so, convert.so.0.0.0 in the specified lib folder.
>
> What output I should get for the pnacl build? I see only convert.a and
> convert.la but no convert.so or convert.so.0.0.0 in the specified folder.

This is because the pnacl toolchain does not support shared libraries.

> Please let me know how to get it and what is the correct lib format for
> pnacl? Actually i wanted to get .pexe out of the build so that i can use it
> with chrome browser using manifest.nmf file. How is this possible? Can i use
> the library only to communicate with the chrome browser, if yes, then how?

If you have an executalbe target in your Makefile.in or Makefile.am
then it will build as a .pexe file when you cross compile with the
pnacl toolchain (it might not have the .pexe extension).

>
> For one another library when i run command
> "TOOLCHAIN=pnacl build_tools/nacl-configure.sh ./configure" after running my
> autogen.sh script I get error like this:
>
> Failed to find patch 1 location in configure script: ./configure
> Failed to find config.sub (./config.sub).
> Please specify using $CONFIG_SUB.
>
> configure file is already generated but still above error. Could you please
> let me know what could be the issue? But this same error I don't see for my
> previous package.

The package should have a config.sub file somewhere. You should set
$CONFIG_SUB to point to the location of the config.sub file.

[Anoop Singh]

Hi Sam,

Thanks a lot for all your valuable guidance. It made me to learn pNacl in better way. My pnacl build is successful. :). 

I have some conceptual problem yet due to being new to this technology but I will ask those questions as part of another thread. 

Thanks a lot once again. Hats off to your deep knowledge. 

With best regards,

Anoop.