[PATCH] Add ssl support to curl in opkg

[Camille Moncelier]

This patch adds to opkg several option which allows to
configure curl to use openssl autenticate an https session.

The client when downloading packages will be able to verify
the sever signature, authenticate itself, via certificates on
disk or on a smartcard.

# example opkg.conf settings

option ssl_ca_path /etc/ssl/certs
option ssl_key_type ENG

option ssl_cert_type PEM
option ssl_cert /etc/opkg/client.pem

option ssl_engine pkcs11

[Camille Moncelier]

[Camille Moncelier]

Can somebody test this ? I'd like to commit it but since I'm
the only one who tested that I would like some comments
before pushing that into SVN.

[Graham Gower]

2009/10/29 Camille Moncelier <monc...@devlife.org>:

> Can somebody test this ? I'd like to commit it but since I'm
> the only one who tested that I would like some comments
> before pushing that into SVN.

Yep, I'll have a play with it in the next day or two.

[Graham Gower]

There is a build issue when configured with --disable-ssl-curl, but
that is just a matter of moving around a #define.

I'm testing on x86 with an offline root, using package lists etc.
copied from a rootfs generated with OpenEmbedded.

My opkg.conf contains the following:
----
option ssl_ca_path /tmp/

option ssl_key_type ENG
option ssl_cert_type PEM

option ssl_cert /dev/null
option ssl_engine pkcs11

dest root /home/grg/opkg
lists_dir ext /home/grg/opkg/var/lib/opkg
----

And I get a segfault when trying to retrieve Packages.gz.

grg@ubiquitous:~/opkg/code/opkg/src$ ./opkg-cl -V 3 -o ~/opkg/ update
Downloading http://10.0.0.13/ipk/mipsel/Packages.gz
Downloading http://10.0.0.13/ipk/all/Packages.gz
Segmentation fault (core dumped)

The core dump isn't useful as I don't have debug symbols in
libcurl.... In any event, the problem can be solved by putting "curl =
NULL;" in opkg_curl_init() error paths, under each of the two calls to
curl_easy_cleanup().

Now that the error messages get collected at the end, I get:

grg@ubiquitous:~/opkg/code/opkg/src$ ./opkg-cl -V 3 -o ~/opkg/ update
Downloading http://10.0.0.13/ipk/mipsel/Packages.gz
Downloading http://10.0.0.13/ipk/all/Packages.gz
Downloading http://10.0.0.13/ipk/quokka/Packages.gz
Collected errors:
* can't set crypto engine: 'pkcs11'
* can't set crypto engine: 'pkcs11'
* can't set crypto engine: 'pkcs11'
hash_table[pkg-hash] n_buckets=0 n_elements=5814 max_conflicts=0 n_conflicts=0
hash_table[file-hash] n_buckets=0 n_elements=0 max_conflicts=0 n_conflicts=0
hash_table[obs-file-hash] n_buckets=0 n_elements=0 max_conflicts=0 n_conflicts=0

Sorry, I've not played with ssl certs in the past, beyond self signing
for use with apache. Do you have any details on what your test setup
is? And perhaps a really really quick guide on how to reproduce it?

-Graham

[Graham Gower]

Also, for each of the OPKG_OPT_TYPE_STRING conf options, there needs
to be a corresponding call to opkg_conf_free_string() in
opkg_conf_deinit().

Yes, this really is weird and unnatural. The conf options array should
be made static so we can just iterate the array...

[Camille Moncelier]

Graham Gower wrote:
> There is a build issue when configured with --disable-ssl-curl, but
> that is just a matter of moving around a #define.

Done ;)

> My opkg.conf contains the following:
> ----
> option ssl_ca_path /tmp/

> option ssl_engine pkcs11
This option is only usefull when you have defined an crypto engine
in openssl (I use a smartcard, for testing purpose you can omit
ssl_engine)

> option ssl_key_type ENG
> option ssl_cert_type PEM

Here ENG stands for engine, better select PEM and provide a certificate.

Anyway, ssl stuff is meant to be used with https with client authentication
Currently if you provide a repository using https which require user
authentication you can provide the client cert and key using

ssl_key_type, ssl_key, ssl_cert_type and ssl_cert.

If you're only using https ssl_ca_{file,path} might help :-)

> Sorry, I've not played with ssl certs in the past, beyond self signing
> for use with apache. Do you have any details on what your test setup
> is? And perhaps a really really quick guide on how to reproduce it?

My fault, I should have provided more informations than just a patch.
I've made more tests ( multiple servers) and a single curl handle
doesn't seems to cause harm.

So if you just want to test the single curl handle part of the
patch you can omit all the new options introduced. :P

A new patch is attached. What do you think ? If there isn't objections
I would like to commit it.

[Graham Gower]

2009/10/31 Camille Moncelier <monc...@devlife.org>:

> A new patch is attached. What do you think ? If there isn't objections
> I would like to commit it.
>

The original problems I raised have not been fixed. In fact there are
now more build problems.

$ ./autogen.sh --disable-ssl-curl && make
...
gcc -DHAVE_CONFIG_H -I. -Wall -Werror -DHOST_CPU_STR=\"i686\"
-DBUILD_CPU=i686 -DLIBDIR=\"/usr/local/lib\" -DOPKGLIBDIR=\"/usr/lib\"
-DOPKGETCDIR=\"/etc\" -DDATADIR=\"/usr/local/share\" -I.. -g -O2 -MT
opkg_download.lo -MD -MP -MF .deps/opkg_download.Tpo -c
opkg_download.c -fPIC -DPIC -o .libs/opkg_download.o
cc1: warnings being treated as errors
opkg_download.c: In function ‘opkg_curl_init’:
opkg_download.c:538: warning: implicit declaration of function ‘openssl_init’
opkg_download.c:534: warning: unused variable ‘res’
make[2]: *** [opkg_download.lo] Error 1
make[2]: Leaving directory `/home/grg/opkg/code/svn/libopkg'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/grg/opkg/code/svn/libopkg'
make: *** [all-recursive] Error 1

$ ./autogen.sh && make
...
/bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I.
-Wall -Werror -DHOST_CPU_STR=\"i686\" -DBUILD_CPU=i686
-DLIBDIR=\"/usr/local/lib\" -DOPKGLIBDIR=\"/usr/lib\"
-DOPKGETCDIR=\"/etc\" -DDATADIR=\"/usr/local/share\" -I.. -g -O2
-MT libopkg.lo -MD -MP -MF .deps/libopkg.Tpo -c -o libopkg.lo
libopkg.c
gcc -DHAVE_CONFIG_H -I. -Wall -Werror -DHOST_CPU_STR=\"i686\"
-DBUILD_CPU=i686 -DLIBDIR=\"/usr/local/lib\" -DOPKGLIBDIR=\"/usr/lib\"
-DOPKGETCDIR=\"/etc\" -DDATADIR=\"/usr/local/share\" -I.. -g -O2 -MT
libopkg.lo -MD -MP -MF .deps/libopkg.Tpo -c libopkg.c -fPIC -DPIC -o
.libs/libopkg.o
cc1: warnings being treated as errors
opkg_download.h:44: warning: ‘opkg_curl_cleanup’ declared ‘static’ but
never defined
opkg_download.h:45: warning: ‘opkg_curl_init’ declared ‘static’ but
never defined
make[2]: *** [libopkg.lo] Error 1
make[2]: Leaving directory `/home/grg/opkg/code/svn/libopkg'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/grg/opkg/code/svn/libopkg'
make: *** [all-recursive] Error 1

The attached patch is based on your first one, but fixes the problems
I saw. It would be nice if you could document the use case for
authenticating an https server's certificate - this is bound to be
common.

-Graham

[Camille Moncelier]

> The original problems I raised have not been fixed. In fact there are
> now more build problems.
>

I seriously don't know what happened when I posted this patch !
Shame on me :/ I seriously should clean my home directory
or something. Anyways, this should be the right patch.

- I removed the CURLcode res variable, (spatch is such a nice tool)

- Made some improvement on OpenSSL detection.

- Added AC_SYS_LARGEFILE in configure.as since it's needed
when gpgme is compiled with large file support

> It would be nice if you could document the use case for
> authenticating an https server's certificate - this is bound to be
> common.

Yes, perhaps we should start a documentation folder inside opkg
and also while documenting why not adding the proposed manpages
here ( http://groups.google.com/group/opkg-devel/files ) ?

Either we add them or rewrite them using another tool (txt2tags
for example) which is ihmo easier to maintain since groff/troff/nroff
syntax not the most readable syntax in the world :-)

--
Camille Moncelier
http://devlife.org/

[Graham Gower]

2009/11/2 Camille Moncelier <monc...@devlife.org>:

>> The original problems I raised have not been fixed. In fact there are
>> now more build problems.
>>
> I seriously don't know what happened when I posted this patch !
> Shame on me :/ I seriously should clean my home directory
> or something. Anyways, this should be the right patch.
>

I like this patch :) Passes my basic testing.

>
>> It would be nice if you could document the use case for
>> authenticating an https server's certificate - this is bound to be
>> common.
>
> Yes, perhaps we should start a documentation folder inside opkg
> and also while documenting why not adding the proposed manpages
> here ( http://groups.google.com/group/opkg-devel/files ) ?

Both sound like a good ideas.

>
> Either we add them or rewrite them using another tool (txt2tags
> for example) which is ihmo easier to maintain since groff/troff/nroff
> syntax not the most readable syntax in the world :-)

I think that requiring an additional dependency for manual page
installation is undesirable. Especially when there are already
existing manual pages written in the standard document markup
language.

Is there a reason that these manual pages have not yet been added?

-Graham

[wfai...@googlemail.com]

Am Dienstag, 3. November 2009 00:19:05 UTC+1 schrieb Graham Gower:
> Is there a reason that these manual pages have not yet been added?
>
> -Graham

Hi any chance to get the manual?

[Paul Barker]

Sorry you'll have to give a little more context than that, the email
you're quoting is from 2009.

Thanks,

--
Paul Barker

Email: pa...@paulbarker.me.uk
http://www.paulbarker.me.uk

[wfai...@googlemail.com]

Well I am trying to get opkg use ssl certificates when downloading packages. I found this thread talking about setting up opkg to use ssl certificate and curl. In the end of the conversation the question stays unanswered where the documentation should be published. So my questions are: Is there a documentation and/or where can I find it?

Btw. thank you for responding to something that old. ^^

[Paul Barker]

On 11 March 2014 15:40, <wfai...@googlemail.com> wrote:
> Well I am trying to get opkg use ssl certificates when downloading packages. I found this thread talking about setting up opkg to use ssl certificate and curl. In the end of the conversation the question stays unanswered where the documentation should be published. So my questions are: Is there a documentation and/or where can I find it?
>
> Btw. thank you for responding to something that old. ^^
>

Ah ok. I've re-read the old thread and it means very little to me. I
don't think there's currently any documentation for the opkg config
options.

What version of opkg are you using? What configure options did you use
when building? And what options did you put in the opkg config files
(usually /etc/opkg/*.conf)? I'll see if I can make some sense of what
is going on.

[wfai...@googlemail.com]

> What version of opkg are you using? What configure options did you use
>
> when building? And what options did you put in the opkg config files
>
> (usually /etc/opkg/*.conf)? I'll see if I can make some sense of what
>
> is going on.

I am using these options in my opkg config:

option ssl_ca_path /home/root
option ssl_cert /home/root/root.pem
option ssl_cert_type PEM
option ssl_key /home/root/client.pem
option ssl_key_type PEM

And I am getting this Error when executing "opkg update":

Collected errors:
* opkg_download: Failed to download https://xxx.com/lng-ag300h/wf-test/all/Packages.gz: Problem with the local SSL certificate.
* opkg_download: Failed to download https://xxx.com/lng-ag300h/wf-test/mips32-nf/Packages.gz: Problem with the local SSL certificate.
* opkg_download: Failed to download https://xxx.com/lng-ag300h/wf-test/buffalo_wzr_hp_ag300h/Packages.gz: Problem with the local SSL certificate.

also to debug I tried to use curl directly to grep Packages.gz:

curl https://xxx.com/lng-ag300h/wf-test/mips32-nf/Packages.gz --cacert /home/root/client.pem --capath /home/root > Packages.gz
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.

When I use the extra "-k" option it will download it correctly.

Could it be that I don't use a bundle as specified in http://curl.haxx.se/docs/sslcerts.html ?

Also do I need to have the CA on the system that I execute "opkg update" on?

Thanks

[wfai...@googlemail.com]

Also I used the options "--enable-curl --enable-ssl" when building opkg.

[Paul Barker]

On 12 March 2014 07:37, <wfai...@googlemail.com> wrote:
>> What version of opkg are you using? What configure options did you use
>>
>> when building? And what options did you put in the opkg config files
>>
>> (usually /etc/opkg/*.conf)? I'll see if I can make some sense of what
>>
>> is going on.
>
> I am using these options in my opkg config:
>
> option ssl_ca_path /home/root
> option ssl_cert /home/root/root.pem
> option ssl_cert_type PEM
> option ssl_key /home/root/client.pem
> option ssl_key_type PEM
>
> And I am getting this Error when executing "opkg update":
>
> Collected errors:
> * opkg_download: Failed to download https://xxx.com/lng-ag300h/wf-test/all/Packages.gz: Problem with the local SSL certificate.
> * opkg_download: Failed to download https://xxx.com/lng-ag300h/wf-test/mips32-nf/Packages.gz: Problem with the local SSL certificate.
> * opkg_download: Failed to download https://xxx.com/lng-ag300h/wf-test/buffalo_wzr_hp_ag300h/Packages.gz: Problem with the local SSL certificate.
>
> also to debug I tried to use curl directly to grep Packages.gz:
>
> curl https://xxx.com/lng-ag300h/wf-test/mips32-nf/Packages.gz --cacert /home/root/client.pem --capath /home/root > Packages.gz
> curl: (60) SSL certificate problem: unable to get local issuer certificate

This is way beyond my understanding of how SSL works. The best advice
I can give is to get it working with curl on the command line and then
ask again for help if you can't translate your curl command into the
correct opkg options. You may be able to find solutions online for how
to do this with curl or you may need to ask on their mailing list
(looks like it is http://cool.haxx.se/mailman/listinfo/curl-users).

Sorry I can't be more helpful.

[Camille Moncelier]

On Wed, Mar 12, 2014 at 2:19 PM, Paul Barker <pa...@paulbarker.me.uk> wrote:

ᐧ

On 12 March 2014 07:37,  <wfai...@googlemail.com> wrote:
>> What version of opkg are you using? What configure options did you use
>>
>> when building? And what options did you put in the opkg config files
>>
>> (usually /etc/opkg/*.conf)? I'll see if I can make some sense of what
>>
>> is going on.
>
> I am using these options in my opkg config:
>
> option ssl_ca_path  /home/root
> option ssl_cert  /home/root/root.pem
> option ssl_cert_type PEM
> option ssl_key   /home/root/client.pem
> option ssl_key_type PEM
>
> And I am getting this Error when executing "opkg update":
>
> Collected errors:
>  * opkg_download: Failed to download https://xxx.com/lng-ag300h/wf-test/all/Packages.gz: Problem with the local SSL certificate.
>  * opkg_download: Failed to download https://xxx.com/lng-ag300h/wf-test/mips32-nf/Packages.gz: Problem with the local SSL certificate.
>  * opkg_download: Failed to download https://xxx.com/lng-ag300h/wf-test/buffalo_wzr_hp_ag300h/Packages.gz: Problem with the local SSL certificate.
>
> also to debug I tried to use curl directly to grep Packages.gz:
>
> curl https://xxx.com/lng-ag300h/wf-test/mips32-nf/Packages.gz --cacert /home/root/client.pem --

​​

capath /home/root > Packages.gz
> curl: (60) SSL certificate problem: unable to get local issuer certificate

Hi, ​Just​ popping up (I'm not working on opkg anymore) but I recall the 

​

capath should contains hashed symlinks (using openssl's c_rehash tool) Maybe that's the reason ?

Kind regards,

CM.

--

Camille Moncelier 
+33 (0)6 33 37 88 82 / cam...@moncelier.fr

 

[Stéphane Massot]

Hi,

I'm also trying to set up an opkg repository that uses HTTPS (and basic auth).

Here is my opkg configuration file :

src/gz myrepo https://foo:boo@myrepo_IP
option ssl_ca_path /etc/ssl/certs
option ssl_cert /etc/ssl/certs/myrepo.pem
option ssl_cert_type PEM
option ssl_dont_verify_peer

root@my_target:~# opkg --noaction update
Downloading https://foo:boo@myrepo_IP/Packages.gz.
Collected errors:
* opkg_download: Failed to download https://foo:boo@myrepo_IP/Packages.gz: Problem with the local SSL certificate.

When tryin directly with CURL, I get the repository's main page using the command :

curl --cacert /etc/ssl/certs/myrepo.pem https://foo:boo@myrepo_IP

Le mercredi 12 mars 2014 15:09:46 UTC+1, Camille Moncelier a écrit :

> Hi, ​Just​ popping up (I'm not working on opkg anymore) but I recall the 
> ​capath should contains hashed symlinks (using openssl's c_rehash tool) Maybe that's the reason ?
>

I tried that (root@my-target:/etc/ssl/certs# c_rehash .), but it didn't solve my problem.

Any idea ?

Thanks.