opkg-0.3.2 on DD-WRT problems

[Florin]

I've tried to compile opkg-0.3.2 for a DD-WRT router with ARM cortex a9 (I already have a running native gcc on it with all dev tools working perfectly).

Of course I had to edit libopkg/opkg_conf.h in order to specify the location for info dir to /opt/lib/opkg/info and status file as /opt/lib/opkg/status, as the original opkg-0.1.8 provided for DD-WRT uses.

But I have two problems:

1) opkg list-installed does not show the installed packages even if they really are listed in the status file and info dir. Tried to specified these as options in an opkg.conf file, the conf file gets read but the same behaviour. Is there a total incompatibility between the two versions (0.1.8 and 0.3.2) status files??

2) I've also tried to install some ipk files with another compilation of opkg where I set the location of info dir and status files on /opt/usr/share/opkg (for not to break the original status file and info), and this does not work neither, I get the following messages:

Collected errors:
 * open_outer: Failed to open package './libopkg-dev_0.3.2-1_armv7soft.ipk': Unrecognized archive format
 * pkg_extract_control_file_to_stream: Failed to extract control.tar.gz from package './libopkg-dev_0.3.2-1_armv7soft.ipk'.
 * pkg_init_from_file: Failed to extract control file from ./libopkg-dev_0.3.2-1_armv7soft.ipk

but the ipk files are ok, tar compressed, they can be installed with the original opkg with no trouble.

What am I missing? It shouldn't be a problem with my libarchive, the libarchive binaries can decompress the files.

Thanks,

Florin

[Alejandro del Castillo]

On 07/01/2016 05:59 AM, Florin wrote:
> I've tried to compile opkg-0.3.2 for a DD-WRT router with ARM cortex a9 (I
> already have a running native gcc on it with all dev tools working perfectly).

That's great to hear!, OpenWRT is using a pretty old version of opkg....would be
great to see it using 0.3.x

> Of course I had to edit libopkg/opkg_conf.h in order to specify the location for
> info dir to /opt/lib/opkg/info and status file as /opt/lib/opkg/status, as the
> original opkg-0.1.8 provided for DD-WRT uses.
> But I have two problems:
>
> 1) opkg list-installed does not show the installed packages even if they really
> are listed in the status file and info dir. Tried to specified these as options
> in an opkg.conf file, the conf file gets read but the same behaviour. Is there a
> total incompatibility between the two versions (0.1.8 and 0.3.2) status files??

Not that I know off. I played a bit with this and was able to correctly read a
status file created with 0.3.2 with opkg 0.1.8. Might be worth digging deeper on
this one.

> 2) I've also tried to install some ipk files with another compilation of opkg
> where I set the location of info dir and status files on /opt/usr/share/opkg
> (for not to break the original status file and info), and this does not work
> neither, I get the following messages:
>
> Collected errors:
> * open_outer: Failed to open package './libopkg-dev_0.3.2-1_armv7soft.ipk':
> Unrecognized archive format
> * pkg_extract_control_file_to_stream: Failed to extract control.tar.gz from
> package './libopkg-dev_0.3.2-1_armv7soft.ipk'.
> * pkg_init_from_file: Failed to extract control file from
> ./libopkg-dev_0.3.2-1_armv7soft.ipk
>
> but the ipk files are ok, tar compressed, they can be installed with the
> original opkg with no trouble.
>
> What am I missing? It shouldn't be a problem with my libarchive, the libarchive
> binaries can decompress the files.

Which tool (and which version) did you use to create the ipks? Did you use
opkg-build?

I tested ipks built via the latest version of opkg-build on opkg 0.1.8 and found
that it doesn't work since opkg 0.1.8 is expecting the control file inside
control.tar.gz to be on the index as "./control", instead of "control".

Collected errors:
* pkg_init_from_file: Malformed package file /tmp/opk/b_1.0_all.opk.

Just for testing, I switched the expected "./control" for "control", which made
opkg v0.1.8 correctly install new ipks:

diff --git a/libopkg/pkg_extract.c b/libopkg/pkg_extract.c
index b7d924c..5f8a861 100644
--- a/libopkg/pkg_extract.c
+++ b/libopkg/pkg_extract.c
@@ -29,7 +29,7 @@ pkg_extract_control_file_to_stream(pkg_t *pkg, FILE *stream)
deb_extract(pkg->local_filename, stream,
extract_control_tar_gz
| extract_to_stream,
- NULL, "./control", &err);
+ NULL, "control", &err);

return err;
}

I am less worried about having new ipks work with old opkg than the other way
around....that needs to get fixed.

--
Cheers,

Alejandro

[Florin Gherendi]

Hi again,

I've played around a little bit with 0.3.2 sources. Finally noticed that
if architecture is not set in the conf file, it will not list any
package, whether installed or in repository. Ok, I don't mind.
It now can list and uninstall packages on DD-WRT with minor tweaks.
I've ported some patches from the OpenWRT opkg, made one more patch for
changing some default dirs that were read only on DD-WRT; all the useful
patches worked. The only one I still have to work on is the support for
gzipped package list file, but it is not urgent since it could be used
without it. I will do it when I get more familiar with libarchive
programming, never used it before.

And now the bigger problem: OpenWRT packages cannot be installed because
the outer ipk archive for all of them, even the ones recently published,
is in tar.gz format.
I saw in the opkg_archive.c source that the open_outer() function only
supports uncompressed ar...

It seems support for tar.gz packages has been dropped somewhere in opkg
history; OpenWRT is using the 9c97d5ecd795709c8584e972bfdf3aee3a5b846d
revision of 0.1.8 from the Git repository, which was still supporting
it. Is there any undergoing work to bring back tar.gz package support in
0.3.x? I could start to study the problem but it might take me some time
to start it from scratch.

Cheers,
Florin

[Florin Gherendi]

On 07/05/2016 10:25 AM, Florin Gherendi wrote:

> And now the bigger problem: OpenWRT packages cannot be installed because
> the outer ipk archive for all of them, even the ones recently published,
> is in tar.gz format.
> I saw in the opkg_archive.c source that the open_outer() function only
> supports uncompressed ar...
>
> It seems support for tar.gz packages has been dropped somewhere in opkg
> history; OpenWRT is using the 9c97d5ecd795709c8584e972bfdf3aee3a5b846d
> revision of 0.1.8 from the Git repository, which was still supporting
> it. Is there any undergoing work to bring back tar.gz package support in
> 0.3.x? I could start to study the problem but it might take me some time
> to start it from scratch.
>

I didn't think it would take me so short! and now it works like a charm!

So here are my patches; the "500"-one addresses the openwrt packages
problem, the others (12 - 260) I've ported from the openwrt source tree
and are pretty useful; and the "555" is a default config files patch
that suits my DD-WRT (I've changed the directories from /opt/lib to
/opt/var/lib and used copies in order not to disturb my local packages
installation). Hope they will be useful for somebody! :)

I will still try to change the implementation for repository packages
file list in order to support gzipped locally stored file (like the
patch 070 from OpenWRT version of opkg does); this would be useful for
OpenWRT/DD-WRT people who want to upgrade opkg without decompressing the
package list; but I will try a less radical implementation than the
OpenWRT one (who switches entirely to gzipped repository cache).

Still have to figure out some way to change the free space checking to
/opt only for DD-WRT/OpenWRT without totally disabling free space
checking. Using the --force-space switch for every installation can be
annoying.

Have a nice time!
Florin

[Paul Barker]

Just wanted to jump in on this one to give a bit of context and advice.

Support for things used by OpenWRT wasn't really dropped deliberately,
it was just lost accidentally as there wasn't anyone within OpenWRT
picking up and testing the newer opkg releases or interacting on this
list. Everyone gets busy and package managers aren't the sexiest bits
of something like OpenWRT to work on so that's not too much of a
surprise. Great to see some interest again though! It'd be really good
to combine efforts around opkg and improve things for all its users.

Looking at "500-tar-gz-ipk-openwrt-support.patch", I think you've taken
an overly complicated approach. You should just be able to modify the
preceding lines to add calls to archive_read_support_format_tar(outer)
and archive_read_support_filter_gzip(outer) before calling
archive_read_open_filename(...). libarchive will then examine the file
and determine which of the supported formats and filters to use. See
for example the open_compressed_file(...) function, it supports both
"raw" and "empty" formats by calling both support functions one after
the other before calling archive_read_open_filename(...). Hope that
makes sense.

I'll leave the rest to others to look at as I'm not too up-to-date with
recent changes to opkg.

Cheers,
Paul Barker

[Florin Gherendi]

On 07/06/2016 01:12 AM, Paul Barker wrote:

Thanks for the answer!

Well I hope to go on up to as usable stuff as possible.

> Looking at "500-tar-gz-ipk-openwrt-support.patch", I think you've taken
> an overly complicated approach. You should just be able to modify the
> preceding lines to add calls to archive_read_support_format_tar(outer)
> and archive_read_support_filter_gzip(outer) before calling
> archive_read_open_filename(...). libarchive will then examine the file
> and determine which of the supported formats and filters to use. See
> for example the open_compressed_file(...) function, it supports both
> "raw" and "empty" formats by calling both support functions one after
> the other before calling archive_read_open_filename(...). Hope that
> makes sense.
>

As I did not know libarchive can be so smart (my first time working with
it), I used a preventive fail-proof solution for a quick try. I will
also try to simplify, and post the patch again.

> I'll leave the rest to others to look at as I'm not too up-to-date with
> recent changes to opkg.
>

I finally might not try to add support for gzipped packages list cache,
I found out it is not really useful; opkg update solves the problem of
reading the package list, and the uncompressed list is not so big as to
create problems for anyone.
The other patches are just ported from the mainstream OpenWRT source;
they are nice in the sense that OpenWRT users will find there the "find"
command and the --size and --nocase switches they might be familiar with
(the idea is not bad for any user, finally).
The 555 patch can basically be ignored; I intend to use the --sysconfig
option in ./configure to specify the config files dir, and the lists and
status location are user configurable anyway).

Have a nice day!
Florin

[Florin Gherendi]

Hi again!

On 07/06/2016 10:43 PM, Alejandro del Castillo wrote:

>
>
> On 07/01/2016 01:15 PM, Florin wrote:
>> Thanks for the answer!
>

> np, I am really glad you have interest in getting OpenWRT working with the
> latest opkg! The 2 biggest opkg clients are OpenEmbedded/Yocto and OpenWRT.
> While OE/Yocto stays up to date with opkg, OpenWRT does not...so this is a great
> step into getting it in sync
>
It is useful for me too, and it does not hurt giving a helpful hand if I
can.

>
> I believe "opkg info" should be able to give you package information of all
> installed packages, regardless of the way they were installed (repository of
> not)....not sure if that's what you were referring to...
>
Hmm, nope. The problem is that the status file does not store the
package description and size, that info is only stored in the repository
package list file, but not extracted from the installed package. I will
give it a look, maybe I can change that; it would be a nice feature to have.

> Weird....I was not able to install ipk's generated with the latest version of
> opkg-build without the change below. Regardless, the backwards compatibility
> case is the one that we should focus on (new opkg, old ipk's).
>
I managed to get OpenWRT optware packages to install; I just needed to
add tar.gz support for the outer ipk archive.

> I'll review your patchset probably tomorrow.....thanks again for taking the
> initiative to fix all the OpenWRT incompatibilities!
>
Well, here come the final patches that make opkg-0.3.2 work on my
DD-WRT, and an opkg.conf file with directories configured like in the
standard DD-WRT installation, that makes the upgrade to work out-of-the
box without losing installed packages.

Some explanations about them:
-501-tar-gz-openwrt-support.patch is a simplified variant of
500-tar-gz-openwrt-support.patch, taking into account Paul Barker's
suggestions.

-510-nonstandard-install-prefix.patch: this enables the --prefix,
--sysconfdir and --localstatedir options to be used with configure such
as directories like /etc, /var/run and /tmp can reside on /opt or other
mountpoint. Compiling with --prefix=/opt makes it run on DD-WRT without
changing any header. This way I've sacrified
OPKG_CONF_DEFAULT_TMP_DIR_BASE, OPKG_CONF_DEFAULT_CONF_FILE_DIR and
OPKG_CONF_DEFAULT_LOCK_FILE from opkg_conf.h; they no longer have effect
but I did not remove them from the header. If the patch would enter the
main source, I could suggest removing all those defines from
opkg_conf.h and let them in the duty of "configure"; I can remove them
all if you think it's useful. I think it's not a problem changing the
Makefile.am, autoreconf must be run anyway before compiling.

-opkg.conf: for now I let it configure the cache,lists, info dirs and
status file; but on my DD-WRT it would fail if I remove those options
from opkg.conf (since I did not drop the OPKG_CONF_DEFAULT equivalents).
The option overlay_root /opt is only to prevent space checking on "/"
which would fail.

With all these patches, it compiles and works out-of-the-box.
I hope what I did is useful.

[Jo-Philipp Wich]

Hi Florin,

great to see someone spending work on it :)

Disclaimer: I am the former maintainer of opkg on OpenWrt.

>> Support for things used by OpenWRT wasn't really dropped deliberately,
>> it was just lost accidentally as there wasn't anyone within OpenWRT
>> picking up and testing the newer opkg releases or interacting on this
>> list.

The reason that OpenWrt stopped following upstream is that opkg became
totally unusable on the targets OpenWrt typically supports, with the
introduction of libarchive the opkg + dependencies binary size grew way
too big to be useful anymore.

I did various attempts to do a tiny libarchive build, maybe linked
statically but deemed it to be too much work so I put that project aside
again.

Iiic I also raised my concerns before libarchive got introduced but we
never reached any conclusion there.

>> [...]

> I finally might not try to add support for gzipped packages list cache,
> I found out it is not really useful; opkg update solves the problem of
> reading the package list, and the uncompressed list is not so big as to
> create problems for anyone.

On OpenWrt the main repository index is around 1.2MB, enough to trigger
the OOM killer on many lower spec devices when an "opkg update" is
invoked, placing the uncompressed file into tmpfs.

Maybe the platforms commonly targeted by OE are big enough that a
megabyte of wasted RAM does not matter but on many OpenWrt supported
devices it makes the difference between being able to use opkg or not.


Regards,
Jo-Philipp

[Alejandro del Castillo]

On 07/07/2016 05:42 AM, Jo-Philipp Wich wrote:
> Hi Florin,
>
> great to see someone spending work on it :)
>
> Disclaimer: I am the former maintainer of opkg on OpenWrt.

Good to know you are following the list, =)

>>> Support for things used by OpenWRT wasn't really dropped deliberately,
>>> it was just lost accidentally as there wasn't anyone within OpenWRT
>>> picking up and testing the newer opkg releases or interacting on this
>>> list.
>
> The reason that OpenWrt stopped following upstream is that opkg became
> totally unusable on the targets OpenWrt typically supports, with the
> introduction of libarchive the opkg + dependencies binary size grew way
> too big to be useful anymore.
>
> I did various attempts to do a tiny libarchive build, maybe linked
> statically but deemed it to be too much work so I put that project aside
> again.
>
> Iiic I also raised my concerns before libarchive got introduced but we
> never reached any conclusion there.

I am hoping Paul will jump with some context here, but last time we
spoke he mentioned that there might be a way forward here as much of the
libarchive size (if I recall correctly) comes from a feature not used by
opkg. The work would be to add a way for libarchive to make that
functionality optional.

I can certainly continue Paul's work to get a libarchive on the ballpark
of what's acceptable for OpenWRT.

>>> [...]
>> I finally might not try to add support for gzipped packages list cache,
>> I found out it is not really useful; opkg update solves the problem of
>> reading the package list, and the uncompressed list is not so big as to
>> create problems for anyone.
>
> On OpenWrt the main repository index is around 1.2MB, enough to trigger
> the OOM killer on many lower spec devices when an "opkg update" is
> invoked, placing the uncompressed file into tmpfs.
>
> Maybe the platforms commonly targeted by OE are big enough that a
> megabyte of wasted RAM does not matter but on many OpenWrt supported
> devices it makes the difference between being able to use opkg or not.

OE targets a lot of different platforms, I think there is value on
making opkg work on very constrained environments. Good to know that is
important for OpenWRT to keep the local packages cache compressed.

--
Cheers,

Alejandro

[Alejandro del Castillo]

Hi Florin,

My comments below...thanks again for taking the time to make the latest opkg
work with OpenWRT!


> 012-strip-trailing-conffiles-whitespace.patch

Looks good

> 050-add-case-insensitive-flag.patch

Adding case insensitive pattern matching could be dangerous since opkg let's you
have 2 packages with the same name with different capitalization.

Maybe opkg should follow debian policy here and only allow lower case packages?
opinions?

"Package names (both source and binary, see Package, Section 5.6.7) must consist
only of lower case letters (a-z), digits (0-9), plus (+) and minus (-) signs,
and periods (.). They must be at least two characters long and must start with
an alphanumeric character."

> 060-add-find-command.patch

The command currently searches both the repository cache and the installed
packages, which means that you get 2 entries if the repository have a different
version of a package installed on the system.

Maybe we should only search on the repository cache the same way apt-cache
works? Or is there value on returning both entries?

Also the man page on man/opkg.1.in needs to be updated.

> 080-suppress-blank-package-fields.patch

It's missing a #include <malloc.h> (free function), which triggers a warning.

> 100-add-force-checksum.patch

Looks good, but needs man page update on man/opkg.conf.5.in.

> 250-add-lists-dir-switch.patch

Looks good, but needs man page update on man/opkg.1.in.

> 260-add-print-package-size.patch

It returns 0 as the size for installed packages (opkg --size
list-installed).This is an opkg bug since Installed-Size is not being written to
the status file. To follow up, I created:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=9906

However, after manually adding 'Installed-Size' to the status file, opkg --size
list-installed still shows 0.

> 500-tar-gz-ipk-openwrt-support.patch

I believe Paul suggested a simpler approach

> 555_my_configs.patch

I am hesitant to change the opkg default paths...the directories set on the
patch are already configurable with 2 exceptions:

*OPKG_CONF_DEFAULT_CONF_FILE_DIR .- You can set the conffile (not dir) with the
-f options, or you can set the conf dir via the environment variable OPKG_CONF_DIR.
*OPKG_CONF_DEFAULT_TRUSTED_PATH

I would certainly take a patch to make OPKG_CONF_DEFAULT_TRUSTED_PATH
configurable via .conf.

--
Cheers,

Alejandro

[Florin Gherendi]

Hi Alejandro,

>> 050-add-case-insensitive-flag.patch
>
> Adding case insensitive pattern matching could be dangerous since opkg let's you
> have 2 packages with the same name with different capitalization.
>
> Maybe opkg should follow debian policy here and only allow lower case packages?
> opinions?
>
> "Package names (both source and binary, see Package, Section 5.6.7) must consist
> only of lower case letters (a-z), digits (0-9), plus (+) and minus (-) signs,
> and periods (.). They must be at least two characters long and must start with
> an alphanumeric character."
>

I agree with you about this one, Debian convention is a good thing. I
just ported the patch from OpenWRT since they apply it by default. But
it neither does harm, since I think packages with the same name and
different capitalization would conflict.

>> 060-add-find-command.patch
>
> The command currently searches both the repository cache and the installed
> packages, which means that you get 2 entries if the repository have a different
> version of a package installed on the system.
>
> Maybe we should only search on the repository cache the same way apt-cache
> works? Or is there value on returning both entries?
>

Hmm, it can be useful by letting you see the available versions. It is
also a ported patch, it is present by default in OpenWRT. It is not
perfect though, regexp searching sometimes misses some hits (when the
package names contain dashes). And opkg certainly could work on OpenWRT
without it.

> Also the man page on man/opkg.1.in needs to be updated.
>

I might try to do it if the patch would be introduced in the main source.

>> 080-suppress-blank-package-fields.patch
>
> It's missing a #include <malloc.h> (free function), which triggers a warning.
>

Yeah, I've eventually seen the warning. Also ported package, the include
wasn't there since the beginning... I can change that.

>> 100-add-force-checksum.patch
>
> Looks good, but needs man page update on man/opkg.conf.5.in.
>
>> 250-add-lists-dir-switch.patch
>
> Looks good, but needs man page update on man/opkg.1.in.
>

I'll try to update the man pages (not an expert in that, but it's not so
difficult).

>> 260-add-print-package-size.patch
>
> It returns 0 as the size for installed packages (opkg --size
> list-installed).This is an opkg bug since Installed-Size is not being written to
> the status file. To follow up, I created:
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=9906
>

Yes, I've noticed that. It would be even nicer if the status file would
include the package description, I was thinking to try to do it. I
understand any effort could be useful in that direction.

> However, after manually adding 'Installed-Size' to the status file, opkg --size
> list-installed still shows 0.
>

That is quite weird. I did not try to manually add the Installed-Size in
the status file, but it does show the size for the packages in the
repository ( opkg --size list ). I will take a look about it, I don't
know why the package size is read from the repository cache but not from
the status file, there must be a clue somewhere :) Maybe
opkg_config->size is just not set and opkg info just lists the ASCII
entry without interpreting it?

>> 500-tar-gz-ipk-openwrt-support.patch
>
> I believe Paul suggested a simpler approach
>

Yeah, already did it, see my later mail, the
501-tar-gz-ipk-openwrt-support.patch. Anyway, this is essential to make
it work on OpenWRT, the other issues can be solved with command line and
conf file options.

>> 555_my_configs.patch
>
> I am hesitant to change the opkg default paths...the directories set on the
> patch are already configurable with 2 exceptions:
>
> *OPKG_CONF_DEFAULT_CONF_FILE_DIR .- You can set the conffile (not dir) with the
> -f options, or you can set the conf dir via the environment variable OPKG_CONF_DIR.
> *OPKG_CONF_DEFAULT_TRUSTED_PATH
>
> I would certainly take a patch to make OPKG_CONF_DEFAULT_TRUSTED_PATH
> configurable via .conf.
>

That patch was to be ignored :) it was only the set of configurations I
used to make it work. In my later e-mail I've sent a
510-nonstandard-install-prefix.patch which links
OPKG_CONF_DEFAULT_CONF_FILE_DIR and OPKG_CONF_DEFAULT_TRUSTED_PATH to
the --sysconfdir at configure time; I think it's a better approach. In
that e-mail I've also posted an opkg.conf that works on OpenWRT for
doing a simple upgrade of opkg without losing the installed files list,
if the 500 or 501 patch is applied.

As an observation, setting the OPKG_CONF_DEFAULT_* variables in a .h
file does not particularly serve for an API, since those vars are
already defined inside the shared lib when it is compiled. I was playing
with the API - yes, I know it's deprecated - to build kindle-opkg-gui
(the app is not very mature since the beginning, but I've managed to
compile it with some modifications and make it run, I still have to make
it list something; it did with the 0.1.8 API).

Cheers!
Florin

[Alejandro del Castillo]

Hi Florin, to close the loop on the patchset state:

Wait for v2
* 060-add-find-command.patch
* 080-suppress-blank-package-fields.patch
* 100-add-force-checksum.patch
* 250-add-lists-dir-switch.patch
* 260-add-print-package-size.patch

Drop
* 050-add-case-insensitive-flag.patch
* 500-tar-gz-ipk-openwrt-support.patch
* 555_my_configs.patch

would it be too much trouble to send each patch on a separate email via git
send-email? that would make it easier to review/pull in.

>>> 050-add-case-insensitive-flag.patch
>>
>> Adding case insensitive pattern matching could be dangerous since opkg let's you
>> have 2 packages with the same name with different capitalization.
>>
>> Maybe opkg should follow debian policy here and only allow lower case packages?
>> opinions?
>>
>> "Package names (both source and binary, see Package, Section 5.6.7) must consist
>> only of lower case letters (a-z), digits (0-9), plus (+) and minus (-) signs,
>> and periods (.). They must be at least two characters long and must start with
>> an alphanumeric character."
>>
> I agree with you about this one, Debian convention is a good thing. I
> just ported the patch from OpenWRT since they apply it by default. But
> it neither does harm, since I think packages with the same name and
> different capitalization would conflict.
>

Ok, sounds good. I created a bugzilla ticket to track the changes needed to
follow the debian package convention, and we can drop this patch.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=9920

>>> 060-add-find-command.patch
>>
>> The command currently searches both the repository cache and the installed
>> packages, which means that you get 2 entries if the repository have a different
>> version of a package installed on the system.
>>
>> Maybe we should only search on the repository cache the same way apt-cache
>> works? Or is there value on returning both entries?
>>
> Hmm, it can be useful by letting you see the available versions. It is
> also a ported patch, it is present by default in OpenWRT. It is not
> perfect though, regexp searching sometimes misses some hits (when the
> package names contain dashes). And opkg certainly could work on OpenWRT
> without it.

I am OK accepting the patch if it provides value to OpenWRT (with the man page
addition)

>> Also the man page on man/opkg.1.in needs to be updated.
>>
> I might try to do it if the patch would be introduced in the main source.

>>> 080-suppress-blank-package-fields.patch
>>
>> It's missing a #include <malloc.h> (free function), which triggers a warning.
>>
> Yeah, I've eventually seen the warning. Also ported package, the include
> wasn't there since the beginning... I can change that.

Thanks, I'll wait for a v2.

>>> 100-add-force-checksum.patch
>>
>> Looks good, but needs man page update on man/opkg.conf.5.in.
>>

>>> 250-add-lists-dir-switch.patch
>>
>> Looks good, but needs man page update on man/opkg.1.in.
>>
> I'll try to update the man pages (not an expert in that, but it's not so
> difficult).

Lemme know if you struggle...I usually just copy-paste guessing a bit, then
check with a build that things look OK.

>>> 260-add-print-package-size.patch
>>
>> It returns 0 as the size for installed packages (opkg --size
>> list-installed).This is an opkg bug since Installed-Size is not being written to
>> the status file. To follow up, I created:
>>
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=9906
>>
> Yes, I've noticed that. It would be even nicer if the status file would
> include the package description, I was thinking to try to do it. I
> understand any effort could be useful in that direction.

Yeah, that looks like an oversight since the /var/lib/opkg/lists/* files do
include Description. If you can get to it, would be great!

>> However, after manually adding 'Installed-Size' to the status file, opkg --size
>> list-installed still shows 0.
>>
> That is quite weird. I did not try to manually add the Installed-Size in
> the status file, but it does show the size for the packages in the
> repository ( opkg --size list ). I will take a look about it, I don't
> know why the package size is read from the repository cache but not from
> the status file, there must be a clue somewhere :) Maybe
> opkg_config->size is just not set and opkg info just lists the ASCII
> entry without interpreting it?

The following code works when manually adding "Installed-Size":

if (opkg_config->size) {
if (pkg->state_status == SS_INSTALLED || pkg->state_status == SS_UNPACKED)
printf(" - %lu", pkg->installed_size);
else
printf(" - %lu", pkg->size);
}

you can add it to the patch, so when opkg starts writing 'Installed-Size', the
code will work.

>>> 500-tar-gz-ipk-openwrt-support.patch
>>
>> I believe Paul suggested a simpler approach
>>
> Yeah, already did it, see my later mail, the
> 501-tar-gz-ipk-openwrt-support.patch. Anyway, this is essential to make
> it work on OpenWRT, the other issues can be solved with command line and
> conf file options.
>

For some reason I lost your email on my inbox....I will reply with my comments
on that patch on the right email thread then.

>>> 555_my_configs.patch
>>
>> I am hesitant to change the opkg default paths...the directories set on the
>> patch are already configurable with 2 exceptions:
>>
>> *OPKG_CONF_DEFAULT_CONF_FILE_DIR .- You can set the conffile (not dir) with the
>> -f options, or you can set the conf dir via the environment variable OPKG_CONF_DIR.
>> *OPKG_CONF_DEFAULT_TRUSTED_PATH
>>
>> I would certainly take a patch to make OPKG_CONF_DEFAULT_TRUSTED_PATH
>> configurable via .conf.
>>
>
> That patch was to be ignored :) it was only the set of configurations I
> used to make it work. In my later e-mail I've sent a
> 510-nonstandard-install-prefix.patch which links
> OPKG_CONF_DEFAULT_CONF_FILE_DIR and OPKG_CONF_DEFAULT_TRUSTED_PATH to
> the --sysconfdir at configure time; I think it's a better approach. In
> that e-mail I've also posted an opkg.conf that works on OpenWRT for
> doing a simple upgrade of opkg without losing the installed files list,
> if the 500 or 501 patch is applied.

Ok, I'll review that patch on the other email thread too

> As an observation, setting the OPKG_CONF_DEFAULT_* variables in a .h
> file does not particularly serve for an API, since those vars are
> already defined inside the shared lib when it is compiled. I was playing
> with the API - yes, I know it's deprecated - to build kindle-opkg-gui
> (the app is not very mature since the beginning, but I've managed to
> compile it with some modifications and make it run, I still have to make
> it list something; it did with the 0.1.8 API).

--
Cheers,

Alejandro

[Alejandro del Castillo]

For some weird reason this email disappeared on my other email address....I'll respond here.

 

Well, here come the final patches that make opkg-0.3.2 work on my
DD-WRT, and an opkg.conf file with directories configured like in the
standard DD-WRT installation, that makes the upgrade to work out-of-the
box without losing installed packages.

Some explanations about them:
-501-tar-gz-openwrt-support.patch is a simplified variant of
500-tar-gz-openwrt-support.patch, taking into account Paul Barker's
suggestions.

Looks good to me, I will pull this patch in once 0.3.3 is out.

-510-nonstandard-install-prefix.patch: this enables the --prefix,
--sysconfdir and --localstatedir options to be used with configure such
as directories like /etc, /var/run and /tmp can reside on /opt or other
mountpoint. Compiling with --prefix=/opt makes it run on DD-WRT without
changing any header. This way I've sacrified
OPKG_CONF_DEFAULT_TMP_DIR_BASE, OPKG_CONF_DEFAULT_CONF_FILE_DIR and
OPKG_CONF_DEFAULT_LOCK_FILE from opkg_conf.h; they no longer have effect
but I did not remove them from the header. If the patch would enter the
main source, I could suggest removing all those defines from
opkg_conf.h and let them in the duty of "configure"; I can remove them
all if you think it's useful. I think it's not a problem changing the
Makefile.am, autoreconf must be run anyway before compiling.

I like the approach, I think it works for sysconfdir and localstatedir. For tmp_dir_base using prefix doesn't work since the default prefix is '/usr/local'. Since there is not automake variable (that I know off..) for tmpdir, I think it would work if you hardcode the default value on the Make file, something like -DTEMPDIR="/tmp".

Also, it's probably cleaner to keep the #defines for the different dirs (lists, info, cache, etc) on opkg_conf.h, but change the #define to use configure time variables like:

#define OPKG_CONF_DEFAULT_LISTS_DIR VARDIR "lib/opkg/lists"

...

...

[Florin Gherendi]

Hi!

I'm a bit late with the answer because of some deadlines that I had
during this time.

On 07/13/2016 12:06 AM, Alejandro del Castillo wrote:
> Hi Florin, to close the loop on the patchset state:
>
> Wait for v2
> * 060-add-find-command.patch
> * 080-suppress-blank-package-fields.patch
> * 100-add-force-checksum.patch
> * 250-add-lists-dir-switch.patch
> * 260-add-print-package-size.patch
>
> Drop
> * 050-add-case-insensitive-flag.patch
> * 500-tar-gz-ipk-openwrt-support.patch
> * 555_my_configs.patch
>
> would it be too much trouble to send each patch on a separate email via git
> send-email? that would make it easier to review/pull in.
>

I've worked the v2 of the patches on the opkg-0.3.3. Still have to think
about the 260-add-print-package-size.patch, I want to find a better
approach than the original openwrt one, and I will give it another apply
order number instead of 260. when I have a form.

Since I'm doing the compilation directly on my DD-WRT router with native
compiler, I want also to re-compile git (the openwrt supplied version
does not have Perl support compiled in, therefore git send-email doesn't
work). I will send the patches once that will be done.

I have it already, I will send the v2 with man page updated.

>
>>>> 080-suppress-blank-package-fields.patch
>>>
>>> It's missing a #include <malloc.h> (free function), which triggers a warning.
>>>
>> Yeah, I've eventually seen the warning. Also ported package, the include
>> wasn't there since the beginning... I can change that.
>
> Thanks, I'll wait for a v2.
>

It was #include <stdlib.h>; I have it too now, so I can send it.

>>>> 100-add-force-checksum.patch
>>>
>>> Looks good, but needs man page update on man/opkg.conf.5.in.
>>>
>

Done too, updated both man/opkg.conf.5.in and man/opkg.1.in for v2.

>>>> 250-add-lists-dir-switch.patch
>>>
>>> Looks good, but needs man page update on man/opkg.1.in.
>>>
>> I'll try to update the man pages (not an expert in that, but it's not so
>> difficult).
>
> Lemme know if you struggle...I usually just copy-paste guessing a bit, then
> check with a build that things look OK.

It was not so hard, I did it. I've also arranged a little bit the usage
help of the opkg:
-t Specify tmp-dir.
--tmp-dir Specify tmp-dir.
-l Specify lists-dir.
--lists-dir Specify lists-dir.

looked untidy to me, I've replaced the sequence by:
-t, --tmp-dir <directory> Specify tmp-dir.
-l, --lists-dir <directory> Specify lists-dir.

Not yet done; I can insert the sequence you suggest and send it as such
if you like, but I would rather try to implement direct checking of the
package for sizes and update the status file with "Size" and
"Installed-size", and even "Description"; that would be a totally
different approach; not done yet.

>>>> 500-tar-gz-ipk-openwrt-support.patch
>>>
>>> I believe Paul suggested a simpler approach
>>>
>> Yeah, already did it, see my later mail, the
>> 501-tar-gz-ipk-openwrt-support.patch. Anyway, this is essential to make
>> it work on OpenWRT, the other issues can be solved with command line and
>> conf file options.
>>
>
> For some reason I lost your email on my inbox....I will reply with my comments
> on that patch on the right email thread then.
>

The 501-tar-gz-ipk-openwrt-support.patch was ok already; so I will
resend the v2 for opkg-0.3.3 using git.

>>>> 555_my_configs.patch
>>>
>>> I am hesitant to change the opkg default paths...the directories set on the
>>> patch are already configurable with 2 exceptions:
>>>
>>> *OPKG_CONF_DEFAULT_CONF_FILE_DIR .- You can set the conffile (not dir) with the
>>> -f options, or you can set the conf dir via the environment variable OPKG_CONF_DIR.
>>> *OPKG_CONF_DEFAULT_TRUSTED_PATH
>>>
>>> I would certainly take a patch to make OPKG_CONF_DEFAULT_TRUSTED_PATH
>>> configurable via .conf.
>>>
>>
>> That patch was to be ignored :) it was only the set of configurations I
>> used to make it work. In my later e-mail I've sent a
>> 510-nonstandard-install-prefix.patch which links
>> OPKG_CONF_DEFAULT_CONF_FILE_DIR and OPKG_CONF_DEFAULT_TRUSTED_PATH to
>> the --sysconfdir at configure time; I think it's a better approach. In
>> that e-mail I've also posted an opkg.conf that works on OpenWRT for
>> doing a simple upgrade of opkg without losing the installed files list,
>> if the 500 or 501 patch is applied.
>
> Ok, I'll review that patch on the other email thread too
>

In the other mail there was the 510-nonstandard-install-prefix.patch,
which sets the default paths via "configure" options. I'm working to
tidy up that patch and I'll send it later (I really think it would be
useful)
In that e-mail I've also sent an opkg.conf that would do good defaults
for DD-WRT upgrade of opkg without changing the default dirs in the
source code.

I'll be back soon with git e-mails. Should I post them to the same
mailing list? or to you directly?

[Alejandro del Castillo]

Thanks so much, it is a lot easier to review/pull individual patches.
Posting the new patches to the mailing list directly in a v2 patchset
would be great, that way other opkg developers can comment too.

--
Cheers,

Alejandro

[Alejandro del Castillo]

Hi Florin,

Just wanted to check back with you regarding the status of the OpenWRT
patchset. I am planning on cutting a new opkg release by mid December,
it would be great if those patches could make it to 0.3.4!

[Florin Gherendi]

Hi Alejandro,

Besides what I've already committed in git, I think I will be able to
work on them until the beginning of December. I will have to check again
what is still left to do, it was a bit of a pause.
Still a bit busy for the moment but I will find time.

[Alejandro del Castillo]

On 10/04/2016 02:40 AM, Florin Gherendi wrote:
> Hi Alejandro,
>
> Besides what I've already committed in git, I think I will be able to
> work on them until the beginning of December. I will have to check again
> what is still left to do, it was a bit of a pause.
> Still a bit busy for the moment but I will find time.

Sounds good, thanks for keeping it on the radar still. I am going
through the bug list scheduled for 0.3.4, I believe the ticket to show
Installed-Size is on it. I'll try to get to it, so that way your
260-add-pinrt-package-size.patch will work.

--
Cheers,

Alejandro