[PATCH 0/1] compress kernels with minigzip

Andrew Boie

unread,

Oct 22, 2013, 5:10:02 PM10/22/13

to

This patch is for building kernels on x86-based Android devices.
The binary patches created by its OTA system work much better if
the deflate() algorithm inside zlib is used, rather than the
older version inside gzip.

This patch was committed against the master branch of Linus'
tree.

Andrew Boie (1):
x86: boot: support minigzip bzImage compression

arch/x86/Kconfig | 1 +
arch/x86/boot/compressed/Makefile | 3 +++
arch/x86/boot/compressed/misc.c | 2 +-
init/Kconfig | 18 +++++++++++++++++-
scripts/Makefile.lib | 7 +++++++
5 files changed, 29 insertions(+), 2 deletions(-)

--
1.8.3.2

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Andrew Boie

unread,

Oct 22, 2013, 5:10:03 PM10/22/13

to

Android OTA system computes very efficient diffs of compressed files
if the deflate() algorithm it has access to is the same version as
used to create the original file. Here we add support for compressing
the kernel with 'minigzip' which uses the deflate() inside zlib.
This is much better than using 'gzip' as that tool has a very old
version of deflate() inside the gzip codebase instead of linking against
zlib.

Signed-off-by: Andrew Boie <andrew...@intel.com>
---

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index f67e839..aa91cef 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -62,6 +62,7 @@ config X86
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_DMA_API_DEBUG
select HAVE_KERNEL_GZIP
+ select HAVE_KERNEL_MINIGZIP
select HAVE_KERNEL_BZIP2
select HAVE_KERNEL_LZMA
select HAVE_KERNEL_XZ
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index dcd90df..f000791 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -56,6 +56,8 @@ vmlinux.bin.all-$(CONFIG_X86_NEED_RELOCS) += $(obj)/vmlinux.relocs

$(obj)/vmlinux.bin.gz: $(vmlinux.bin.all-y) FORCE
$(call if_changed,gzip)
+$(obj)/vmlinux.bin.mgz: $(vmlinux.bin.all-y) FORCE
+ $(call if_changed,minigzip)
$(obj)/vmlinux.bin.bz2: $(vmlinux.bin.all-y) FORCE
$(call if_changed,bzip2)
$(obj)/vmlinux.bin.lzma: $(vmlinux.bin.all-y) FORCE
@@ -68,6 +70,7 @@ $(obj)/vmlinux.bin.lz4: $(vmlinux.bin.all-y) FORCE
$(call if_changed,lz4)

suffix-$(CONFIG_KERNEL_GZIP) := gz
+suffix-$(CONFIG_KERNEL_MINIGZIP):= mgz
suffix-$(CONFIG_KERNEL_BZIP2) := bz2
suffix-$(CONFIG_KERNEL_LZMA) := lzma
suffix-$(CONFIG_KERNEL_XZ) := xz
diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c
index 434f077..4e55d32 100644
--- a/arch/x86/boot/compressed/misc.c
+++ b/arch/x86/boot/compressed/misc.c
@@ -125,7 +125,7 @@ static char *vidmem;
static int vidport;
static int lines, cols;

-#ifdef CONFIG_KERNEL_GZIP
+#if defined(CONFIG_KERNEL_GZIP) || defined(CONFIG_KERNEL_MINIGZIP)
#include "../../../../lib/decompress_inflate.c"
#endif

diff --git a/init/Kconfig b/init/Kconfig
index 3ecd8a1..818f225 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -100,6 +100,9 @@ config LOCALVERSION_AUTO
config HAVE_KERNEL_GZIP
bool

+config HAVE_KERNEL_MINIGZIP
+ bool
+
config HAVE_KERNEL_BZIP2
bool

@@ -118,7 +121,7 @@ config HAVE_KERNEL_LZ4
choice
prompt "Kernel compression mode"
default KERNEL_GZIP
- depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4
+ depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_MINIGZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4
help
The linux kernel is a kind of self-extracting executable.
Several compression algorithms are available, which differ
@@ -144,6 +147,19 @@ config KERNEL_GZIP
The old and tried gzip compression. It provides a good balance
between compression ratio and decompression speed.

+config KERNEL_MINIGZIP
+ bool "Minigzip"
+ depends on HAVE_KERNEL_MINIGZIP
+ help
+ Use minigzip to compress the bzImage. This is very similar to gzip
+ but uses the zlib library to compress, rather than the very old version
+ of zlib inside the gzip codebase. This is used for Android kernels
+ so that the same version of the deflate() algorithm is used when
+ building the kernel and constructing diffs with OTA applypatch, which
+ uncompresses sections of files that it detects are gzipped before computing
+ the diffs. If the versions of deflate() are out of alignment the binary
+ diffs tend to be very large.
+
config KERNEL_BZIP2
bool "Bzip2"
depends on HAVE_KERNEL_BZIP2
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 49392ec..deb1bb8 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -240,6 +240,13 @@ quiet_cmd_gzip = GZIP $@
cmd_gzip = (cat $(filter-out FORCE,$^) | gzip -n -f -9 > $@) || \
(rm -f $@ ; false)

+# Minigzip
+# ---------------------------------------------------------------------------
+
+quiet_cmd_minigzip = MINGZIP $@
+cmd_minigzip = (cat $(filter-out FORCE,$^) | minigzip -c -9 > $@) || \
+ (rm -f $@ ; false)
+
# DTC
# ---------------------------------------------------------------------------

H. Peter Anvin

unread,

Oct 22, 2013, 8:30:02 PM10/22/13

to

Wouldn't it be better to fix gzip than hacking around this in the kernel?

Sent from my mobile phone. Please excuse brevity and lack of formatting.

Guenter Roeck

unread,

Oct 22, 2013, 11:00:02 PM10/22/13

to

On 10/22/2013 05:26 PM, H. Peter Anvin wrote:
> Wouldn't it be better to fix gzip than hacking around this in the kernel?
>

Or just change the build system to have /bin/gzip point to minigzip if so
desired. I have done the same to replace it with pigz.
Debian/Ubuntu provides the update-alternatives command for that
purpose. If that is not available, just hard-link it.

In general, I don't think it would be a good idea to add tool variant
dependencies like this one to the kernel configuration.

Guenter

Michal Marek

unread,

Oct 23, 2013, 6:00:02 AM10/23/13

to

On 23.10.2013 04:53, Guenter Roeck wrote:
> On 10/22/2013 05:26 PM, H. Peter Anvin wrote:
>> Wouldn't it be better to fix gzip than hacking around this in the kernel?
>>
> Or just change the build system to have /bin/gzip point to minigzip if so
> desired. I have done the same to replace it with pigz.

Yes. And if the Makefiles must be patched, then I'd suggest adding a
$(GZIP) variable to the main Makefile and using that:

make GZIP=pigz bzImage
make GZIP=minigzip bzImage

Michal

Boie, Andrew P

unread,

Oct 23, 2013, 12:50:02 PM10/23/13

to

> From: Guenter Roeck [li...@roeck-us.net]

> Or just change the build system to have /bin/gzip point to minigzip if so
> desired. I have done the same to replace it with pigz.
> Debian/Ubuntu provides the update-alternatives command for that
> purpose. If that is not available, just hard-link it.

Android minigzip and gzip unfortunately don't have the same command line syntax. Before I wrote this patch I had a horrible hack in place involving a shell script named "gzip" that called minigzip internally.

Andrew--

Guenter Roeck

unread,

Oct 23, 2013, 1:00:02 PM10/23/13

to

On Wed, Oct 23, 2013 at 04:47:38PM +0000, Boie, Andrew P wrote:
> > From: Guenter Roeck [li...@roeck-us.net]
> > Or just change the build system to have /bin/gzip point to minigzip if so
> > desired. I have done the same to replace it with pigz.
> > Debian/Ubuntu provides the update-alternatives command for that
> > purpose. If that is not available, just hard-link it.
>
> Android minigzip and gzip unfortunately don't have the same command line syntax. Before I wrote this patch I had a horrible hack in place involving a shell script named "gzip" that called minigzip internally.
>

Why is that a horrible hack ? I think it is better than clogging
kernel makefiles with that kind of tools detail. Actually, you have
(at least) three other options: fix/update gzip, update minigzip
to match the gzip cli, or use a script to map one to the other.

Guenter