zlib compressed data in firmware

[indy arm]

Hello,

zlib is used in the firmware, see strings around FF05B234 in 550d 1.0.8 & 1.0.9, just after the SVG lib.
by zlib I mean: http://www.faqs.org/rfcs/rfc1950.html

so I tried to identify and decode zlib compressed data.

1. identification method used (to be checked)

I first use 2 bytes signatures 0x7801, 0x789c and 0x78da, as explained here:
http://hexblog.com/files/recon%202010%20Skochinsky.pdf, slide 6
if these 2 bytes are found in the file, I then compute the floating frequency (how many different byte values in a 64 byte sliding window) and let growing the region if frequency become < 40.
see http://www.cryptool.org/download/CrypToolPresentation-en.pdf, slide 65

for 550d 1.0.8, I have:

$ ./float_freq.exe  ROM0.bin
[ 0x0480818 -> 0x0494a35 ], len=82461/0x1421d, max=53. 7801544d2e78a34e4ee0
[ 0x0494a61 -> 0x04ade34 ], len=103379/0x193d3, max=48. 789cf352a840e50054d6 <-- false positive as size is different than 1.0.9 ?
[ 0x04dc69e -> 0x050eefc ], len=206942/0x3285e, max=59. 78dac345f863acf8d742 <-- false positive ? size is really diff then 1.0.9
[ 0x05ab22c -> 0x05c5587 ], len=107355/0x1a35b, max=54. 78014966c3dc0d9860fd
[ 0x05d0811 -> 0x062b51f ], len=371982/0x5ad0e, max=57. 7801cc897f81cefd0376
[ 0x0641b2c -> 0x0646325 ], len=18425/0x47f9, max=51. 780166818c3a0522cecb
[ 0x0650bf4 -> 0x0654c55 ], len=16481/0x4061, max=54. 789c0cf716c89946b640

the file start at RAM offset 0xff010000, so first region is at 0xff490818 with approx len of 82461 bytes. 10 first bytes in hexa are 7801544d2e78a34e4ee0

for 1.0.9 :
$ ./float_freq.exe  550d/1.0.9/ROM0.BIN
[ 0x048084c -> 0x0494a69 ], len=82461/0x1421d, max=53. 7801544d2e78a34e4ee0
[ 0x0494a95 -> 0x04ade64 ], len=103375/0x193cf, max=48. 789cf352a840e50054d6
[ 0x04dc6ce -> 0x0513654 ], len=225158/0x36f86, max=59. 78dac345f863acf8d742
[ 0x05ab22c -> 0x05c5587 ], len=107355/0x1a35b, max=54. 78014966c3dc0d9860fd
[ 0x05d0811 -> 0x062b51f ], len=371982/0x5ad0e, max=57. 7801cc897f81cefd0376
[ 0x0641b2c -> 0x0646325 ], len=18425/0x47f9, max=51. 780166818c3a0522cecb
[ 0x0650bf4 -> 0x0654c55 ], len=16481/0x4061, max=54. 789c0cf716c89946b640

in 60d 1.0.8:

$ ./float_freq.exe  ROM0.BIN
wrong matches :
[ 0x00263dc -> 0x004662c ], len=131664/0x20250, max=41. 78019f153840bd18b120
[ 0x01e9414 -> 0x01eca20 ], len=13836/0x360c, max=42. 789c000030970000e0c8
[ 0x02ebc80 -> 0x02fafcc ], len=62284/0xf34c, max=41. 78019de5bc219de5c031
[ 0x02ff468 -> 0x02ffa3c ], len=1492/0x5d4, max=43. 780184e5183094e50500
possible matches :
[ 0x0521b10 -> 0x0524308 ], len=10232/0x27f8, max=43. 780190e5210a00eb0000
[ 0x056ac24 -> 0x057ee41 ], len=82461/0x1421d, max=53. 7801544d2e78a34e4ee0 <-- length seen before
[ 0x057ee6d -> 0x05987e8 ], len=104827/0x1997b, max=48. 789cf352a840e50054d6
[ 0x05c79e2 -> 0x05f4596 ], len=183220/0x2cbb4, max=59. 78dac345f863acf8d742
[ 0x072d93e -> 0x0772b13 ], len=283093/0x451d5, max=57. 78da5a982d5eb2d8c663
[ 0x0775b62 -> 0x077603d ], len=1243/0x4db, max=42. 7801b2893c1e28002800
[ 0x0779d17 -> 0x0798cd5 ], len=126910/0x1efbe, max=48. 780124398e86ffffff4a
[ 0x07a1108 -> 0x07a5315 ], len=16909/0x420d, max=54. 789c0cf716c89946b640


float_freq.c is available upon request.


2. decompression

of course I tried to decompress the 3 first 550d 1.0.8 regions using python zlib, with no success.



--> any idea to confirm (or not) the detection method  and to decode zlib data ?


Indy

[Morgan Look]

Hi Indy.

I used to pull apart router firmwares by finding compressed areas, and using zip/b2z to decompress them.
I would simply find the string that lives at the start of the file (take a look at the first few bytes of a zip file), and then use dd to cut the binary file at the correct point (so that when opening the new file it starts with the same pattern as a .zip file) then open it with unzip.  Any additional non-compressed data at the end should get ignored.

Not sure if this will work in this case or not.

I don't have a firmware dump at this time. Maybe when I get back home I'll look into it, and see if any of the old tricks work.

Hope this helps.

Morgan.

--
http://magiclantern.wikia.com/
 
To post to this group, send email to ml-d...@googlegroups.com
To unsubscribe from this group, send email to ml-devel+u...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/ml-devel?hl=en

[arm.indy]

zlib-1.2.5, inflate.c, line 643 :



if (
#endif
((BITS(8) << 8) + (hold >> 8)) % 31) {
strm->msg = (char *)"incorrect header check";
state->mode = BAD;
break;
}
if (BITS(4) != Z_DEFLATED) {
strm->msg = (char *)"unknown compression method";
state->mode = BAD;
break;
}
DROPBITS(4);
len = BITS(4) + 8;
if (state->wbits == 0)
state->wbits = len;
else if (len > state->wbits) {
strm->msg = (char *)"invalid window size";
state->mode = BAD;
break;
}
state->dmax = 1U << len;
Tracev((stderr, "inflate: zlib header ok\n"));
strm->adler = state->check = adler32(0L, Z_NULL, 0);
state->mode = hold & 0x200 ? DICTID : TYPE;
INITBITS();
break;

550d 108 :

ROM:FF1BF480 LDR R7, [R4,#0x20] ; fcheck
ROM:FF1BF484 MOV R1, #0x1F
ROM:FF1BF488 MOV R0, R7,LSL#24
ROM:FF1BF48C MOV R0, R0,LSR#16
ROM:FF1BF490 ADD R0, R0, R7,LSR#8
ROM:FF1BF494 BL div_maybe
ROM:FF1BF498 CMP R1, #0
ROM:FF1BF49C ADRNE R0, aIncorrentHeade ; "incorrent
header check"
ROM:FF1BF4A0 BNE loc_FF1BF4E0
ROM:FF1BF4A4 AND R0, R7, #0xF
ROM:FF1BF4A8 CMP R0, #8 ; deflate
ROM:FF1BF4AC ADRNE R0, aUnknownCompres ; "unknown
compression method"
ROM:FF1BF4B0 BNE loc_FF1BF4E0
ROM:FF1BF4B4 MOV R0, R7,LSR#4 ; cminfo = win
size
ROM:FF1BF4B8 STR R0, [R4,#0x20]
ROM:FF1BF4BC LDR R1, [R4,#0x24]
ROM:FF1BF4C0 AND R0, R0, #0xF
ROM:FF1BF4C4 SUB R1, R1, #4
ROM:FF1BF4C8 ADD R7, R0, #8
ROM:FF1BF4CC STR R1, [R4,#0x24]
ROM:FF1BF4D0 LDR R0, [R4,#0xC]
ROM:FF1BF4D4 CMP R0, R7
ROM:FF1BF4D8 BGE loc_FF1BF4EC
ROM:FF1BF4DC ADR R0, aInvalidWindowS ; "invalid
window size"
ROM:FF1BF4E0
ROM:FF1BF4E0 loc_FF1BF4E0 ; CODE XREF:
check_zlib_header_maybe+84 j
ROM:FF1BF4E0 ;
check_zlib_header_maybe+94 j
ROM:FF1BF4E0 MOV R6, #5
ROM:FF1BF4E4 STR R0, [R5,#0x18]
ROM:FF1BF4E8 B loc_FF1BF530
ROM:FF1BF4EC ;
---------------------------------------------------------------------------
ROM:FF1BF4EC
ROM:FF1BF4EC loc_FF1BF4EC ; CODE XREF:
check_zlib_header_maybe+BC j
ROM:FF1BF4EC MOV R2, #0
ROM:FF1BF4F0 MOV R1, #0
ROM:FF1BF4F4 MOV R0, #0
ROM:FF1BF4F8 BL sub_FF05C3C4
ROM:FF1BF4FC STR R0, [R5,#0x30]
ROM:FF1BF500 STR R0, [R4,#0x28]
ROM:FF1BF504 MOV R0, #1
ROM:FF1BF508 MOV R0, R0,LSL R7
ROM:FF1BF50C STR R0, [R4,#0x30]
ROM:FF1BF510 LDR R0, [R4,#0x20]
ROM:FF1BF514 TST R0, #0x200 ; check DICT ?
ROM:FF1BF518 MOVEQ R0, #4
ROM:FF1BF51C MOVNE R0, #2
ROM:FF1BF520 STR R0, [R4,#4]
ROM:FF1BF524 MOV R0, #0
ROM:FF1BF528 STR R0, [R4,#0x20]
ROM:FF1BF52C STR R0, [R4,#0x24]

On Jul 19, 9:52 pm, indy arm <arm.indi...@gmail.com> wrote:
> Hello,
>
> zlib is used in the firmware, see strings around FF05B234 in 550d 1.0.8 &
> 1.0.9, just after the SVG lib.
> by zlib I mean:http://www.faqs.org/rfcs/rfc1950.html
>
> so I tried to identify and decode zlib compressed data.
>
> 1. identification method used (to be checked)
>
> I first use 2 bytes signatures 0x7801, 0x789c and 0x78da, as explained here:http://hexblog.com/files/recon%202010%20Skochinsky.pdf, slide 6
> if these 2 bytes are found in the file, I then compute the floating
> frequency (how many different byte values in a 64 byte sliding window) and
> let growing the region if frequency become < 40.

> seehttp://www.cryptool.org/download/CrypToolPresentation-en.pdf, slide 65

[arm.indy]

ROM:FF1BF4EC MOV R2, #0
ROM:FF1BF4F0 MOV R1, #0
ROM:FF1BF4F4 MOV R0, #0

ROM:FF1BF4F8 BL adler32_crc

["Sztupák Sz. Zsolt"]

Back there when I was reversing files from a game, it contained zlib compressed data without the 2 byte signature. It might be worth trying to find decompressable areas within the files without the signature (which means that the identification part is simply to try and feed the block to zlib).

 SztupY

["Sztupák Sz. Zsolt"]

Okay, so I wrote a program that goes through every byte of the firmware (used 60D/1.0.9 and 550D/1.0.9), and tries to decompress the contents from that point using deflate (it doesn't try to find a valid gzip header, as there are no valid headers in the file (checked using another program) ). It did find some stuff, but only at the initial (=code) region, so they are probably just some valid noise. I attached both dumps however (they are the same actually).

The fact that the firmware does not contain deflateable parts doesn't mean anything though, zlib is used in a lot of other libs as well. Finding the gzopen / gzclose / gzread and gzwrite (maybe gzerror, gzscanf, gzprintf, etc.) functions would be probably useful for ML though (like the ability to dump large memory contents compressed, or to read compressed bmp files, etc.).

 SztupY

[Pelican]

http://www.usa.canon.com/cusa/consumer/products/cameras/slr_cameras/eos_60d#
DriversAndSoftware

["Sztupák Sz. Zsolt"]

I found these functions inside 60D/1.0.9:

zlib_deflateEnd_maybe          ROM FF3F84B4 000000FC R . . . . . .
zlib_deflateInit2__maybe       ROM FF3F8730 00000250 R . . . B . .
zlib_deflateInit__maybe        ROM FF3F8980 00000040 R . . . . . .
zlib_deflate_fast              ROM FF3F9B20 00000258 R . . . . . .
zlib_deflate_maybe             ROM FF3F8C7C 000001BC R . . . . . .
zlib_deflate_slow              ROM FF3F9D78 00000008 R . . . . . .
zlib_deflate_stored            ROM FF3F98A8 00000134 R . . . . . .
zlib_fill_window               ROM FF3F9758 00000150 R . . . . . .
zlib_inflate                   ROM FF1C3780 000003CC R . . . . . .
zlib_inflateEnd                ROM FF1C1A58 00000074 R . . . . . .
zlib_inflateInit_              ROM FF1C1A48 00000010 R . . . . . .
zlib_inflateInit2_             ROM FF1C195C 000000EC R . . . . . .
zlib_inflate_x                 ROM FF1C3664 0000011C R . . . . . .
zlib_uncompress_maybe          ROM FF32E388 000000A4 R . . . . . .
zlib_uncompress_maybe2         ROM FF05EE50 000000B4 R . . . . . .

On the ones declared with "_maybe" I'm not so sure, but they do something similar than they have to do. Zlib version seems to be "1.1.f-LZC-1.1.2", and it seems only low level functions (like inflate and deflate) are included, high level ones (gzopen/gzread, etc) are not (or at least I couldn't find them). inflate,deflate and their init/end variants are actually enough for most purposes though. I compared the disassembly to the source code of zlib 1.1.4 for comparison. I'll try to create a compressed memory dumper to test them soon.

 SztupY

["Sztupák Sz. Zsolt"]

2011.07.20. 16:55 keltez�ssel, Pelican �rta:
> http://www.usa.canon.com/cusa/consumer/products/cameras/slr_cameras/eos_60d#
> DriversAndSoftware
>
>
"Firmware Version 1.1.0 incorporates the following fixes and improvements.
1. Fixes a phenomenon where the wireless built-in flash settings are
reset to defaults when the battery is removed from the camera.
2. Fixes an intermittent phenomenon where image-rotation information is
not recognized correctly when shooting with the camera in the vertical
position depending on the timing of the shutter release."

not much imho,

SztupY

[KB Photo]

Is the new firmware compatible with AJ5.x? Should one consider upgrading the factory firmware - benefits?

Many Thanks!

On Wed, Jul 20, 2011 at 10:55 AM, Pelican <p...@pel.hu> wrote:

http://www.usa.canon.com/cusa/consumer/products/cameras/slr_cameras/eos_60d#
DriversAndSoftware

[Alex]

Here are the equivalents for 550D/109, which has the same zlib version:

FF37B298 zlib_deflateEnd_maybe
FF37B514 zlib_deflateInit2__maybe
FF37B764 zlib_deflateInit__maybe
FF37BA60 zlib_deflate_maybe
FF1BF538 zlib_inflate
FF1BD810 zlib_inflateEnd
FF1BD800 zlib_inflateInit_
FF1BD714 zlib_inflateInit2_
FF1BF41C zlib_inflate_x
FF2D817C zlib_uncompress_maybe
FF05B234 zlib_uncompress_maybe2

Here are the last two functions decompiled, hopefully it's easier to
understand what they do and how to call zlib_inflate:
http://codepad.org/cCmumbLn

I wasn't able to find those:

zlib_deflate_fast ROM FF3F9B20 00000258 R . . . . . .

zlib_deflate_slow ROM FF3F9D78 00000008 R . . . . . .
zlib_deflate_stored ROM FF3F98A8 00000134 R . . . . . .
zlib_fill_window ROM FF3F9758 00000150 R . . . . . .

Also, zlib_deflate_slow is only a "return 0".

[arm.indy]

Excellent SztupY! this was my homework for tonight ;-|

Indy

> >>> seehttp://www.cryptool.org/download/CrypToolPresentation-en.pdf,

[ma...@sztupy.hu]

On Wed, 20 Jul 2011 20:27:21 +0300, Alex <broscu...@gmail.com> wrote:

deflate_fast/slow/stored were function pointers, they might actually be
different therefore, and deflate_slow is only a return in 60D too.

according to the decompilation (btw what are using to decompile arm?)
uncompress_maybe1 is mostly the same as uncompress.c (zlib uses a lot of
defines, to hide some ugliness for inflate/deflate(Init))

SztupY

[Alex]

I use a decompiler written by myself in Python, based on symbolic code
emulation (with SymPy), which is included in arm-console.

http://magiclantern.wikia.com/wiki/GPL_Tools/ARM_console/API#deco

In latest version, you have to call the decompiler like this:

dec some_function_name

or:

dec FF123456

[arm.indy]

in zlib-1.2.5/examples/zlib_how.html

/* allocate inflate state */
strm.zalloc = Z_NULL;
strm.zfree = Z_NULL;
strm.opaque = Z_NULL;
strm.avail_in = 0;
strm.next_in = Z_NULL;
ret = inflateInit(&strm);
if (ret != Z_OK)
return ret;

On Jul 20, 8:54 pm, Alex <broscutama...@gmail.com> wrote:
> I use a decompiler written by myself in Python, based on symbolic code
> emulation (with SymPy), which is included in arm-console.
>
> http://magiclantern.wikia.com/wiki/GPL_Tools/ARM_console/API#deco
>
> In latest version, you have to call the decompiler like this:
>
> dec some_function_name
>
> or:
>
> dec FF123456
>
>
>
>
>
>
>

> On Wed, Jul 20, 2011 at 9:45 PM,  <m...@sztupy.hu> wrote:

> >> On Wed, Jul 20, 2011 at 6:09 PM, "Sztupák Sz. Zsolt" <m...@sztupy.hu>

> >>> seehttp://www.cryptool.org/download/CrypToolPresentation-en.pdf, slide

> >>> For more options, visit...
>
> read more »

[Antony Newman]

KB.  The AJ 5.x only works on the 5D mark 2.

AJ.

[Bohemian Imports]

I feel dumb - I only run AJ on my 5D - the 60D is the second angle cam so its been kept native.

Many Thanks!

On Wed, Jul 20, 2011 at 5:58 PM, Antony Newman <antony...@gmail.com> wrote:

KB.  The AJ 5.x only works on the 5D mark 2.

AJ.

--

[arm.indy]

SztupY,

could you please check the ff800000-ffffffff with your method ?
did you check the entire ff010000-ff910000 area ?

and apart finding compressed data in firmware (and give mean to 'black
data'), we can also use zlib for ML compression/decompression.

Indy

On 20 juil, 23:58, Antony Newman <antony.new...@gmail.com> wrote:
> KB.  The *AJ 5.x* only works on the 5D mark 2.
>
> AJ.

[arm.indy]

zlib seems used within what is called BitmapCodec:

ROM:FF2D8710 ; jumptable FF2D85CC entry 7
ROM:FF2D8710 ADD R0, R5, #7
ROM:FF2D8714 MOV R0, R0,LSR#3
ROM:FF2D8718 MUL R1, R6, R0
ROM:FF2D871C
ROM:FF2D871C loc_FF2D871C ; CODE XREF:
sub_FF2D85A8+164 j
ROM:FF2D871C MOV R0, R4
ROM:FF2D8720 BL zlib_something
ROM:FF2D8724 B loc_FF2D862C
...
ROM:FF2D8738 ; jumptable FF2D85CC default entry
ROM:FF2D8738 ; jumptable FF2D85CC entries 0,6
ROM:FF2D8738 LDR R2, =0x19A
ROM:FF2D873C ADR R1, aCodecBitmapcod ; "Codec\
\BitmapCodec.c"

[Alex]

Here's my first port of ML on 60D firmware 1.1.0.

How I upgraded:

1) Formatted card from the camera, copied Canon FIR, upgraded.

2) Copied autoexec.bin, made card bootable; ML didn't boot (just like
if a non-ML card was inserted). Double-checked boot flags on the card,
re-ran make_bootable.sh, still nothing.

3) First guess: maybe the firmware upgrade disabled the bootflag.
Upgraded the installer to 1.1.0, ran it, set bootflag, installer
displayed OK. Autoboot still not working.

4) Formatted card from the camera a second time, made it bootable,
copied autoexec.bin and it worked.

Upgrading from 1.0.8 to 1.0.9 did not clear the bootflag, so probably
upgrading to 1.1.0 did not either.

WARNING: double-check your firmware version before running! Running
1.1.0 fir on 1.0.9 can be VERY DANGEROUS!

(actually, I think instead of init_task it will go to init_task_error
-> DryosPanic, so the chances of setting an invalid flag in NVRAM are
small, but don't try this at home)

I'm thinking to implement some firmware checks (a checksum maybe) to
prevent ML from booting on wrong firmware versions.

Back to our port. I've started the camera, took a picture, recorded a
movie, opened the menus and seems to work. Can you test and tell me
what works and what not?

Thanks to Arm.Indy for the dump, the initial stubs file and the FIR tools!

[Morgan Look]

Yes a simple refusal to boot an incorrect version sounds like a great idea.
Maybe write a log file if that's safely possible, so the user can see why it didn't boot.

Morgan.

On 31/07/2011, at 9:19 AM, Alex wrote:

> Here's my first port of ML on 60D firmware 1.1.0.
>

[Alex]

It's not safe to access file I/O functions, since the addresses differ
between cameras. But the card LED address is the same in 550D, 60D,
600D and 500D (
http://magiclantern.wikia.com/wiki/Led_addresses#550D.2C_60D.2C_600D.2C_500D
), so blinking the card LED would be a good option (maybe the only
one). Also, writing to C022xxxx should be safe according to this post:

http://chdk.setepontos.com/index.php?topic=1493.msg13476#msg13476

[arm.indy]

property 0x2000001 contains firmware inner verion
len = 16

on 600d 1.0.1 :
312e302e310035302831612900000000 (inner version)
"1.0.1"

tested also on 60d and 550d, but not found on 7d.

>  magiclantern-2011Jul31.60D.fw110.alex.zip
> 3737KAfficherTélécharger

[Antony Newman]

7D:   Maybe it's  0x2000005

AJ

[arm.indy]

Hi Antony,

0x2000005 is "outer version" (len=16)
and 0x2000006 is another version (len=8)

found at ff800000+0x1d0024/3c/94 in 60d memory (where properties are
stored)

0x02000001, 16 = 0x1d0024, 312e302e350036352832662900000000 (inner
version)
0x02000005, 16 = 0x1d003c, 332e322e382036352832662900000000 (outer
version)
0x02000006, 8 = 0x1d0094, 080203002f000065 (0x02000006)

Indy

[Colin Peart]

Here is a question for you:

Would you say that the changes necessary to get ML working are just a side effect of the changes to the canon firmware (i.e. small shuffles of function pointers, etc), or would you say that some of the changes necessary are due to Canon actively trying to disrupt ML?   

Obviously this is an opinion question, unless someone is a mind reader or has an inside scoop on Canon's people, it's hard to say for sure.  None the less, I was wondering if it had a feel of intentional disruption or just a side effect of development?

--Colin

Colin Peart
cgap...@gmail.com

[Alex]

For the 60D 1.0.9 -> 1.1.0, I only changed the stubs file (i.e.
function pointers). All buffers and data structures were in the same
place.

[w022a]

Upgrade works ok for me. I followed these steps;

1) Started with a card in camera running ML Jun 8 1.0.9 build
2) Formatted card in camera (not low level format)
3) Copied Canon 1.1.0 firmware to card
4) Upgrade Canon firmware in camera
5) Downloaded ML July 31 1.1.0 build and copied to card
6) Set bootflag with EOS card
7) Put card in camera and switched on camera
8) ML boots ok.

[Grandmaster Neil]

I've never installed ML but thought I'd give the firmware 1.1.0 a try
and it worked well with the instructions provided. Still playing with
the firmware and have only run into a few minor issues so far. I'll
look through the bug tracker to see if they've been addressed.