netatmo decoding for base station firmware 101

[mwall]

beware of letting your weather station update its firmware.

so i finally got around to decoding the netatmo data transmissions.  here is a data payload from a tcp/ip packet:

61 00 25 00 e7 8a 80 55 37 30 3a 65 65 3a 35 30 3a 30 30 3a 30 30 3a 30 30 00 01 01 01 38 0f 51 27 03 7e 27 07 35 05 31 06 61 00 1a 00 e2 8a 80 55 30 32 3a 30 30 3a 30 30 3a 30 36 3a 38 36 3a 30 30 00 ff 00 01 36 61 00 25 00 17 8c 80 55 37 30 3a 65 65 3a 35 30 3a 30 30 3a 30 30 3a 30 30 00 01 01 01 38 0f 4f 27 03 7c 27 07 33 05 59 06 61 00 1a 00 16 8c 80 55 30 32 3a 30 30 3a 30 30 3a 30 36 3a 38 36 3a 30 30 00 ff 00 01 36 61 00 00 00

and this is how it decodes:

61 00 25 00 e7 8a 80 55 37 30 3a 65 65 3a 35 30 3a 30 30 3a 30 30 3a 30 30 00 01 01 01 38 0f 51 27 03 7e 27 07 35 05 31 06
      ??    ts ts ts ts  7  0  :  e  e  :  5  0  :  0  0  :  0  0  :  0  0    T1 T2    HH    ?? ??    P1 P2    NN    C1 C2

61 00 1a 00 e2 8a 80 55 30 32 3a 30 30 3a 30 30 3a 30 36 3a 38 36 3a 30 30 00 ff 00 01 36
      ??    ts ts ts ts  0  2  :  0  0  :  0  0  :  0  6  :  8  6  :  0  0    t1 t2    hh

61 00 25 00 17 8c 80 55 37 30 3a 65 65 3a 35 30 3a 30 36 3a 30 30 3a 30 30 00 01 01 01 38 0f 4f 27 03 7c 27 07 33 05 59 06
      ??    ts ts ts ts  7  0  :  e  e  :  5  0  :  0  6  :  0  0  :  0  0    T1 T2    HH    ?? ??    P1 P2    NN    C1 C2

61 00 1a 00 16 8c 80 55 30 32 3a 30 30 3a 30 30 3a 30 36 3a 38 36 3a 30 30 00 ff 00 01 36
      ??    ts ts ts ts  0  2  :  0  0  :  0  0  :  0  6  :  8  6  :  0  0    tt tt    hh   

61 00 00 00

ts - timestamp as unix epoch: ts = ts1+ts2*256+ts3*256*256+ts4*256*256*256
TT - inside temperature (C): T = (T1+T2*256)/10.0
HH - inside humidity (%)
PP - pressure (mbar): P = (P1+P2*256)/10.0
NN - noise level (db)
CC - CO2: C = C1+C2*256
tt - outside temperature (C): t = (t1+t2*256)/10.0
hh - outside humidity (%)

25 - type code for base station?
1a - type code for remote t/h sensor?
61 00 - indicates beginning of a record
00 00 - termination

the base station identifier is the mac address.  the remote sensor identifier is the serial number, with 02:00:00 prepended to it.

the netatmo base station transmits on port 25050 to one of the netatmo.net servers (b1.netatmo.net through at least b11.netatmo.net - there seems to be a load balancer that hands off the transmissions).

this was with base station firmware 101.

so then i switched the netatmo base station to a different wifi ssid so i could test for a few of the other fields that i had not yet decoded (wifi signal strength, remote signal strength, battery level).  unfortunately the netatmowizard updates the firmware on the base station before you can do anything else.  foolishly i let it do the update.  the firmware went from v101 to v102.

now the data payloads are completely different.  it looks like the new firmware obfuscates the data.

sigh.

m

[Thomas Keffer]

I don't get why vendors feel the need to keep their protocols proprietary. An open protocol encourages an ecosystem of software to be developed around your hardware, which would encourage sales.

An open (serial) protocol sure hasn't hurt Davis.

-tk

[mwall]

On Friday, June 19, 2015 at 9:15:36 AM UTC-4, Tom Keffer wrote:

I don't get why vendors feel the need to keep their protocols proprietary. An open protocol encourages an ecosystem of software to be developed around your hardware, which would encourage sales.

agreed.  i think this change was in response to complaints that netatmo was exposing information about your wifi network.  but instead of being smart about fixing the problem (do not upload that information!), they just obfuscated everything.

http://www.securityweek.com/netatmo-weather-stations-expose-wi-fi-passwords-researcher

i bought a netatmo because i wanted the CO2 and noise level monitoring.  unfortunately you *must* use the netatmo web services - the netatmo hardware does not work without a connection to the internet.  when the company 'netatmo' goes away, your hardware will be useless.  when the company 'netatmo' decides that it wants to change your firmware, it changes your firmware.

if you want to keep a copy of your data, you must write a program to query the netatmo servers to get your data.  if you want to integrate the netatmo into any other system, you must write a program to query the netatmo servers.

that means there are many use cases where netatmo simply will not work.

hopefully there will be rpi or other solutions that are so inexpensive and easy to install that companies such as acurite and netatmo get the message.

the as3959 lightning sensor for rpi is one example.  hopefully we'll get air quality (particulate), CO2, audio, and other sensors soon.

[mwall]

the folks at netatmo responded to my request for a way to revert to the 101 firmware.  i had to run the netatmowizard software again, but this time it put the 101 firmware on the base station.

so i am back to unencrypted packets, local data storage, and easy integration using all of the other weewx extensions.  yay!

sniffing the traffic during the revert process showed no exposed wifi passwords.  it is possible that the passwords are exposed only when you (re)configure the wifi.

i can imagine a few ways that netatmo could avoid the 'exposed data' problem but still address the needs of those with no/spotty internet access and/or desire to not use the netatmo servers:

- only encrypt the things that need to be encrypted

- provide some way for any netatmo owner to decrypt his/her packets, but not those from other stations (e.g., a shared key based on the account name or something entered at the netatmo web site)

- introduce a logger/relay device.  the base station would send data to the logger/relay.  the logger/relay would save the data, then upload to netatmo servers when/if they are available.  (it could be some low-end, rpi-a-class hardware, in a nice aluminum enclosure of course, running weewx and a wunderfixer variant for logging, catchup, and local dashboards ;)

anyway, i'll make a netatmo driver available sometime after we get weewx 3.2 released.

m

[Deborah Pickett]

25 00, 1a 00 and 00 00 would appear to be the byte lengths of the remainder of the packets in little-endian format.  For what it's worth.

On Friday, June 19, 2015 at 1:13:18 PM UTC+10, mwall wrote:

25 - type code for base station?
1a - type code for remote t/h sensor?

00 00 - termination

[Deborah Pickett]

And I bet the 0f field (51 27 and 4f 27) are some kind of pressure field, perhaps the actual station pressure rather than corrected for sea level.  Are you about 125 feet above sea level?

On Friday, June 19, 2015 at 1:13:18 PM UTC+10, mwall wrote:

61 00 25 00 e7 8a 80 55 37 30 3a 65 65 3a 35 30 3a 30 30 3a 30 30 3a 30 30 00 01 01 01 38 0f 51 27 03 7e 27 07 35 05 31 06
      ??    ts ts ts ts  7  0  :  e  e  :  5  0  :  0  0  :  0  0  :  0  0    T1 T2    HH    ?? ??    P1 P2    NN    C1 C2

[mwall]

On Friday, June 26, 2015 at 7:36:40 AM UTC-4, Deborah Pickett wrote:

And I bet the 0f field (51 27 and 4f 27) are some kind of pressure field, perhaps the actual station pressure rather than corrected for sea level.  Are you about 125 feet above sea level?

nice sleuthing deborah!  the station altitude is exactly 125 feet.
 

[mwall]

here are more bits that i have not yet decoded.  what follows are 4 conversations, overlaid so you can see the patterns.  the netatmo station is 192.168.32.197, with mac address 70:ee:50:06:84:72.  mac address is easy to spot (xx xx 3a xx xx 3a xx xx 3a xx xx 3a xx xx 3a xx xx).  so are the time stamps (xx xx 80 55).

pkt time_offset src              dst             len port        bytes
  4 0.323922    195.154.176.198  192.168.32.197  78  25050→19930 03 00 14 00 cd 7f 80 55 0b 2c ad 44 03 4b e3 d2 0b bb 26 b7 71 69 f3 40
 24 606.746324  195.154.176.41   192.168.32.197  78  25050→19931 03 00 14 00 2b 82 80 55 0a de 00 ea af 2a f6 2b bb a4 d5 06 4c 4a ca 6b
 44 730.353719  195.154.178.216  192.168.32.197  78  25050→19932 03 00 14 00 a7 82 80 55 65 01 60 25 14 bd e8 7b 41 8d  2b 74 37 ad d5 d5
 62 1338.822386 195.154.178.228  192.168.32.197  78  25050→19933 03 00 14 00 07 85 80 55 d3 f1 c2 91 d2 d6 95 a5 3d 08 4b bb 10 7f 08 c9

pkt time_offset src              dst             len port        bytes
  5 0.410523    192.168.32.197   195.154.176.198 105 19930→25050 10 00 01 00 03 04 00 2a 00 37 30 3a 65 65 3a 35 30 3a 30 36 3a 38 34 3a 37 32 65 00 e3 3e aa f9 01 00 fb 79 70 09 47 c1 21 1a 49 c1 21 1a 49 c1 21 1a 49
 25 606.835107  192.168.32.197   195.154.176.41  105 19931→25050 10 00 01 00 03 04 00 2a 00 37 30 3a 65 65 3a 35 30 3a 30 36 3a 38 34 3a 37 32 65 00 43 99 a0 ec 01 00 fb 2f df 70 51 e7 2d 45 43 e7 2d 45 43 e7 2d 45 43
 45 730.434003  192.168.32.197   195.154.178.216 105 19932→25050 10 00 01 00 03 04 00 2a 00 37 30 3a 65 65 3a 35 30 3a 30 36 3a 38 34 3a 37 32 65 00 a0 a1 11 bb 01 00 fb f8 5c 92 08 f8 5c 92 08 f8 5c 92 08 90 7e 45 b3
 63 1338.907227 192.168.32.197   195.154.178.228 105 19933→25050 10 00 01 00 03 04 00 2a 00 37 30 3a 65 65 3a 35 30 3a 30 36 3a 38 34 3a 37 32 65 00 3e ee 5f cf 01 00 fb b8 70 fa 7b b8 70 fa 7b f9 41 e1 62 41 10 f2 6c

pkt time_offset src              dst             len port        bytes
  7 0.530964    195.154.176.198  192.168.32.197  62  25050→19930 78 01 04 00 1a c8 67 6d
 27 606.953529  195.154.176.41   192.168.32.197  62  25050→19931 78 01 04 00 94 c1 7f 19
 47 730.555874  195.154.178.216  192.168.32.197  62  25050→19932 78 01 04 00 4f 13 cf 2c
 65 1339.030282 195.154.178.228  192.168.32.197  62  25050→19933 78 01 04 00 72 64 2c 11

packets with payloads such as the one in the first posting of this thread are the last part of the conversation.

the data corresponding to these times are:

packet idx  pkt_ts     row_from_csv_data
packet  50: 1434485403 1434485403;"2015/06/16 16:10:03";25.5;56;1523;45;1011
            1434485370 ?
packet  68: 1434485705 1434485705;"2015/06/16 16:15:05";25.6;56;1572;43;1011
            1434486010 1434486010;"2015/06/16 16:20:10";25.6;56;1581;47;1010.8
packet  86: 1434486314 1434486314;"2015/06/16 16:25:14";25.6;57;1586;46;1010.9
            1434486618 1434486618;"2015/06/16 16:30:18";25.6;57;1556;48;1011
packet 104: 1434486921 1434486921;"2015/06/16 16:35:21";25.6;56;1592;40;1011.1
            1434487224 1434487224;"2015/06/16 16:40:24";25.7;57;1572;40;1011
packet 122: 1434487527 1434487527;"2015/06/16 16:45:27";25.7;56;1585;53;1011
            1434487831 1434487831;"2015/06/16 16:50:31";25.7;56;1625;51;1010.8

packet  52: 1434485370 1434485370;"2015/06/16 16:09:30";25.3;53
                       1434485678;"2015/06/16 16:14:38";25.3;53
packet  70: 1434485985 1434485985;"2015/06/16 16:19:45";25.4;53
                       1434486293;"2015/06/16 16:24:53";25.4;54
packet  88: 1434486601 1434486601;"2015/06/16 16:30:01";25.4;54
                       1434486908;"2015/06/16 16:35:08";25.4;54
packet 106: 1434487216 1434487216;"2015/06/16 16:40:16";25.5;54
                       1434487522;"2015/06/16 16:45:22";25.5;54
packet 124: 1434487830 1434487830;"2015/06/16 16:50:30";25.5;54
                       1434488086;"2015/06/16 16:54:46";25.5;54

finally, here is a raw conversation (the first of the 4 above), showing only packets with data payloads:

    idx time_offset src                dst              port        data
      4 0.323922    195.154.176.198    192.168.32.197   25050→19930 03 00 14 00 cd 7f 80 55 0b 2c ad 44 03 4b e3 d2 0b bb 26 b7 71 69 f3 40
      5 0.410523    192.168.32.197     195.154.176.198  19930→25050 10 00 01 00 03 04 00 2a 00 37 30 3a 65 65 3a 35 30 3a 30 36 3a 38 34 3a 37 32 65 00 e3 3e aa f9 01 00 fb 79 70 09 47 c1 21 1a 49 c1 21 1a 49 c1 21 1a 49
      7 0.530964    195.154.176.198    192.168.32.197   25050→19930 78 01 04 00 1a c8 67 6d
     10 0.821842    192.168.32.197     195.154.176.198  19930→25050 61 00 25 00 90 7e 80 55 37 30 3a 65 65 3a 35 30
                                                              0010  3a 30 36 3a 38 34 3a 37 32 00 fe 00 01 38 0f 52
                                                              0020  27 03 7f 27 07 2f 05 c8 05 61 00 1a 00 79 7e 80
                                                              0030  55 30 32 3a 30 30 3a 30 30 3a 30 36 3a 38 36 3a
                                                              0040  32 38 00 fd 00 01 35 61 00 25 00 bf 7f 80 55 37
                                                              0050  30 3a 65 65 3a 35 30 3a 30 36 3a 38 34 3a 37 32
                                                              0060  00 fe 00 01 38 0f 52 27 03 7f 27 07 2b 05 f5 05
                                                              0070  61 00 1a 00 ac 7f 80 55 30 32 3a 30 30 3a 30 30
                                                              0080  3a 30 36 3a 38 36 3a 32 38 00 fd 00 01 35 61 00
                                                              0090  00 00
     11 0.942141    195.154.176.198    192.168.32.197   25050→19930 06 00 00 00
     12 1.024135    192.168.32.197     195.154.176.198  19930→25050 10 00 01 00 05 07 00 e8 00 2d 01 00 00 00 00 32
                                                              0010  30 32 3a 30 30 3a 30 30 3a 30 36 3a 38 36 3a 32
                                                              0020  38 43 ba 16 00 00 2b 00 ac 7f 80 55 cc 7f 80 55
                                                              0030  55 30 32 3a 30 30 3a 30 30 3a 30 36 3a 38 36 3a
                                                              0040  32 38 00 fd 00 01 35 61 00 25 00 bf 7f 80 55 37
                                                              0050  30 3a 65 65 3a 35 30 3a 30 36 3a 38 34 3a 37 32
                                                              0060  00 fe 00 01 38 0f 52 27 03 7f 27 07 2b 05 f5 05
                                                              0070  61 00 1a 00 ac 7f 80 55 30 32 3a 30 30 3a 30 30
                                                              0080  3a 30 36 3a 38 36 3a 32 38 00 fd 00 01 35 61 00
                                                              0090  00 00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff
                                                              00a0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
                                                              00b0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
                                                              00c0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
                                                              00d0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
                                                              00e0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
                                                              00f0  64
     14 4.158644    195.154.176.198    192.168.32.197   25050→19930 08 00 00 00
     15 4.197530    192.168.32.197     195.154.176.198  19930→25050 09 00 00 00

m

[Deborah Pickett]

Reverse-engineering protocols is fun.  It gives me an inkling of how people decoding the Enigma messages in WWII felt...

The third and fourth bytes are definitely a length field.

My best guess is that a conversation goes like this:

On Friday, June 26, 2015 at 10:23:48 PM UTC+10, mwall wrote:

finally, here is a raw conversation (the first of the 4 above), showing only packets with data payloads:

    idx time_offset src                dst              port        data
      4 0.323922    195.154.176.198    192.168.32.197   25050→19930 03 00 14 00 cd 7f 80 55 0b 2c ad 44 03 4b e3 d2 0b bb 26 b7 71 69 f3 40

Internet -> station: Message 0003 "Hello, it's currently 0x55807fcd, and the secret password is 0b2cad44034be3d20bbb26b77169f340"

[The randomness in the last 16 bytes, coupled with the fact that it's 16 bytes long, makes me think that this is an MD5 sum of a set of properties, which might include the time, the serial number of the station, and some secret salt.  I guessed a few layouts and never got a match.  My money is on salt.]

      5 0.410523    192.168.32.197     195.154.176.198  19930→25050 10 00 01 00 03 04 00 2a 00 37 30 3a 65 65 3a 35 30 3a 30 36 3a 38 34 3a 37 32 65 00 e3 3e aa f9 01 00 fb 79 70 09 47 c1 21 1a 49 c1 21 1a 49 c1 21 1a 49

Station -> Internet: Message 0010 ("three")

[Is this a count, an enumeration or something else? It might also serve as an acknowledgment of the 0003 packet that was just received.]

Station -> Internet: Message 0004 ( "I'm 70:ee:50:06:84:72, [0065 f9aa3ee3 0001 fb 47097079 491a21cf 491a21c1 491a21c1]"

[I've taken a guess at the breakdown of the payload.  Some of it clearly is 32 bits in length. The last four values seem too regular to be CRC32 checksums, and vary too much to be sensor data.]

      7 0.530964    195.154.176.198    192.168.32.197   25050→19930 78 01 04 00 1a c8 67 6d

Internet -> Station: Message 0178 ("6d67c81a")

[I bet this is some kind of CRC, also it probably acts as an acknowledgment of the station's 0004 message.]

     10 0.821842    192.168.32.197     195.154.176.198  19930→25050 61 00 25 00 90 7e 80 55 37 30 3a 65 65 3a 35 30
                                                              0010  3a 30 36 3a 38 34 3a 37 32 00 fe 00 01 38 0f 52
                                                              0020  27 03 7f 27 07 2f 05 c8 05 61 00 1a 00 79 7e 80
                                                              0030  55 30 32 3a 30 30 3a 30 30 3a 30 36 3a 38 36 3a
                                                              0040  32 38 00 fd 00 01 35 61 00 25 00 bf 7f 80 55 37
                                                              0050  30 3a 65 65 3a 35 30 3a 30 36 3a 38 34 3a 37 32
                                                              0060  00 fe 00 01 38 0f 52 27 03 7f 27 07 2b 05 f5 05
                                                              0070  61 00 1a 00 ac 7f 80 55 30 32 3a 30 30 3a 30 30
                                                              0080  3a 30 36 3a 38 36 3a 32 38 00 fd 00 01 35 61 00
                                                              0090  00 00

Station -> Internet: Message 0061 [x4]

[This is the sensor data which you've already decoded]
 

     11 0.942141    195.154.176.198    192.168.32.197   25050→19930 06 00 00 00

Internet -> station: Message 0006

[No content, I'm sure it's just an acknowledgment.]

     12 1.024135    192.168.32.197     195.154.176.198  19930→25050 10 00 01 00 05 07 00 e8 00 2d 01 00 00 00 00 32
                                                              0010  30 32 3a 30 30 3a 30 30 3a 30 36 3a 38 36 3a 32
                                                              0020  38 43 ba 16 00 00 2b 00 ac 7f 80 55 cc 7f 80 55
                                                              0030  55 30 32 3a 30 30 3a 30 30 3a 30 36 3a 38 36 3a
                                                              0040  32 38 00 fd 00 01 35 61 00 25 00 bf 7f 80 55 37
                                                              0050  30 3a 65 65 3a 35 30 3a 30 36 3a 38 34 3a 37 32
                                                              0060  00 fe 00 01 38 0f 52 27 03 7f 27 07 2b 05 f5 05
                                                              0070  61 00 1a 00 ac 7f 80 55 30 32 3a 30 30 3a 30 30
                                                              0080  3a 30 36 3a 38 36 3a 32 38 00 fd 00 01 35 61 00
                                                              0090  00 00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff
                                                              00a0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
                                                              00b0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
                                                              00c0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
                                                              00d0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
                                                              00e0  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
                                                              00f0  64

Station -> Internet: Message 0010 ("five")

[See "three" above.]

Station -> Internet: Message 0007

[It wouldn't surprise me if this is some kind of debug dump of the station's RAM.  I can see two MAC addresses, two timestamps 32 seconds apart, the most recent sensor data and a bunch of padding bits.]

     14 4.158644    195.154.176.198    192.168.32.197   25050→19930 08 00 00 00

Internet -> station: Message 0008

[Probably an acknowledgment of message 0007. Given that three seconds have elapsed, it might even be telling the station that the data has been committed to their database and can be erased locally.]

     15 4.197530    192.168.32.197     195.154.176.198  19930→25050 09 00 00 00

Station -> Internet: Message 0009

[My guess: "No more data".]

[John Newgas]

I am not interested in weather, but am playing at interacting with the Netatmo welcome camera.   They have exposed their APIfor the Weather and Welcom camera at https://dev.netatmo.com/doc/intro

so you might have an easier time that way.

I plan on just using a little Python which should be able to do all that is needed.

John

who will leaver this group shortly !