Issues with new FW in Acu-Link bridge and the interceptor driver

[Jerome Helbert]

I've had a weewx system running for a couple years uses a variant of the hackulink driver that I had created (used libpcap python libraries to directly sniff the traffic passing through the router (linux based distro running on a laptop) without needing to have an external ngrep or tcpdump command. It worked pretty well, but in hindsight its fairly cobbled together and is losing its usefulness as I add more sensors to the system.

About a month ago the system died, being and electrical engineer i started trying to diagnose a hardware issue and thought that when I saw the power supply putting out 7.5V instead of 4.5V that that was was killed it. As I've read around the internet I'm starting to wonder if it was a botched upgrade to the new SmartHub stuff... Either way I picked up a great deal for a new bridge + 3 tower sensors off ebay, I fired it up and it did its FW update - now my hackulink driver no longer works...

I went out looking for some information on the new format and came across the interceptor driver and I've decided to shift over to that. Unfortunately I am having all kinds of problems getting the sniffing or redirecting part working. Ive tried all kinds of variants of tcpdump and ngrep with various sed filters and for the life of me cannot get the interceptor to parse everything correctly (running it stand alone for now until I get the whole things working.) I've narrowed it down to the GET strings coming in as what looks like multiple packets:

ngrep -l -q -d eth0 '&' 'src host 10.0.7.1 && dst port 80' | sed -u '/&/!d'
filter: (ip or ip6) and ( src host 10.0.7.1 && dst port 80 )
match: &
  GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1
  &id=24C86E068291&mt=5N1x38&sensor=00000588
  &windspeedmph=9&humidity=98
  &tempf=63.9
  &baromin=30.12&battery=normal&rssi=1
  GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1
  &id=24C86E068291&mt=tower&sensor=00002029
  &humidity=63&tempf=68.9
  &baromin=30.12&battery=normal&rssi=2

when I pipe this into "nc 10.0.0.1 9999" and run the interceptor at that port, all the interceptor seems to see is:

ngrep -l -q -d eth0 '&' 'src host 10.0.7.1 && dst port 80' | sed -u '/&/!d' | sed -u -n 3~1p | nc 10.0.0.1 9999
{ "success": 1, "checkversion": "126" }

PYTHONPATH=./ python user/interceptor.py --port 9999 --debug
identifiers: dateutc=now&action=updateraw&realtime=1
{'bridge_id': None, 'sensor_type': None, 'sensor_id': None}
raw data: dateutc=now&action=updateraw&realtime=1
raw packet: {'usUnits..': 1, 'dateTime..': 1476547464, 'usUnits': 1, 'dateTime': 1476547464}
mapped packet: {'usUnits': 1, 'dateTime': 1476547464}

An interesting note is that if I do a DNS hijack, and kill the apache server and link ncat up to port 80 directly I see:

nc -l -p 80
GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1&id=24C86E068291&mt=5N1x31&sensor=00000588&windspeedmph=8&winddir=158&rainin=0.00&dailyrainin=0.00&baromin=30.19&battery=normal&rssi=2 HTTP/1.1
Host: hubapi.myacurite.com
User-Agent: Hub/224
Connection: close

and if then hook interceptor up to port 80:

PYTHONPATH=./ python user/interceptor.py --port 80 --debug
identifiers: dateutc=now&action=updateraw&realtime=1&id=24C86E068291&mt=tower&sensor=00004647&humidity=61&tempf=69.8&baromin=30.19&battery=normal&rssi=3
{'bridge_id': '24C86E068291', 'sensor_type': 'tower', 'sensor_id': '00004647'}
raw data: dateutc=now&action=updateraw&realtime=1&id=24C86E068291&mt=tower&sensor=00004647&humidity=61&tempf=69.8&baromin=30.19&battery=normal&rssi=3
raw packet: {'barometer.00004647.24C86E068291': 30.19, 'rssi.00004647.24C86E068291': 0.75, 'sensor_id.00004647.24C86E068291': '00004647', 'dateTime.00004647.24C86E068291': 1476509500, 'dateTime': 1476509500, 'humidity.00004647.24C86E068291': 61.0, 'usUnits.00004647.24C86E068291': 1, 'temperature.00004647.24C86E068291': 69.8, 'sensor_type.00004647.24C86E068291': 'tower', 'bridge_id.00004647.24C86E068291': '24C86E068291', 'usUnits': 1, 'battery.00004647.24C86E068291': 0}
mapped packet: {'txBatteryStatus': 0, 'outHumidity': 61.0, 'dateTime': 1476509500, 'outTemp': 69.8, 'rxCheckPercent': 0.75, 'usUnits': 1}
identifiers: id=24C86E068291&mt=firmware&line=0&count=1
{'bridge_id': '24C86E068291', 'sensor_type': 'firmware', 'sensor_id': None}
raw data: id=24C86E068291&mt=firmware&line=0&count=1
raw packet: {'dateTime..24C86E068291': 1476509501, 'bridge_id..24C86E068291': '24C86E068291', 'sensor_type..24C86E068291': 'firmware', 'usUnits..24C86E068291': 17, 'dateTime': 1476509501, 'usUnits': 17}
mapped packet: {'usUnits': 17, 'dateTime': 1476509501}
identifiers: id=24C86E068291&mt=firmware&line=0&count=1
{'bridge_id': '24C86E068291', 'sensor_type': 'firmware', 'sensor_id': None}
raw data: id=24C86E068291&mt=firmware&line=0&count=1
raw packet: {'dateTime..24C86E068291': 1476509545, 'bridge_id..24C86E068291': '24C86E068291', 'sensor_type..24C86E068291': 'firmware', 'usUnits..24C86E068291': 17, 'dateTime': 1476509545, 'usUnits': 17}
mapped packet: {'usUnits': 17, 'dateTime': 1476509545}
identifiers: id=24C86E068291&mt=firmware&line=0&count=1
{'bridge_id': '24C86E068291', 'sensor_type': 'firmware', 'sensor_id': None}
raw data: id=24C86E068291&mt=firmware&line=0&count=1
raw packet: {'dateTime..24C86E068291': 1476509588, 'bridge_id..24C86E068291': '24C86E068291', 'sensor_type..24C86E068291': 'firmware', 'usUnits..24C86E068291': 17, 'dateTime': 1476509588, 'usUnits': 17}
mapped packet: {'usUnits': 17, 'dateTime': 1476509588}
identifiers: id=24C86E068291&mt=firmware&line=0&count=1
{'bridge_id': '24C86E068291', 'sensor_type': 'firmware', 'sensor_id': None}
raw data: id=24C86E068291&mt=firmware&line=0&count=1
raw packet: {'dateTime..24C86E068291': 1476509632, 'bridge_id..24C86E068291': '24C86E068291', 'sensor_type..24C86E068291': 'firmware', 'usUnits..24C86E068291': 17, 'dateTime': 1476509632, 'usUnits': 17}
mapped packet: {'usUnits': 17, 'dateTime': 1476509632}

Two things here...

1. I need some way to get ngrep or tcpdump to piece together the http stream properly... I tried to use sed, but since sed operates on a line by line basis - it is not very easy (if possible) for it to remove newlines...
2. I seem to have found a bug with the version reporting... since it reported a different version than what was in it the bridge went into reprogramming mode and would not leave until it updated firmware (undid the DNS Hijack, etc and let it re-update to 224 even though it was already at that version)

[Radar]

hi,
i tested the interceptor driver using tcpdump piped to a perl script to put the packets in one string then send it to weewx interceptor driver using using LWP::UserAgent
that seamed to work

tcpdump -A -n -p -l -i eth0 -s0 -W tcp dst port 80 | stdbuf -oL strings -n8 | stdbuf -oL grep "&" | to the perl script -> weewx interceptor driver

[Radar]

and i can't use it because i use two 5n1 sensors and a rain gauge and a about 10 temp/humid sensors i was just playing with it but it worked

[mwall]

On Saturday, October 15, 2016 at 3:26:23 PM UTC-4, Radar wrote:

and i can't use it because i use two 5n1 sensors and a rain gauge and a about 10 temp/humid sensors i was just playing with it but it worked

you should be able to capture data from all of those sensors using the interceptor driver.

you'll have to define your own database schema to hold them all, but the sensor_map lets you associate each sensor with a database field.

m

[Radar]

i thought that it would but i have it working with a put togohter driver i made that works and edited some parts of weewx for the extra rain and wind and wind gust and noaa reports
was looking at how it did the rain after the fireware upgrade for some reason i could not figure it out how to do the rain (feel real dumb about that) after seeing the code in interceptor driver
when i was playing with it was just using another stock weewx install

[mwall]

On Saturday, October 15, 2016 at 12:21:13 PM UTC-4, Jerome Helbert wrote:

 

Two things here...

1. I need some way to get ngrep or tcpdump to piece together the http stream properly... I tried to use sed, but since sed operates on a line by line basis - it is not very easy (if possible) for it to remove newlines...
   

have you tried radar's approach with stdbuf and strings? once the fragments are reassembled, you should be able to send the data along using curl.
 

1. I seem to have found a bug with the version reporting... since it reported a different version than what was in it the bridge went into reprogramming mode and would not leave until it updated firmware (undid the DNS Hijack, etc and let it re-update to 224 even though it was already at that version)

there is an (untested) fix for this at commit b27acc1 (interceptor driver 0.13a)

[Interceptor]
    ...
    firmware_version = 224

m


 

[Jerome Helbert]

Yeah, this is the primary reason I wanted to switch to the interceptor instead of rewriting my version of the hackulink

[Jerome Helbert]

I didnt get notifications that these messages came in, and just now saw them. I'll give it a shot in a bit after the kid goes to bed.

[Jerome Helbert]

This is also only an issue for someone not sniffing and redirecting, only if the bridge is talking directly to the driver.

On Saturday, October 15, 2016 at 3:04:22 PM UTC-5, mwall wrote:

[Jerome Helbert]

ok, I see what radar is doing... Most of the "magic" is going to happen in a perl script that he is piping the stream into. His tcpdump and stdbuf all pretty much give the same stuff as the ngrep/sed stuff I have been trying to use (spread across multiple lines still) and then he has a perl script that glues them all together.

I already wasn't super thrilled about having to run the helper script outside the driver to sniff the data, and I really want to avoid a third step...

I am brushing up on my iptables... hoping that some combination of a TEE and a DNAT or REDIRECT can sniff off the relevant stream and change the dest ip/port. That then feeds into the interceptor driver as normal. If I cant get that to work, I am thinking about bringing in the libpcap stuff I had added to my hackulink driver and eliminating the helper script all together.

If I got the libpcap working, is that something we'd be interesting in bringing into the driver itself?

On Saturday, October 15, 2016 at 3:04:22 PM UTC-5, mwall wrote:

[Radar]

i have tried ngrep too but the only way i have got it on a single line is to out put to a file. don't know why it won't put that to the console out put.
i think i read that the latest version of ngrep would work but haven't tried it

[Radar]

yep its in the script i like to watch it on the console

here is what it looks like if you want to see

[Jerome Helbert]

Finally got some time it port my libpcap based stuff into the driver. I figure I'd post it here in case you had any interest in incorporating it into the main driver. I could see it having its advantages over the socket based approach.

I'm able to run this on a server already running a webserver or port 80, the myacurite functionality still works, and (my favorite part) all functionality is encompassed within the driver and nothing is required outside of it (ie no ngrep, or tcpdump redirects.

My formal OOP skills are a bit rusty, and I am sure the pcap stuff could be incorporated into the architecture of the base driver better than I have. My intent was to allow for both methods of operation, defaulting to the socket approach unless a flag was specified. FYI, I didn't resolve the standalone portion with its current form (initially I passed in the whole options dict for the various parameters, now it reads those out of stn_dict), just an FYI if you try to run it as is - it will probably fail without some code massaging,

On Sunday, October 16, 2016 at 12:40:14 AM UTC-5, Jerome Helbert wrote:

[Jerome Helbert]

I forgot to mention, this uses the python-libpcap and libpcap libraries as dependencies. Both were available via apt-get on ubuntu.

[mwall]

jerome,

thank you for posting this!

i have incorporated your changes into the interceptor driver at commit 59f09a6 (driver version 0.15).

m

[Jerome Helbert]

I just assumed that when I didn't see anything after I posted this a few weeks ago that you didn't like the idea :P