Interceptor using bridge new Accurate firmware

[Brad Tucker]

I have setup a bridge running raspberry pi. It is connected as follows:

Acurite Bridge -> RaspberyPi WeeWX Bridge -> Internet Connection

It is passing packets through the bridge. The bridge has been updating myaccurite & weather underground for hours and all is well with it.

I am having problems passing the sniffed data to the weewx driver. I have picked together what I think is a decent script but its not parsing properly.

sudo tcpdump -A -n -p -l -i eth0 -s0 -w - tcp dst port 80 | stdbuf -oL strings -n8

output:

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1R

&id=24C86E06B15C&mt=tower&sensor=00012694R

&humidity=65&tempf=72.8R

&baromin=29.35&battery=normal&rssi=3R

 HTTP/1.1

hubapi.myacurite.com

User-Agent: Hub/224R

Connection: close

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1]

&id=24C86E06B15C&mt=5N1x31&sensor=00002179]

&windspeedmph=0&winddir=270]

&rainin=0.00&dailyrainin=0.20&humidity=96&tempf=57.2&dewptf=56]

&baromin=29.35&battery=normal&rssi=3]

 HTTP/1.1

hubapi.myacurite.com

User-Agent: Hub/224]

Connection: closeGET /weatherstation/updateweatherstation.php?ID=KCATHOUS110&PASSWORD=password&dateutc=now&action=updateraw&realtime=1^

&rtfreq=36^

&id=24C86E06B15C&mt=5N1x31&sensor=00002179^

&windspeedmph=0&winddir=270^

&rainin=0.00&dailyrainin=0.20&humidity=96&tempf=57.2&dewptf=56^

&baromin=29.35&battery=normal&rssi=3^

 HTTP/1.1

rtupdate.wunderground.com^

Connection: close

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1b

&id=24C86E06B15C&mt=tower&sensor=00008384b

&humidity=48&tempf=81.4b

&baromin=29.35&battery=normal&rssi=3b

 HTTP/1.1

hubapi.myacurite.com

User-Agent: Hub/224b

Connection: close

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1c

&id=24C86E06B15C&mt=tower&sensor=00012694c

&humidity=65&tempf=72.8c

&baromin=29.35&battery=normal&rssi=3c

 HTTP/1.1

hubapi.myacurite.com

User-Agent: Hub/224c

Connection: close

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1{

&id=24C86E06B15C&mt=5N1x38&sensor=00002179{

&windspeedmph=0&humidity=96{

&tempf=56.9{

&baromin=29.35&battery=normal&rssi=3{

 HTTP/1.1

hubapi.myacurite.com{

User-Agent: Hub/224{

Connection: close

when I run the following through netcat like this:

sudo tcpdump -A -n -p -l -i eth0 -s0 -w - tcp dst port 80 | stdbuf -oL strings -n8 | nc 192.168.1.22 8080

the process only stays alive for seconds and I get this:

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

19 packets captured

19 packets received by filter

0 packets dropped by kernel

and syslog reports this:

Oct 30 19:58:57 weather weewx[781]: interceptor: MainThread: empty queue

Oct 30 19:59:07 weather weewx[781]: interceptor: MainThread: empty queue

Oct 30 19:59:17 weather weewx[781]: interceptor: MainThread: empty queue

Oct 30 19:59:27 weather weewx[781]: interceptor: MainThread: empty queue

Oct 30 19:59:37 weather weewx[781]: interceptor: MainThread: empty queue

After spending the better part of a day with this Im out of ideas.

Please Help!

Brad

[Radar]

i use a perl script to put the obs on one line then feed it to weewx

if you have tcpflow 1.4.4 you can try
tcpflow -C -p -0 -i eth0

[Brad Tucker]

Thanks for the reply Radar. Is your perlscript available for download or is it currently part of the weewx pacakage and I'm missing it?

I'll give tcpflow a shot as well.

Any idea why nc only stays alive for a short period?

Thanks!
Brad

[Radar]

the perl script is not part of weewx i made it long time ago
i don't have it for download (it's real messy) it does a little more than just feed weewx it prints to the console too
you should be able to write one to suit you
don't know about nc

[Brad Tucker]

If only it was that easy. I'm not a coder and I haven't touched perl in a good 20years... on your tcpflow you have -0 in the syntax. My version on the pi doesn't understand -0. What should the -0 do?

Thanks,
B

[Radar]

it puts it on one line
i had to download version 1.4.4 its not yet in fedora rpm so had to compile from source
i'm looking for the script i used when i was playing with interceptor driver can't remember it's name or where i left it :)

[Brad Tucker]

Ha! Of course the most important part ;) I have 1.44 but I'll dig for the source and see what I can find.

Thanks for the help!
Brad

[Radar]

here is the one i used to play with interceptor driver

[Radar]

sorry it is tcpflow 1.4.6

[Brad Tucker]

Im on GitHub now looking at the code for 1.46, Thanks for the perl script as well. I will def give this a shot in the morning.

-B

[Radar]

i use tcpdump with the script

[Brad Tucker]

Radar,

Thanks again for script and all the help. I ran my tcpdump through your script and it actually logs data!!! I have added some of the logs so you can see. One thing Ive noticed is it doesn't log barometric pressure. Not sure why. I did see that after getting processed by the perl script it adds two areas for humidity. Not sure if the duplicate value is the culprit or not.

Couple Questions for you if you have the time:

1. Is my "tcpdump" syntax what you would use?

2. Any idea on why the Barometric Pressure is not logging?

3. Why does the Console say ""checkversion": "126" when my bridge is the latest 2.24 firmware?

Here is the command Im running:

sudo tcpdump -A -n -p -l -i eth0 -s0 -W tcp dst port 80 | stdbuf -oL strings -n8 | stdbuf -oL grep "&" | ./acurite-lwp.pl

Raw Dump not processed with the perl script for 5n1:

..P.....C.&P....0..GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1

..P...0.C.&P....p..&id=24C86E06B15C&mt=5N1x38&sensor=00002179

..P...Z.C.&P...cl..&windspeedmph=2&humidity=50

..P...u.C.&P.......&tempf=71.0

..P.....C.&P.......&baromin=29.30&battery=normal&rssi=3

..P.....C.&P....... HTTP/1.1

..P.....C.&P.......hubapi.myacurite.com

From Console for 5n1 Sensor after issuing the above command:

dateutc=now&action=updateraw&realtime=1&id=24C86E06B15C&mt=5N1x38&sensor=00002179&windspeedmph=3&humidity=50&humidity=50&tempf=71.5&baromin=29.30&battery=normal&rssi=3

{ "success": 1, "checkversion": "126" }

syslog for 5n1:

Oct 31 13:25:39 weather weewx[15893]: interceptor: ServerThread: POST: dateutc=now&action=updateraw&realtime=1&id=24C86E06B15C&mt=5N1x38&sensor=00002179&windspeedmph=3&humidity=50&humidity=50&tempf=71.5&baromin=29.30&battery=normal&rssi=3

Oct 31 13:25:39 weather weewx[15893]: interceptor: MainThread: raw data: dateutc=now&action=updateraw&realtime=1&id=24C86E06B15C&mt=5N1x38&sensor=00002179&windspeedmph=3&humidity=50&humidity=50&tempf=71.5&baromin=29.30&battery=normal&rssi=3

Oct 31 13:25:39 weather weewx[15893]: interceptor: MainThread: ignored parameter realtime=1

Oct 31 13:25:39 weather weewx[15893]: interceptor: MainThread: ignored parameter action=updateraw

Oct 31 13:25:39 weather weewx[15893]: interceptor: MainThread: raw packet: {'sensor_type.00002179.24C86E06B15C': '5N1x38', 'sensor_id.00002179.24C86E06B15C': '00002179', 'temperature.00002179.24C86E06B15C': 71.5, 'dateTime.00002179.24C86E06B15C': 1477945540, 'usUnits.00002179.24C86E06B15C': 1, 'battery.00002179.24C86E06B15C': 0, 'dateTime': 1477945540, 'humidity.00002179.24C86E06B15C': 50.0, 'bridge_id.00002179.24C86E06B15C': '24C86E06B15C', 'barometer.00002179.24C86E06B15C': 29.3, 'rssi.00002179.24C86E06B15C': 0.75, 'windspeed.00002179.24C86E06B15C': 3.0, 'usUnits': 1}

Oct 31 13:25:39 weather weewx[15893]: interceptor: MainThread: mapped packet: {'txBatteryStatus': 0, 'outHumidity': 50.0, 'dateTime': 1477945540, 'outTemp': 71.5, 'windSpeed': 3.0, 'rxCheckPercent': 0.75, 'usUnits': 1}

Oct 31 13:25:40 weather weewx[15893]: manager: added record 2016-10-31 13:25:00 PDT (1477945500) to database 'weewx.sdb'

I also have 2 Tower Sensors attached so I included logs for this incase one of them is conflicting...

Raw Dump not processed with the perl script for the TowerSensor:

..P}hNg.MdvP.......GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1

..P}hN..MdvP...K...&id=24C86E06B15C&mt=tower&sensor=00008384

..P}hN..MdvP.......&humidity=44&tempf=83.8

..P}hN..MdvP...U"..&baromin=29.30&battery=normal&rssi=3

From Console for the TowerSensor after issuing the above command:

dateutc=now&action=updateraw&realtime=1&id=24C86E06B15C&mt=tower&sensor=00012694&humidity=61&tempf=73.4&baromin=29.30&battery=normal&rssi=3

{ "success": 1, "checkversion": "126" }

syslog for the TowerSensor:

Oct 31 13:25:42 weather weewx[15893]: interceptor: ServerThread: POST: dateutc=now&action=updateraw&realtime=1&id=24C86E06B15C&mt=tower&sensor=00012694&humidity=61&tempf=73.4&baromin=29.30&battery=normal&rssi=3

Oct 31 13:25:43 weather weewx[15893]: interceptor: MainThread: raw data: dateutc=now&action=updateraw&realtime=1&id=24C86E06B15C&mt=tower&sensor=00012694&humidity=61&tempf=73.4&baromin=29.30&battery=normal&rssi=3

Oct 31 13:25:43 weather weewx[15893]: interceptor: MainThread: ignored parameter realtime=1

Oct 31 13:25:43 weather weewx[15893]: interceptor: MainThread: ignored parameter action=updateraw

Oct 31 13:25:43 weather weewx[15893]: interceptor: MainThread: raw packet: {'dateTime.00012694.24C86E06B15C': 1477945543, 'sensor_type.00012694.24C86E06B15C': 'tower', 'rssi.00012694.24C86E06B15C': 0.75, 'battery.00012694.24C86E06B15C': 0, 'dateTime': 1477945543, 'bridge_id.00012694.24C86E06B15C': '24C86E06B15C', 'usUnits': 1, 'humidity.00012694.24C86E06B15C': 61.0, 'usUnits.00012694.24C86E06B15C': 1, 'barometer.00012694.24C86E06B15C': 29.3, 'temperature.00012694.24C86E06B15C': 73.4, 'sensor_id.00012694.24C86E06B15C': '00012694'}

Oct 31 13:25:43 weather weewx[15893]: interceptor: MainThread: mapped packet: {'txBatteryStatus': 0, 'outHumidity': 61.0, 'dateTime': 1477945543, 'outTemp': 73.4, 'rxCheckPercent': 0.75, 'usUnits': 1}

Oct 31 13:25:43 weather weewx[15893]: manager: unable to add record 2016-10-31 13:25:00 PDT (1477945500) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime

Added Bonus:

I have the latest tcpflow and it now works with -0. I have a nice pretty output from that by issuing the following:

sudo tcpflow -C -0 -s tcp dst port 80:

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1&id=24C86E06B15C&mt=5N1x38&sensor=00002179&windspeedmph=2&humidity=50&tempf=70.9&baromin=29.31&battery=normal&rssi=3 HTTP/1.1

Host: hubapi.myacurite.com

User-Agent: Hub/224

Connection: close

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1&id=24C86E06B15C&mt=tower&sensor=00008384&humidity=44&tempf=83.8&baromin=29.31&battery=normal&rssi=3 HTTP/1.1

Host: hubapi.myacurite.com

User-Agent: Hub/224

Connection: close

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1&id=24C86E06B15C&mt=tower&sensor=00012694&humidity=63&tempf=73.0&baromin=29.31&battery=normal&rssi=3 HTTP/1.1

Host: hubapi.myacurite.com

User-Agent: Hub/224

Connection: close

Thanks Again!

Brad

[mwall]

On Monday, October 31, 2016 at 4:47:51 PM UTC-4, Brad Tucker wrote:

3. Why does the Console say ""checkversion": "126" when my bridge is the latest 2.24 firmware?

brad,

this is the response from the interceptor.  you can change the response by doing this in your weewx config file:

[Interceptor]
    ...
    firmware_version = 226

version 126 is the chaney format (pre-july-2016).  version 224 is the wu format which was introduced in july 2016.

i've been trying to figure out a clever way to make the response automatic, but i do not yet have a future-proof solution.  until then, set it in the config file.

m

[Brad Tucker]

Thanks for the reply mwall. Good to hear its working as expected and that I can change it if I like.

Thanks!

Brad

[mwall]

On Monday, October 31, 2016 at 4:47:51 PM UTC-4, Brad Tucker wrote:

I also have 2 Tower Sensors attached so I included logs for this incase one of them is conflicting...

brad,

if you want to capture data from two different towers, you should distinguish them in your sensor map and possibly extend the database schema.  otherwise you'll get the 'UNIQUE constraint failed' for dateTime or you'll get mixed data from two different sensor clusters.

the sensor map would look something like this:

[Interceptor]
    ...
    [[sensor_map]]
        pressure = pressure..*
        inTemp = temperature..*
        inHumidity = humidity..*
        # first tower
        outTemp = temperature.AAAAA.*
        outHumidity = humidity.AAAAA.*
        windSpeed.AAAAA.* = windspeed.AAAAA.*
        # second tower
        extraTemp1 = temperature.BBBBB.*
        extraHumid1 = humidity.BBBBB.*
        extraWindSpeed1 = windspeed.BBBBB.*

AAAAA is the identifier for the first tower, BBBBB is the identifier for the second tower

if you want to retain the wind speed from second tower, then you must extend the database schema to include another column extraWindSpeed1

note that the pressure, inTemp, and inHumidity come from the acurite bridge itself.

m

[Brad Tucker]

Good info in here Thanks Mwall!

couple of questions for you, before I tackle the sensor map.

I tried to figure out, based on what I have, a mock sensor map. Im sure I have called out some of the values incorrect. would you please take a look and let me know if Im on the right track? If you can point me to more info and the proper syntax of the sensor map in the documentation that might be helpful. Im having a hard time finding it.

These are the sensors I have:

5n1 (on the roof)

Temp

Wind Speed

Wind Direction

Rain

Humidity

Tower Sensor (Office)

Temp

Humidity

Tower Sensor (Living Room)

Temp

Humidity

Bridge

Temp

Humidity

Pressure

[Interceptor]
    ...
    [[sensor_map]]
        pressure = pressure..*
        inTemp = temperature..*
        inHumidity = humidity..*

        # 5n1 Sesor

        windSpeed = windspeed..*

        windDirecion = winddirection..*

        Temp = temperature..*     

        Rain = rain..*

        Humidity = humidity..*             
        # first tower
        outTemp = temperature.AAAAA.*
        outHumidity = humidity.AAAAA.*

[mwall]

On Monday, October 31, 2016 at 6:03:30 PM UTC-4, Brad Tucker wrote:

I tried to figure out, based on what I have, a mock sensor map. Im sure I have called out some of the values incorrect. would you please take a look and let me know if Im on the right track? If you can point me to more info and the proper syntax of the sensor map in the documentation that might be helpful. Im having a hard time finding it.

These are the sensors I have:

5n1 (on the roof)

Temp

Wind Speed

Wind Direction

Rain

Humidity

Tower Sensor (Office)

Temp

Humidity

Tower Sensor (Living Room)

Temp

Humidity

Bridge

Temp

Humidity

Pressure

brad,

you probably want something like this:

[Interceptor]
    ...
    [[sensor_map]]
        # bridge

        pressure = pressure..*
        inTemp = temperature..*
        inHumidity = humidity..*

        # 5n1

        outTemp = temperature.AAAAA.*
        outHumidity = humidity.AAAAA.*

        windSpeed = windspeed.AAAAA.*
        windDir = winddir.AAAAA.*
        rain = rainfall.AAAAA.*
        # first tower

        extraTemp1 = temperature.BBBBB.*
        extraHumid1 = humidity.BBBBB.*

        # second tower
        extraTemp2 = temperature.CCCCC.*
        extraHumid2 = humidity.CCCCC.*

to see the full sensor identifiers, run weewx directly with debug=1

sorry about the sparse documentation for interceptor - still working on the concepts and code.

m

[mwall]

On Monday, October 31, 2016 at 4:47:51 PM UTC-4, Brad Tucker wrote:

2. Any idea on why the Barometric Pressure is not logging?

you need an entry for 'barometer' in your sensor_map.  for example:

[Interceptor]
    ...
    [[sensor_map]]
        barometer = barometer..*

m

[Brad Tucker]

Fantastic. Thanks Mwall! Not sure Ill get to this tonight, being halloween and all, but Ill def tackle it late night or tomorrow sometime. Thanks for the info. Also thanks for the nice drivers. I played around with the SDR driver and had a blast. Im a ham operator so playing with computers, and radios was fun times. It was only after I got that working completely that I realized that using the SDR version would not pull Pressure off the bridge. I thought Pressure was from the 5n1... So onto the next project which led me here...

Thanks again,

Brad

[Radar]

glad to help

think i got the double humidity fixed attached the fix i hope script if you still want it
but tcpflow looks better fit for the interceptor driver

that's the same tcpdump line that i use too

[Brad Tucker]

Thanks for the help Mwall.

For some reason it is still not mapping the barometer. i have included my sensor map as well as logs.

Thanks!

B

This is what I have used for my sensor map:

[[sensor_map]]

        # bridge

        pressure = pressure..*

        inTemp = temperature..*

        inHumidity = humidity..*

        barometer = barometer..*

        # 5n1

        outTemp = temperature.00002179.*

        outHumidity = humidity.00002179.*

        windSpeed = windspeed.00002179.*

        windDir = winddir.00002179.*

        rain = rainfall.00002179.*

        # living room tower

        extraTemp1 = temperature.00012694.*

        extraHumid1 = humidity.00012694.*

        # second tower

        extraTemp2 = temperature.00008384.*

        extraHumid2 = humidity.00008384.*

I have tried issuing this command via curl to the web server.

curl -s -d 'dateutc=now&action=updateraw&realtime=1&id=24C86E06B15C&mt=5N1x31&sensor=00002179&windspeedmph=2&winddir=158&rainin=0.00&dailyraini&baromin=29.27&battery=normal&rssi=3' http://192.168.1.22:8080

This is the output Im getting in the log. barometer data is there till it gets mapped...

Oct 31 23:01:14 weather weewx[2895]: interceptor: ServerThread: POST: dateutc=now&action=updateraw&realtime=1&id=24C86E06B15C&mt=5N1x31&sensor=00002179&windspeedmph=2&winddir=158&rainin=0.00&dailyrainin=0.01&humidity=69&tempf=62.2&dewptf=51&baromin=29.27&battery=normal&rssi=3

Oct 31 23:01:14 weather weewx[2895]: interceptor: MainThread: raw data: dateutc=now&action=updateraw&realtime=1&id=24C86E06B15C&mt=5N1x31&sensor=00002179&windspeedmph=2&winddir=158&rainin=0.00&dailyrainin=0.01&humidity=69&tempf=62.2&dewptf=51&baromin=29.27&battery=normal&rssi=3

Oct 31 23:01:14 weather weewx[2895]: interceptor: MainThread: ignored parameter realtime=1

Oct 31 23:01:14 weather weewx[2895]: interceptor: MainThread: unrecognized parameter dewptf=51

Oct 31 23:01:14 weather weewx[2895]: interceptor: MainThread: ignored parameter dailyrainin=0.01

Oct 31 23:01:14 weather weewx[2895]: interceptor: MainThread: ignored parameter action=updateraw

Oct 31 23:01:14 weather weewx[2895]: interceptor: MainThread: raw packet: {'sensor_type.00002179.24C86E06B15C': '5N1x31', 'sensor_id.00002179.24C86E06B15C': '00002179', 'winddir.00002179.24C86E06B15C': 158.0, 'temperature.00002179.24C86E06B15C': 62.2, 'rssi.00002179.24C86E06B15C': 0.75, 'usUnits.00002179.24C86E06B15C': 1, 'battery.00002179.24C86E06B15C': 0, 'rainfall.00002179.24C86E06B15C': 0.0, 'dateTime.00002179.24C86E06B15C': 1477980075, 'bridge_id.00002179.24C86E06B15C': '24C86E06B15C', 'barometer.00002179.24C86E06B15C': 29.27, 'dateTime': 1477980075, 'windspeed.00002179.24C86E06B15C': 2.0, 'humidity.00002179.24C86E06B15C': 69.0, 'usUnits': 1}

Oct 31 23:01:14 weather weewx[2895]: interceptor: MainThread: mapped packet: {'outHumidity': 69.0, 'rain': 0.0, 'dateTime': 1477980075, 'windDir': 158.0, 'outTemp': 62.2, 'windSpeed': 2.0, 'usUnits': 1}

[Brad Tucker]

Thanks for this Radar. Also thanks for commenting your new code! Helps a ton. Ill let it run for a bit but it seems like it works well.

For some reason I can't get the Barometer data to map but once I figure that out I should be in a good place.

Thanks Again,

Brad

[mwall]

On Tuesday, November 1, 2016 at 2:08:14 AM UTC-4, Brad Tucker wrote:

Thanks for the help Mwall.

For some reason it is still not mapping the barometer. i have included my sensor map as well as logs.

the wu format sends barometric info in every packet, whereas the chaney format did not.

instead of this:

    barometer = barometer..*

you need to do this:

    barometer = barometer.*.*

however, you will probably find that your temperatures are mixed up.  if so, download the latest interceptor (commit 53f6c5e or later) then we'll have to make a few more changes to the sensor map.

m

[Brad Tucker]

MHall,

I went ahead and upgraded the driver and added barometer.*.* to the map. It all appears to be working now. I can now open the standard skin. I get all the values filling in. I removed the Pond Temp, and added Living Room and a new Value for Office and they all work!!!

Thanks so much for the help!.

My next question is this. Getting data into the drive is tough. Currently Im using a perl script written by Radar. The script works so Im happy to continue using it, however it would be awesome to know your thoughts and syntax on using tcpdump or tcpflow. I have used both but once I pipe the data into netcat things go bad. nectar either closes after 1 or two entries, or it crashes weeks all together...

this is how Ive been using tcpdump to the perl script:

sudo tcpdump -A -n -p -l -i eth0 -s0 -W tcp dst port 80 | stdbuf -oL strings -n8 | stdbuf -oL grep "&" | ./acurite-lwp2.pl

however this tcpflow has a really nice input if you grab the latest version off GitHub and compile yourself...

sudo tcpflow -C -0 -s tcp dst port 80 

once I try and add it to the script with nectar its a no go...

sudo tcpflow -C -0 -s tcp dst port 80 | nc 192.168.1.22 8080

Here is an example of the output tcpflow gives, maybe there is something wrong with the output...

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1&id=24C86E06B15C&mt=5N1x38&sensor=00002179&windspeedmph=2&humidity=50&tempf=70.9&baromin=29.31&battery=normal&rssi=3 HTTP/1.1

Host: hubapi.myacurite.com

User-Agent: Hub/224

Connection: close

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1&id=24C86E06B15C&mt=tower&sensor=00008384&humidity=44&tempf=83.8&baromin=29.31&battery=normal&rssi=3 HTTP/1.1

Host: hubapi.myacurite.com

User-Agent: Hub/224

Connection: close

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1&id=24C86E06B15C&mt=tower&sensor=00012694&humidity=63&tempf=73.0&baromin=29.31&battery=normal&rssi=3 HTTP/1.1

Host: hubapi.myacurite.com

User-Agent: Hub/224

Connection: close

My ultimate goal would not to have to use an extra perl script to help parse the data, but its a fix for now.

Thanks again for all your help!

Brad

Here is a copy of my latest sensor map to anyone who is curious:

    [[sensor_map]]

        # bridge

        pressure = pressure..*

        inTemp = temperature..*

        inHumidity = humidity..*

        barometer = barometer.*.*

        # 5n1

        outTemp = temperature.00002179.*

        outHumidity = humidity.00002179.*

        windSpeed = windspeed.00002179.*

        windDir = winddir.00002179.*

        rain = rainfall.00002179.*

        # living room tower

        extraTemp1 = temperature.00012694.*

        extraHumid1 = humidity.00012694.*

        # second tower

        extraTemp2 = temperature.00008384.*

        extraHumid2 = humidity.00008384.*

[Pat Hayes]

Hello everyone!

I've been following this thread thanks to Brad letting me know of it. It seems I ran into a snag. It seems the interceptor driver doesn't like the input. 

syslog

Nov  2 18:48:22 weewx weewx[20752]: interceptor: MainThread: parse failed for dateutc=now&action=updateraw&realtime=1&id=24C86E08150D&mt=5N1x31&sensor=00002701&windspeedmph=0&winddir=68&rainin=0.00&dailyrainin=0.00&humidity=83&tempf=60.0&dewptf=55&baromin=30.34&battery=normal&rssi=3 HTTP/1.1#015&windspeedmph=0&winddir=68&rainin=0.00&dailyrainin=0.00&humidity=83&tempf=60.0&dewptf=55&baromin=30.34&battery=normal&rssi=3 HTTP/1.1#015&humidity=83&tempf=60.0&dewptf=55&baromin=30.34&battery=normal&rssi=3 HTTP/1.1#015&baromin=30.34&battery=normal&rssi=3 HTTP/1.1#015: invalid literal for float(): 3 HTTP/1.1

Command I'm using.

tcpflow -C -0 -s tcp dst port 80 | ./acurite-lwp.pl

Debian 8 with the latest tcpflow 1.4.6

Thanks for any pointers.

Pat

[Brad Tucker]

I too would love to use tcpflow but had problems with it as well. Ive been using tcpdump in the meantime...

sudo tcpdump -A -n -p -l -i eth0 -s0 -W tcp dst port 80 | stdbuf -oL strings -n8 | stdbuf -oL grep "&" | ./acurite-lwp2.pl

Gluck,

Brad

[mwall]

On Wednesday, November 2, 2016 at 6:53:13 PM UTC-4, Pat Hayes wrote:

I've been following this thread thanks to Brad letting me know of it. It seems I ran into a snag. It seems the interceptor driver doesn't like the input. 

pat,

it looks like your tcpflow/acurite-lwp.pl combination is butchering the string that it is posting to the interceptor.

what do you see when you just do the tcpflow command?

m

[Pat Hayes]

With just the tcpflow command, it shows this.

[root@weewx ~]# tcpflow -C -0 -s tcp dst port 80

tcpflow: listening on bridge0

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1&id=24C86E08150D&mt=5N1x38&sensor=00002701&windspeedmph=0&humidity=86&tempf=58.8&baromin=30.34&battery=normal&rssi=3 HTTP/1.1

Host: hubapi.myacurite.com

User-Agent: Hub/224

Connection: close

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1&id=24C86E08150D&mt=5N1x31&sensor=00002701&windspeedmph=0&winddir=158&rainin=0.00&dailyrainin=0.00&humidity=86&tempf=58.8&dewptf=54&baromin=30.34&battery=normal&rssi=3 HTTP/1.1

Host: hubapi.myacurite.com

User-Agent: Hub/224

Connection: close

GET /weatherstation/updateweatherstation.php?ID=KNJNEWJE4&PASSWORD=pat6854&dateutc=now&action=updateraw&realtime=1&rtfreq=36&id=24C86E08150D&mt=5N1x31&sensor=00002701&windspeedmph=0&winddir=158&rainin=0.00&dailyrainin=0.00&humidity=86&tempf=58.8&dewptf=54&baromin=30.34&battery=normal&rssi=3 HTTP/1.1

Host: rtupdate.wunderground.com

Connection: close

Brad, I tryed your combination, but nothing gets sent to weewx and nothing gets shown in console either.

[root@weewx ~]# tcpdump -A -n -p -l -i bridge0 -s0 -W tcp dst port 80 | stdbuf -oL strings -n8 | stdbuf -oL grep "&" | ./acurite-lwp.pl

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on bridge0, link-type EN10MB (Ethernet), capture size 65535 bytes

^C42 packets captured

42 packets received by filter

0 packets dropped by kernel

[Brad Tucker]

what does you output look like on tcpdump w/o the perl script? mine always looked strange but it worked... Also don't do it off the bridge but use eth0 or your main ethernet to the router.

sudo tcpdump -A -n -p -l -i eth0 -s0 -W tcp dst port 80 | stdbuf -oL strings -n8 | stdbuf -oL grep "&"

output always looks strange but it works well with the perl script:

pi@weather:~ $ sudo tcpdump -A -n -p -l -i eth0 -s0 -W tcp dst port 80 | stdbuf -oL strings -n8 | stdbuf -oL grep "&"

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

E..xfd..d.=.....4....}.P.. B...cP...D...GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1

E..Qfe..d.=.....4....}.P.. ....cP....g..&id=24C86E06B15C&mt=tower&sensor=00012694

....4....}.P.. ....cP...OD..&humidity=53&tempf=73.0

E..Lfg..d.=.....4....}.P.. ....cP....|..&baromin=29.31&battery=normal&rssi=3

E..xfs..d.......4....~.P..0....jP.......GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1

E..Qft..d.......4....~.P..0....jP...UH..&id=24C86E06B15C&mt=tower&sensor=00008384

E..?fu..d.......4....~.P..1....jP....$..&humidity=37&tempf=82.3

E..Lfv..d.......4....~.P..1/...jP..._c..&baromin=29.31&battery=normal&rssi=2

E..xf...d.^.....46._...P/.......P...3...GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1

E..Rf...d.^.....46._...P/..L....P....5..&id=24C86E06B15C&mt=5N1x38&sensor=00002179

E..Cf...d.^.....46._...P/..v....P....6..&windspeedmph=3&humidity=15

E..3f...d.^.....46._...P/.......P...Rg..&tempf=78.4

E..Lf...d.^.....46._...P/.......P.......&baromin=29.31&battery=normal&rssi=3

GLUCK!

B

[Pat Hayes]

Looks like yours minus the good stuff.

[root@weewx ~]# tcpdump -A -n -p -l -i bridge0 -s0 -W tcp dst port 80 | stdbuf -oL strings -n8 | stdbuf -oL grep "&"

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on bridge0, link-type EN10MB (Ethernet), capture size 65535 bytes

E..4l.@.1..+b..7...*...P.&........9.R...............

E..(l.@.1..6b..7...*...P.&...v.PP...3.........

E...l.@.1...b..7...*...P.&...v.PP...{4..GET / HTTP/1.0

E..4l.@.1..(b..7...*...P.&...v.P.....{.....

E..4l.@.1..'b..7...*...P.&...v...."8.......

E..4l.@.1..&b..7...*...P.&...v....'........

E..4l.@.1..%b..7...*...P.&...v.l..-..C.....

E..4l.@.1..$b..7...*...P.&...v.l..-..?.....

E..(..@.1.Y.b..7...*...P.&......P.............

E..(..@.1.Y.b..7...*...P.&......P.............

E..(..@.1.Y.b..7...*...P.&......P.............

E..(..@.1.Y.b..7...*...P.&......P.............

E..(..@.1.Y.b..7...*...P.&......P.............

E.....@.5.&.F,4....*...PX~4..;.......*.....

E....&@.5...F,4....*...P..-..X.R.....$.....

E..4.&@.5.+.F,4....*...PX~6M.<.w...........

f.6.&,.....[.....

f.6.&,...........

f.6.&-.....5.....

^C62 packets captured

62 packets received by filter

0 packets dropped by kernel

[Brad Tucker]

Only thing I noticed is you are running it off the bridge device. I can't get any data off my bridge device. only eth0. I also see a stream similar to yours when its communicating with weatherunderground. Is it possible you didn't let it run long enough? Im assuming you let it run for a min or two and had the same results? 

Past that I have no more ideas :( sorry...

B

[Radar]

the perl script was made to work with tcpdump it won't work with the new tcpflow that puts out every thing on one line

[Jerome Helbert]

I posted a modified version of the interceptor a few weeks ago, I used the libpcap libraries to sniff the data stream directly (no need to run tcpdump or ngrep or any of that external to the driver. My setup has weewx running directly on the machine that is routing traffic to myacurite.com. This would also work in any of the scenarios where you have a router redirecyt

https://groups.google.com/d/msg/weewx-user/3ak8y8h83c0/oKP7QaHeAAAJ

Since we aren't running an HTTP server ourselves, the version response becomes a non-issue. The way my system is set up the entire thing actually still operates with myacurite.com and will still receive all firmware updates as they show up.

[Brad Tucker]

Hello Jerome,

Can you explain how to install and use your modified interceptor? Ive been having problems with the tcpdump method and may give this a try.

Thanks,

Braad

[mwall]

On Saturday, November 5, 2016 at 10:24:08 PM UTC-4, Jerome Helbert wrote:

I posted a modified version of the interceptor a few weeks ago, I used the libpcap libraries to sniff the data stream directly (no need to run tcpdump or ngrep or any of that external to the driver. My setup has weewx running directly on the machine that is routing traffic to myacurite.com. This would also work in any of the scenarios where you have a router redirecyt

https://groups.google.com/d/msg/weewx-user/3ak8y8h83c0/oKP7QaHeAAAJ

Since we aren't running an HTTP server ourselves, the version response becomes a non-issue. The way my system is set up the entire thing actually still operates with myacurite.com and will still receive all firmware updates as they show up.

jerome,

thank you for posting that.  i have merged your changes into the interceptor driver.  now you can choose whether to run the interceptor in 'sniff' mode (uses pcap) or 'listen' mode (uses tcpserver).

i will push the results soon...

m

[mwall]

On Saturday, November 5, 2016 at 10:24:08 PM UTC-4, Jerome Helbert wrote:

I posted a modified version of the interceptor a few weeks ago, I used the libpcap libraries to sniff the data stream directly (no need to run tcpdump or ngrep or any of that external to the driver.

jerome,

i merged your pcap changes into the interceptor driver at commit 59f09a6 (driver version 0.15).  do you mind giving it a try?

you will need a configuration something like this:

[Interceptor]
    driver = user.interceptor
    device_type = acurite-bridge
    mode = sniff
    iface = eth0
    filter = src host X && dst port Y and greater 61

plus a sensor map if you have anything beyond just a single 5n1 sensor cluster.

the implementation should be general enough to work with the os lw30x and fine offset hardware too, but those will probably need some adjustments to the pcap data parsing.

m

[Jerome Helbert]

I looked over the commit, looks good. I will try to get a chance to pull it down and try it out later today.

When I was first writing this stuff into the hackulink driver (and every time I would do a system reinstall) I found there a range of python pcap modules, and they had APIs that all varied. I would recommend adding the specific python module that I called out in my other thread to make installs easier.

[Jerome Helbert]

also should mention that python-libpcap is dependent on the libpcap C libraries.

[Jerome Helbert]

Finally got a chance to check out the driver (latest commit  2ad77b7), had to make some changes to get it to run at all. 

1. I see you had to change from filter to pcap_filter
   
1. The weewx.conf template still references filter
2. The standalone version of the program (__main__) does as well
2. class SniffServer had references to itself in its init when referencing SniffServer.SNAPLEN, SniffServer.PROMISCUOUS, SniffServer.TIMEOUT_MSEnter code here...
   
1. Changed these to self.SNAPLEN, self.PROMISCUOUS, self.TIMEOUT_MS
3. The two logdbg statements in decode_ip_packet are causing problems... some of the values arent strings.
1. I had no solution... just commented them out for now.

With those changes it seems to be running great now

[mwall]

On Tuesday, November 8, 2016 at 8:55:31 AM UTC-5, Jerome Helbert wrote:

Finally got a chance to check out the driver (latest commit  2ad77b7), had to make some changes to get it to run at all. 

1. I see you had to change from filter to pcap_filter
   
1. The weewx.conf template still references filter
2. The standalone version of the program (__main__) does as well
2. class SniffServer had references to itself in its init when referencing SniffServer.SNAPLEN, SniffServer.PROMISCUOUS, SniffServer.TIMEOUT_MSEnter code here...
   
1. Changed these to self.SNAPLEN, self.PROMISCUOUS, self.TIMEOUT_MS
3. The two logdbg statements in decode_ip_packet are causing problems... some of the values arent strings.
1. I had no solution... just commented them out for now.

With those changes it seems to be running great now

jerome,

thank you for testing.  fixes applied at commit 3258d44

i left 'filter' intentionally in the conf and command-line, although if you think 'pcap_filter' is more helpful from a ui point of view i'm happy to change it.  i used pcap_filter as a variable name since 'filter' is a python built-in.

readme and comments have been updated with the pcap details.

thanks again for posting your implementation.  having pcap sniffing as an option in the interceptor is very powerful.

m

[Jerome Helbert]

Filter was an issue because the class Consumer init function uses pcap_filter:

def __init__(self, parser, mode='listen',
 address=DEFAULT_ADDR, port=DEFAULT_PORT, handler=None,
 iface=DEFAULT_IFACE, pcap_filter=DEFAULT_FILTER):

but you are still using filter as an argument name in the __main function__

device = InterceptorDriver.DEVICE_TYPES.get(options.device_type)(
 mode=options.mode,
 iface=options.iface, filter=options.filter,
 address=options.addr, port=options.port)

and class InterceptorDriver init only passes in stn_dict when the conf file is used, so it resolves the conf file name directly:

self._device = self.DEVICE_TYPES.get(self._device_type)(**stn_dict)

If we use pcap_filter in the consumer class init, then we have to use the same argument names in the other places too.

[mwall]

On Tuesday, November 8, 2016 at 3:58:04 PM UTC-5, Jerome Helbert wrote:

Filter was an issue because the class Consumer init function uses pcap_filter:

right you are!  fixed at 2c1a2dd

 

[Jerome Helbert]

This commit looks good! Thanks!

[Radar]

can i ask if you are using a new bridge/smarthub or one you updated
the reason is i use a old bridge i upgraded the firmware on when i use the interceptor 0.15 driver in sniff mode and it dies after some time

and tcpflow put out data that looks like this this could just be how tcpflow printed the data too
{"localtime":"06:56:13","checkversion":"224"}GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1&id=24C86E01765D&mt=tower&sensor=00009686&humidity=43&tempf=57.9&baromin=28.67&battery=normal&rssi=3 HTTP/1.1

On Sunday, October 30, 2016 at 10:01:59 PM UTC-5, Brad Tucker wrote:

I have setup a bridge running raspberry pi. It is connected as follows:

Acurite Bridge -> RaspberyPi WeeWX Bridge -> Internet Connection

It is passing packets through the bridge. The bridge has been updating myaccurite & weather underground for hours and all is well with it.

I am having problems passing the sniffed data to the weewx driver. I have picked together what I think is a decent script but its not parsing properly.

sudo tcpdump -A -n -p -l -i eth0 -s0 -w - tcp dst port 80 | stdbuf -oL strings -n8

output:

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1R

&id=24C86E06B15C&mt=tower&sensor=00012694R

&humidity=65&tempf=72.8R

&baromin=29.35&battery=normal&rssi=3R

 HTTP/1.1

hubapi.myacurite.com

User-Agent: Hub/224R

Connection: close

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1]

&id=24C86E06B15C&mt=5N1x31&sensor=00002179]

&windspeedmph=0&winddir=270]

&rainin=0.00&dailyrainin=0.20&humidity=96&tempf=57.2&dewptf=56]

&baromin=29.35&battery=normal&rssi=3]

 HTTP/1.1

hubapi.myacurite.com

User-Agent: Hub/224]

Connection: closeGET /weatherstation/updateweatherstation.php?ID=KCATHOUS110&PASSWORD=password&dateutc=now&action=updateraw&realtime=1^

&rtfreq=36^

&id=24C86E06B15C&mt=5N1x31&sensor=00002179^

&windspeedmph=0&winddir=270^

&rainin=0.00&dailyrainin=0.20&humidity=96&tempf=57.2&dewptf=56^

&baromin=29.35&battery=normal&rssi=3^

 HTTP/1.1

rtupdate.wunderground.com^

Connection: close

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1b

&id=24C86E06B15C&mt=tower&sensor=00008384b

&humidity=48&tempf=81.4b

&baromin=29.35&battery=normal&rssi=3b

 HTTP/1.1

hubapi.myacurite.com

User-Agent: Hub/224b

Connection: close

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1c

&id=24C86E06B15C&mt=tower&sensor=00012694c

&humidity=65&tempf=72.8c

&baromin=29.35&battery=normal&rssi=3c

 HTTP/1.1

hubapi.myacurite.com

User-Agent: Hub/224c

Connection: close

GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1{

&id=24C86E06B15C&mt=5N1x38&sensor=00002179{

&windspeedmph=0&humidity=96{

&tempf=56.9{

&baromin=29.35&battery=normal&rssi=3{

 HTTP/1.1

hubapi.myacurite.com{

User-Agent: Hub/224{

Connection: close

when I run the following through netcat like this:

sudo tcpdump -A -n -p -l -i eth0 -s0 -w - tcp dst port 80 | stdbuf -oL strings -n8 | nc 192.168.1.22 8080

the process only stays alive for seconds and I get this:

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

19 packets captured

19 packets received by filter

0 packets dropped by kernel

and syslog reports this:

Oct 30 19:58:57 weather weewx[781]: interceptor: MainThread: empty queue

Oct 30 19:59:07 weather weewx[781]: interceptor: MainThread: empty queue

Oct 30 19:59:17 weather weewx[781]: interceptor: MainThread: empty queue

Oct 30 19:59:27 weather weewx[781]: interceptor: MainThread: empty queue

Oct 30 19:59:37 weather weewx[781]: interceptor: MainThread: empty queue

After spending the better part of a day with this Im out of ideas.

Please Help!

Brad

[mwall]

On Saturday, November 12, 2016 at 8:52:20 AM UTC-5, Radar wrote:

can i ask if you are using a new bridge/smarthub or one you updated
the reason is i use a old bridge i upgraded the firmware on when i use the interceptor 0.15 driver in sniff mode and it dies after some time

and tcpflow put out data that looks like this this could just be how tcpflow printed the data too
{"localtime":"06:56:13","checkversion":"224"}GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1GET /weatherstation/updateweatherstation?dateutc=now&action=updateraw&realtime=1&id=24C86E01765D&mt=tower&sensor=00009686&humidity=43&tempf=57.9&baromin=28.67&battery=normal&rssi=3 HTTP/1.1

radar,

how does it die?  please post the exception stack.

you do not need tcpflow when running in sniff mode.

m

[Radar]

here is it running by its self using
PYTHONPATH=. python user/interceptor.py --debug --mode=sniff --iface=p5p1 --filter="src host 192.168.2.14 && dst port 80" --device=acurite-bridge

raw data: dateutc=now&action=updateraw&realtime=1&id=24C86E01765D&mt=tower&sensor=00007411&humidity=51&tempf=52.1&baromin=28.67&battery=normal&rssi=2
raw packet: {'usUnits.00007411.24C86E01765D': 1, 'barometer.00007411.24C86E01765D': 28.67, 'temperature.00007411.24C86E01765D': 52.1, 'bridge_id.00007411.24C86E01765D': '24C86E01765D', 'humidity.00007411.24C86E01765D': 51.0, 'dateTime': 1478958489, 'rssi.00007411.24C86E01765D': 50.0, 'sensor_id.00007411.24C86E01765D': '00007411', 'battery.00007411.24C86E01765D': 0, 'dateTime.00007411.24C86E01765D': 1478958489, 'sensor_type.00007411.24C86E01765D': 'tower', 'usUnits': 1}
mapped packet: {'barometer': 28.67, 'txBatteryStatus': 0, 'outHumidity': 51.0, 'dateTime': 1478958489, 'outTemp': 52.1, 'rxCheckPercent': 50.0, 'usUnits': 1}
identifiers: {'bridge_id': '24C86E01765D', 'sensor_type': 'tower', 'sensor_id': '00006045'}
raw data: dateutc=now&action=updateraw&realtime=1&id=24C86E01765D&mt=tower&sensor=00006045&humidity=56&tempf=41.3&baromin=28.67&battery=normal&rssi=3
raw packet: {'humidity.00006045.24C86E01765D': 56.0, 'barometer.00006045.24C86E01765D': 28.67, 'usUnits.00006045.24C86E01765D': 1, 'dateTime': 1478958494, 'sensor_type.00006045.24C86E01765D': 'tower', 'sensor_id.00006045.24C86E01765D': '00006045', 'battery.00006045.24C86E01765D': 0, 'temperature.00006045.24C86E01765D': 41.3, 'bridge_id.00006045.24C86E01765D': '24C86E01765D', 'dateTime.00006045.24C86E01765D': 1478958494, 'usUnits': 1, 'rssi.00006045.24C86E01765D': 75.0}
mapped packet: {'barometer': 28.67, 'txBatteryStatus': 0, 'outHumidity': 56.0, 'dateTime': 1478958494, 'outTemp': 41.3, 'rxCheckPercent': 75.0, 'usUnits': 1}
Exception in thread ServerThread:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 764, in run
    self.__target(*self.__args, **self.__kwargs)
  File "user/interceptor.py", line 238, in run_server
    self._server.run()
  File "user/interceptor.py", line 275, in run
    self.packet_sniffer.dispatch(1, self.decode_ip_packet)
  File "/usr/lib64/python2.7/site-packages/pcap.py", line 95, in dispatch
    def dispatch(*args): return _pcap.pcapObject_dispatch(*args)
  File "user/interceptor.py", line 297, in decode_ip_packet
    logdbg("SNIFF: %s" % _obfuscate_passwords(data))
  File "user/interceptor.py", line 184, in logdbg
    logmsg(syslog.LOG_DEBUG, msg)
  File "user/interceptor.py", line 181, in logmsg
    (threading.currentThread().getName(), msg))
TypeError: [priority,] message string

[Radar]

this is the part of the config file

[Interceptor]
    # This section is for the network traffic interceptor driver.
   
    # The driver to use:
    driver = user.interceptor
   
    # Specify the hardware device to capture.  Options include:
    #   acurite-bridge - acurite internet bridge
    #   observer - fine offset WH2600/HP1000/HP1003, aka 'observer'
    #   lw30x - oregon scientific LW301/LW302
    #   lacrosse-bridge - lacrosse GW1000U/C84612 internet bridge
    device_type = acurite-bridge
    # listen
    #mode = listen
    #port = 9999
    #address = 192.168.1.6

    # mode sniff
    mode = sniff
    iface = p5p1
    pcap_filter = "src 192.168.2.14 and dst port 80"
   
    [[sensor_map]]
       # smarthub
       barometer = barometer.*.*
       # 5n1 on grage
       outTemp = temperature.00002273.*
       outHumidity = humidity.00002273.*
       windSpeed = windspeed.00002273.*
       windDir = winddir.00002273.*
       rain = rainfall.00002273.*
       txBatteryStatus = battery.00002273.*
       rxCheckPercent = rssi.00002273.*
       # 5n1 on fence
       extraTemp1 = temperature.00002094.*
       extraHumid1 = humidity.00002094.*
       hail = rainfall.00002094.*
       extrawindSpeed = windspeed.00002094.*
       extrawindDir = winddir.00002094.*
       #BatteryStatus1 = battery.00002094.*
       windBatteryStatus = battery.00002094.*
       rssi1 = rssi.00002094.*
       # tower main floor
       inTemp = temperature.00009909.*
       inHumidity = humidity.00009909.*
       inTempBatteryStatus = battery.00009909.*
       # tower
       extraTemp2 = temperature.00009686.*
       extraHumid2 = humidity.00009686.*
       BatteryStatus2 = battery.00009686.*
       rssi2 = rssi.00009686.*
       # tower
       extraTemp3 = temperature.00007411.*
       extraHumid3 = humidity.00007411.*
       BatteryStatus3 = battery.00007411.*
       rssi3 = rssi.00007411.*
       # humidity not working
       extraTemp4 = temperature.00000882.*
       extraHumid4 = humidity.00000882.*
       BatteryStatus4 = battery.00000882.*
       rssi4 = rssi.00000882.*
       # tower
       extraTemp5 = temperature.00006045.*
       extraHumid5 = humidity.00006045.*
       BatteryStatus5 = battery.00006045.*
       rssi5 = rssi.00006045.*
       # rain gauge
       extrarain = rainfall.00011488.*
       #BatteryStatus6 = battery.00011488.*
       rainBatteryStatus = battery.00011488.*
       rssi6 = rssi.00011488.*
       # old tower
       extraTemp8 = temperature.00001626.*
       extraHumid8 = humidity.00001626.*
       BatteryStatus8 = battery.00001626.*
       rssi8 = rssi.00001626.*
       # ProIn water decetor
       extraTemp9 = temperature.00005179.*
       extraHumid9 = humidity.00005179.*
       BatteryStatus9 = battery.00005179.*
       rssi9 = rssi.00005179.*
       probe9 = probe.00005179.*
       check9 = check.00005179.*
       water9 = water.00005179.*
       probeTemp9 = ptempf.00005179.*
       probehumid9 = phumidity.00005179.*
       # ProOut soil and liquid temp
       extraTemp10 = temperature.00003896.*
       extraHumid10 = humidity.00003896.*
       BatteryStatus10 = battery.00003896.*
       rssi10 = rssi.00003896.*
       probe10 = probe.00003896.*
       check10 = check.00003896.*
       water10 = water.00003896.*
       probeTemp10 = ptempf.00003896.*
       probehumid10 = phumidity.00003896.*

[mwall]

On Saturday, November 12, 2016 at 9:08:24 AM UTC-5, Radar wrote:

here is it running by its self using
PYTHONPATH=. python user/interceptor.py --debug --mode=sniff --iface=p5p1 --filter="src host 192.168.2.14 && dst port 80" --device=acurite-bridge

please set debug=1 in weewx.conf

[Radar]

i don't send to WU and thought that maybe something in the http get might be messing it up so was running tcpflow because of this in the stack trace

logdbg("SNIFF: %s" % _obfuscate_passwords(data))

[mwall]

On Saturday, November 12, 2016 at 9:08:24 AM UTC-5, Radar wrote:

here is it running by its self using
PYTHONPATH=. python user/interceptor.py --debug --mode=sniff --iface=p5p1 --filter="src host 192.168.2.14 && dst port 80" --device=acurite-bridge

raw data: dateutc=now&action=updateraw&realtime=1&id=24C86E01765D&mt=tower&sensor=00007411&humidity=51&tempf=52.1&baromin=28.67&battery=normal&rssi=2
raw packet: {'usUnits.00007411.24C86E01765D': 1, 'barometer.00007411.24C86E01765D': 28.67, 'temperature.00007411.24C86E01765D': 52.1, 'bridge_id.00007411.24C86E01765D': '24C86E01765D', 'humidity.00007411.24C86E01765D': 51.0, 'dateTime': 1478958489, 'rssi.00007411.24C86E01765D': 50.0, 'sensor_id.00007411.24C86E01765D': '00007411', 'battery.00007411.24C86E01765D': 0, 'dateTime.00007411.24C86E01765D': 1478958489, 'sensor_type.00007411.24C86E01765D': 'tower', 'usUnits': 1}
mapped packet: {'barometer': 28.67, 'txBatteryStatus': 0, 'outHumidity': 51.0, 'dateTime': 1478958489, 'outTemp': 52.1, 'rxCheckPercent': 50.0, 'usUnits': 1}
identifiers: {'bridge_id': '24C86E01765D', 'sensor_type': 'tower', 'sensor_id': '00006045'}

you're not getting debug log messages, even though you specified --debug.  is your syslog configuration hiding debug?

m

[mwall]

On Saturday, November 12, 2016 at 9:27:33 AM UTC-5, Radar wrote:

i don't send to WU and thought that maybe something in the http get might be messing it up so was running tcpflow because of this in the stack trace
logdbg("SNIFF: %s" % _obfuscate_passwords(data))

there are two modes of operation:

1) listen.  in this mode, you must run tcpflow, tcpdump, or some other packet-sniffing then feed that output to the driver.

2) sniff.  in this mode, the driver does the sniffing itself, so you do not have to run tcpflow, tcpdump, or anything else.

if you run the driver in sniff mode, what you do with tcpflow does not matter, because the driver is not listening for input from tcpflow.

fwiw, your tcpflow command/configuration is dodgey - it is combining multiple http GET requests into a single line.

the exception you posted is due to parsing of non-ascii characters in the data stream from packet sniffing.

we always obfuscate passwords before sending any data to logs, just to be safe.

m

[Radar]

it might be
this is from running it with ./weewxd ../weewx.conf --log-label=weewx-3.6.1

[Brad Tucker]

Hello Radar,

I see you and Matt are already trouble shooting your problems but to answer your question, I believe I am using a new hub. I only bought the unit weeks ago. With that said here is what the bridge reports.

Boot firmware version : 104
Application version : 224

Hope this helps,
Brad

[Radar]

thanks it was just a shoot in the dark my bridge is over 3 years old

[mwall]

On Saturday, November 12, 2016 at 9:37:25 AM UTC-5, Radar wrote:

it might be
this is from running it with ./weewxd ../weewx.conf --log-label=weewx-3.6.1

well, i'd rather not debug your syslog configuration (i find that exercise particularly frustrating on recent ubuntu systems)

so as a quick hack, change the 3 calls to logdbg to loginf in the decode_ip_packet method of SniffServer in interceptor.py

then run again.  that should tell us what is in the data that won't stringify.

m

[Radar]

i'm not liking fedora 19 to much right now for hiding the debug out put but i have it running right now will post back when it quits
it might take some time

thanks for all the help every one

On Saturday, November 12, 2016 at 8:28:17 AM UTC-6, mwall wrote:

[Brad Tucker]

I think the 104 boot version means it is older hardware just updated. I have seen people with a much later boot version. Not positive on the details though.

Good luck.
B

[Radar]

ok i got some of the log here if you need the hole log its about 54 meg big

screen.txt is from running in console

[Radar]

here is the rpm info for libpcap and pylibpcap in case you need it
Name        : libpcap
Epoch       : 14
Version     : 1.4.0
Release     : 1.fc19
Architecture: x86_64
Install Date: Sat 13 Sep 2014 10:29:31 PM CDT
Group       : Development/Libraries
Size        : 312168
License     : BSD with advertising
Signature   : RSA/SHA256, Tue 11 Jun 2013 08:43:52 AM CDT, Key ID 07477e65fb4b18e6
Source RPM  : libpcap-1.4.0-1.fc19.src.rpm
Build Date  : Mon 10 Jun 2013 08:49:44 AM CDT
Build Host  : buildvm-26.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://www.tcpdump.org
Summary     : A system-independent interface for user-level packet capture

Name        : pylibpcap
Version     : 0.6.4
Release     : 3.fc19
Architecture: x86_64
Install Date: Sat 24 Jan 2015 11:08:06 AM CST
Group       : Unspecified
Size        : 91380
License     : BSD
Signature   : RSA/SHA256, Sun 10 Nov 2013 10:59:46 AM CST, Key ID 07477e65fb4b18e6
Source RPM  : pylibpcap-0.6.4-3.fc19.src.rpm
Build Date  : Sat 09 Nov 2013 09:37:55 AM CST
Build Host  : buildvm-16.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://pylibpcap.sourceforge.net/
Summary     : A Python module for libpcap

[Radar]

changed the 3 calls and is running right now

it could be that the bridge didn't take the firmware right it is over 3 years old so might not be able to fix it

[Radar]

your right the 104 boot version is the old one with a red web page
that's what mine is 104 Boot Firmware Version, 224 Application Version

the newer version of Boot Firmware has a black/dark page

[mwall]

radar,

could you try interceptor v0.7a (commit 5d0fe66)

this logs more details about what is actually being parsed from the pcap sniffs

m

[Radar]

interceptor v0.7a it's been running for 23 hours now

[Radar]

looks like its not going to die again, the only things i can find are changed the

pcap_filter = "src 192.168.2.14 and dst port 80"

to
pcap_filter = "src 192.168.2.14 and dst port 80 and http"

and don't know if this had any thing to do with but the bridge was asking dhcpd for a new address and it shouldn't have been i found 3 times in a quick glance at the log
and i have it set to 1 day lease it was like the bridge was rebooting

[Jerome Helbert]

port 80 basically implies http, so that extra bit ought to be pretty redundant...

[Radar]

thanks every one for the help and this great software

[Jerome Helbert]

Botched FW upgrades almost never result in "partial" instabilities... It will almost always result in a bricked device.

[Jerome Helbert]

radar,

What exactly is setup? Are you running the driver as a sniffer on a device like your router that is sniffing the traffic as it flows to the myacurite servers, or are you hijacking the traffic to send it somewhere else?

If it's the former, I'm not sure the bridge is very happy when it doesn't get responses from the server it is trying to reach, and that might explain your excessive dhcp requests. I used to use a DNS hijack (still running a sniffer, but not running on my router) that sent it to a server running weewx as well as an apache server. When the apache server would be down the bridge would actually ignore my DNS hijack and start talking to the aculink servers directly...

Suffice to say, it seems like acurite has some hardcoded defaults built into their software if things on your network aren't functioning 100%... 

[Radar]

i sniff on a AMD Phenom 9750 Quad-Core 64-bit fedora 19 system to weewx 3.5.0 that use tcpdump and a perl script that had no trouble. its been running for over 3 years now
on the same system was checking out the interceptor driver to weewx 3.6.1 in sniff mode that was getting the errors

[Radar]

bridge/smarthub -> netgear EN104 ethernet hub -> linux -> DSL -> web

[Jerome Helbert]

I wasnt trying to suggest hardware failures or anything like that, I was just asking about how things are hooked up on your network.

You second post seems to make it seem like your fedora system is acting as your router? So the bridge traffic is still flowing out into the internet and to the myacurite servers as if weewx wasnt there?

[Radar]

yes it is acting as a router for the bridge
i'm starting to think that myacurite was getting work done to it and couldn't always replay back and like you said the bridge doesn't like that