PWS data format for Interceptor

[Douglas Krug]

I've been discussing the possibility to have the author of "ambient2pwsweather" modify his node.js service to allow input to Matthew Wall's Interceptor plugin. The idea is to allow the continued use of Ambientweather.net, while still capturing weather data for Weewx, so that it can be published to the supported weather services and used with Hubitat or other home automation systems. I'm already able to listen to my WS-0900-IP bridge and get the data into Weewx, but currently that comes at the cost of no longer being able to use Ambientweather.net or any of their services such as IFTTT, Alex Voice Services and Google Assistant.

Is there a guide for how the data must be formatted for input to the interceptor.py plugin? Hoping anyone with knowledge of interceptor or @mwall could comment please.

Here's what the ambient2pwsweather developer wrote:

"If I'm understanding it correctly, it wouldn't be that difficult to enhance this utility to send the data to a different url/port. You would then need to run a new interceptor instance running in "listen mode" that would handle the insertion into weewx.

The minor difficulty would be in ensuring the data is in a format that the interceptor would understand. If you can work that out and get me some documentation (or point me to a URL), I can enhance ambient2pwsweather."

[G Hammer]

How are you capturing the data from the console?

I have this configured so that weewx gets the data, but only by sniffing it. The data is never interrupted/diverted so everyone gets the feed from the console along with weewx.

[Douglas Krug]

LOL. How are you sniffing the data from the console! I could not get that to work with either an Acurite bridge or my Observer IP bridge.

I'm using listen as was recommended for the FineOffset (e.g. Ambient Weather and others) Observer IP bridge. I used Pat O'Brien's instructions for the DD-WRT firewall script together with Matthew Wall's Interceptor plugin (configured for Observer) and it worked.

[G Hammer]

Wow! Looks pretty involved.

I bought a $6 WiFi USB adapter and run an AP on my weewx box.

The WiFi USB is in monitor mode and then you can sniff the data.

There is a way with the routes to mirror the data, letting the data continue to it's original destination while mirroring it to another.

iptables is still the way to do it that method, just mirror instead of redirect.

[G Hammer]

the iptables command is tee

Couldn't edit my other post.

On Monday, August 13, 2018 at 5:06:34 PM UTC-4, Douglas Krug wrote:

[Pat]

@G Hammer - It's not too bad if you have the extra hardware to do the iptables redirect. The $6 dongle is surely cheaper than a new router though. Also I notice you're local to me in Western MA (based off the weewx map). Glad I'm not the only one using weewx around here anymore :)

I don't have any experience with the interceptor driver, but since mwall's given me credit for the observer portion, I thought that maybe I can help a little?

I read the code for the interceptor driver and it looks like if you use the device_type = observer in your weewx.conf file, then the data received to the driver's built in web server needs to be in the Observer format in this example within the code, and similar to the format on my site (which you already linked). 

Since each variable in the URL has a corresponding value (like tempf=43.3 means temperature in Fahrenheit is 43.3), this should be pretty simple for the ambient2pwsweather dev to figure out how to format the observations being sent to the weewx driver. 

Hope this helps a little...

On Monday, August 13, 2018 at 9:19:34 AM UTC-4, Douglas Krug wrote:

[Douglas Krug]

No, not the entire post from Pat O'Brien, just the firewall script. If you can setup DD-WRT (which I've done so many times I don't even need to be awake now), you just enter two lines in the firewall commands. That will pass the data to the interceptor plugin running on Weewx.

iptables -t nat -A PREROUTING -s 192.168.1.55 -p tcp --dport 80 -j DNAT --to-destination x.x.x.x:80
iptables -t nat -A POSTROUTING -j MASQUERADE

Probably there's a way to mirror and not redirect, but I don't know what it is, not do I claim to even fully understand the above script. I just entered it and finally after months of beating my head against the wall, I was able to make something work. From your description, I still really don't understand how exactly you achieved that. I probably don't need an adapter and could sniff with the router I have, but I wouldn't know how. Some have said sniffing isn't as good as listening, but I'm not sure why unless it results in inaccurate data. I have read that many who were originally using the sniff method have switched to the listen method, noting it didn't work as well, but that explanation is a bit short on details.

I just know that if I can find out how the data should be formatted for Interceptor, I can maybe get the ambient2pwsweather developer to modify his code, allowing the preferred listen mode for the Interceptor plug-in and full Ambientweather.net functionality.

[G Hammer]

You don’t need extra hardware if instead of rerouting the data you mirror it.
Somewhere along the line there’s a router with some iptables entries send the console data to weewx.
All I’d suggest is to change them to do mirroring.

@Pat- So you’re in Florida North too! Be glad when this weather breaks.

[Douglas Krug]

Hello again Pat,

Thanks for commenting. This is basically a continuation of the conversation we had on wxforum. I've got experience setting up node.js, that shouldn't be a problem. I stumble across the ambient2pwsweather node project and reached out to see if he would consider modifying it. He wasn't really interested if there was no API for Weewx, which to the best of my understanding, there is not. I suggested Matthew Wall's interceptor plugin, to just capture html traffic from the node server. He said it should not be too difficult, but wants info on how it should be formatted. You say it shouldn't be difficult, and that's cool. @mwall hasn't commented yet and I'm not a developer, so my understanding of it all is pretty rudimentary.

I've got a lot of brilliant minds here, but the data is passing through me and packets are being dropped ;-)

Appreciate the link to Matt's python code. I'll forward this info and see if he can modify from that.

[Pat]

The API into weewx would be a driver. Typically the weather equipment is directly connected and the driver would control the weather station. That's why I wrote the socket driver a few years ago because my equipment isn't directly connected. 

The good news is the interceptor driver has multiple ways to input data into weewx, so it's probably as much of an API as you're going to find. By sending HTTP data to the driver's web server, it can then pass it to weewx like a standard driver would.

That being said, I can't help but to think about others who may want to publish data to Ambient as well. As mentioned on wxforum, the Ambient documentation is kind of lacking for a developer and what the value names are to submit (and how... GET vs. POST). But if there's a project that's successfully posting data to Ambient, then maybe a weewx extension could be created to post to Ambient. An extension like the one for wunderground or PWSWeather might be easy enough to fork for Ambient. This would save the need of setting up and maintaining a node server (and help those here who don't know what node is). The more complicated things get, the more potential for something to break and you not catching it (like your node server crashing and you don't know it for a few days). 

I'm working on a skin for weewx now, but this could be a fun project to think about - forking a PWSWeather extension to the AmbientWeather service - (unless someone else reading this wants to give it a try).

[Pat]

Taking a look at the ambient2pwsweather project, it looks like he's really using the ambient-weather-api project as his heavy lifting for ambient. And looking at the project it looks like what we talked about where AmbientWeather's API documentation talks about how to subscribe to data, but not how to submit data. 

So maybe the thought here is to have your station submit to AmbientWeather natively, and create a driver which gets the data from Ambient's real-time subscribe system and submits it to weewx for archival? 

I've also upgraded to a Davis Vantage Pro 2, so I no longer am using my Ambient ObserverIP, so I don't have test data to use. 

[Douglas Krug]

So maybe the thought here is to have your station submit to AmbientWeather natively, and create a driver which gets the data from Ambient's real-time subscribe system and submits it to weewx for archival?

Yes, that's the idea exactly! Ambient doesn't let you upload to their server unless you buy a $150 license from them. But if you own their equipment, there's no charge to upload. Since there's a way to subscribe to Ambientweather.net via their API, there's for sure a possibility to bring it into Weewx. In fact, there doesn't even need to be node servers and routers passing the data, but I don't know how to code, so I wouldn't be able to execute the concept. The ambient2pwsweather server is needed if you're not running Weewx, but if you are, it's redundant since you can just post to PWSweather from Weewx. It's the ambientweather.net piece that is tricky due to their restrictions. However, everyone with any internet connected Ambient Weather device and up-to-date firmware can upload to ambientweather.net for free. So if there was just a way to subscribe to data from ambientweather.net and bring it into Weewx, there would be no need for Ambient Weather IP sniffing or listening, just an internet connection, Weewx and a plugin.

But I'm a home automation geek. I already have an always on Node.js server running. I was reluctant to add another router, but I have several of them and it was the easiest way to get data into Weewx. So adding another piece to the puzzle doesn't phase me that much. It can all be so simple, but it takes talented developers to do the work.

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/As-rZOD0_Lo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Douglas Krug]

I no longer am using my Ambient ObserverIP, so I don't have test data to use. 

I do of course, so let me know if I could help in this regard.

To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.

[Pat]

It should be doable. Just need to get Python connected to their websocket realtime endpoint and the rest of it (weewx driver wise) should be similar to the drivers I've written in the past. 

I haven't played with Python websockets yet, but sounds fun to tinker with and figure out!

If you're comfortable I may need your Ambient Weather API and APP keys (sent as a private message so its not public) to be able to read the data as a way to validate the driver. You should be able to change these when done to re-secure your keys. 

To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[Douglas Krug]

That sounds amazing. I haven’t requested the key yet, but will now and send to you.

[Pat]

Proving to be a bit of a challenge. Does anyone have any socket.io Python experience? I'm following the dev guides but the endpoint keeps coming up as 404. If I try it with an online socket.io test tool, it works fine (using javascript). Hmm...

[gjr80]

So nothing in the vantage or ws1 drivers or restx.py to help?

Gary

[Pat]

No, those are using sockets like how you and I think sockets are. I'm familiar with traditional sockets (my SocketLogger driver, and I forked the meteostick driver to use sockets).... and I'm familiar with websockets, but this is using socket.io which is a websocket, but not a normal websocket apparently. It's designed to support old browsers so they use a bit of a custom algo to make it happen. 

Since AW is using https and not wss, only the socketIO-client Python lib will work, but when trying to connect to the endpoint it gives a 404. But using the same endpoint on an online JavaScript socket.io test tool, its a success. Could be a limitation with the lib.

The AW socket.io endpoint is for the real-time streaming data from AmbientWeather. Perhaps plan B is to implement a time.sleep() in the driver and just request data every 10 seconds (or something) and submit to the loop. Downside could be duplicate timestamps, unless the weewx loop already allocates for that? Which could be a non-issue. 

[Pat]

I'm still thinking about this, and since I can't get the socket.io real time stream to work in Python, I thought about using a timer to poll the endpoint. However it looks like they only update the endpoint every 5 minutes. I'm thinking that's not really a good option since you probably want real time loop data, and not a 5 minute archive. 

Since it seems to be working with node, and you're familiar with node, maybe that's your best bet right now to get the real time updates out of AW? 

Since you're a customer of AW, you could also email them asking for Python help with the real time endpoint and see what they say. Their API docs say "The easiest way to use the API is to use a Socket.io helper library. They are available in most languages.", but when I use their endpoint with a Python helper library all I get is a 404. 

Here's the code from sockerIO_client's sample code. Nothing fancy here yet, just trying to connect, and it doesn't. 

After doing sudo pip install socketIO_client

I ran this code

import logging
logging.getLogger('socketIO-client').setLevel(logging.DEBUG)
logging.basicConfig()


from socketIO_client import SocketIO, LoggingNamespace


def on_connect():
    print('connect')


with SocketIO('https://api.ambientweather.net/?api=1&applicationKey=YOUR_APPLICATION_KEY', 443, LoggingNamespace ) as socketIO:
    socketIO.on('connect', on_connect)

which outputs:

WARNING:socketIO-client:api.ambientweather.net:443//socket.io [engine.io waiting for connection] unexpected status code (404 {"name":"NotFound","message":"Page not found","code":404,"className":"not-found","errors":{}})

Adding your API Key to that HTTPS endpoint:

with SocketIO('https://api.ambientweather.net/v1/devices?applicationKey=YOUR_APPLICATION_KEY&apiKey=YOUR_API_KEY', 443, LoggingNamespace ) as socketIO:
    socketIO.on('connect', on_connect)

gave me:

WARNING:socketIO-client:api.ambientweather.net:443/v1/devices/socket.io [engine.io waiting for connection] unexpected status code (401 {"error":"apiKey-missing"})

But the API key isn't missing, it's in the string :)

Or if I did something even different (again from the examples)

SocketIO(
    'https://api.ambientweather.net/v1/devices/YOUR_MAC_ADDRESS', 443, LoggingNamespace,
    params={'apiKey': 'YOUR_API_KEY', 'applicationKey': 'YOUR_APPLICATION_KEY' })

I get

WARNING:socketIO-client:api.ambientweather.net:443/v1/devices/YOUR_MAC_ADDRESS/socket.io [engine.io waiting for connection] unexpected status code (404 {"name":"NotFound","message":"Page not found","code":404,"className":"not-found","errors":{}})

However, if I go to an online socket.io test tool, and enter in your real time endpoint, it connects just fine. This is farther than Python is letting me get. 

Again, not sure where to go from here. Maybe Node is your best bet. But also maybe the information above is enough info to get help from them if you do end up reaching out to them?

[Thomas Keffer]

I'm betting this is a ssl certificate problem. Try setting verify=False.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[Pat]

I tried that too and couldn't get it to work. Forgot to mention that. 

Trying it a couple of different ways again, I get the apiKey is missing:

with SocketIO('https://api.ambientweather.net/v1/devices?applicationKey=YOUR_APP_KEY&apiKey=YOUR_API_KEY', 443, LoggingNamespace, verify=False ) as socketIO:

    socketIO.on('connect', on_connect)

/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning)

WARNING:socketIO-client:api.ambientweather.net:443/v1/devices/socket.io [engine.io waiting for connection] unexpected status code (401 {"error":"apiKey-missing"})

Or the 404 again

SocketIO(
    'https://api.ambientweather.net/v1/devices/YOUR_MAC_ADDR', 443, LoggingNamespace,
    verify=False,
    params={'apiKey': 'YOUR_API_KEY', 'applicationKey': 'YOUR_APP_KEY' })


/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning)

WARNING:socketIO-client:api.ambientweather.net:443/v1/devices/YOUR_MAC_ADDR/socket.io [engine.io waiting for connection] unexpected status code (404 {"name":"NotFound","message":"Page not found","code":404,"className":"not-found","errors":{}})

Pat

[Thomas Keffer]

I guess I was wrong!

I find it hard to believe that the Python socket.io library is incapable of handling such a simple case. The problem must be some combination of getting the base host, api endpoint, and parameters right.

-tk

[Douglas Krug]

I've sent this conversation to sup...@ambientweather.com to see if they could offer some guidance or get involved with the discussion here. They have a Facebook API group for questions and discussion, but I think they didn't expect it to be so unloved! Nobody had posted a single comment there in the last 30 days.

[Pat]

That's my thought is trying to find the right combination. The documentation isn't exactly complete, but looking at the "supported" node library they reference, it shows the URL they are using. But they explicitly define websockets there too. 

There's a way to get socket.io to fallback onto websockets where you can use wss, but I haven't figured that bit out yet with AW.

[Douglas Krug]

@pat Have you gotten any further with this? Really disappointed to see that no one from Ambient Weather has chimed in or even sent me an email to say that they won't for [insert reasons here].

[Pat]

I have not. I was hoping someone from AW would reply as well. You may be better off with the node developer for the time being. AW is using socket.io which seems very node friendly, and not so much Python.