GETting data from a site using websockets.

[Andy Gadget]

Hi All.

I recently came across the site https://www.seismicportal.eu/realtime.html which provides a near-realtime feed of seismic events (earthquakes) around the world and looked to display the feed using Annex Basic but I have been unable to receive event updates.

My program is using the socket address from the example javascript code and from the header I receive back it appears to be connecting to the server, but after the 'welcome' message I don't get any further messages even though I can see from the page I linked that events are occuring - irregular of course,  but usually a dozen or so an hour.  (I realise this will be a JSON formatted string but I'll break that down once I start getting data back.)

Can anyone see why my code wouldn't work?

Andy.

CODE: xxxx.bas

---

' *****************
'    EARTHQUAKES
' *****************

MySSID$ = "xxxxxx"
MyPwd$ = "xxxxxxx"


' Connect to wifi
WIFI.CONNECT MySSID$, MyPwd$
wlog "Wifi connecting"
While WIFI.STATUS <> 3
wend
wlog "Wifi connected"

do
  WGETASYNC("seismicportal.eu/standing_order", 80,1)
  ONWGETASYNC MsgReceived
  wait
loop


MsgReceived:

wlog WGETRESULT$

return

This is the log :

Wifi connecting

Wifi connected

HTTP/1.1 200 OK

Server: nginx/1.13.3

Date: Sun, 05 Jan 2020 15:47:12 GMT

Content-Type: text/plain; charset=UTF-8

Content-Length: 19

Connection: close

Access-Control-Max-Age: 31536000

Expires: Mon, 04 Jan 2021 15:47:12

Etag: "2b7fd1fcf1da7e69c8ddef117b774c41cf40e932"

Cache-Control: max-age=31536000, public

Access-Control-Allow-Origin: *

Access-Control-Allow-Origin: *

Set-Cookie: BIGipServerpool-seismicportal=44083392.20480.0000; path=/; Httponly

Welcome to SockJS!

[cicciocb]

Hi Andy,

your program does not works as you are trying to make a websocket connection using a web get connection.

The WGET and WGETASYNC functions close the connection as soon as some data is received and, for your application, the connection should be hold always on.

[Euclides Rezende]

Good!!!!

Is possible access Firebase with any  type of connection with the web???

This is great... if possible.

Think about.

Euclides.

[Andy Gadget]

Ah, I see. WGET expects the data requested to be available immediately as part of the transaction whereas a websocket opens a channel for ongoing communication.

There was supposedly a MQTTT channel available from another source with the seismic data but although I can succesfully subscribe to other topics from that broker the seismic channel produces no data.

Unless websockets magically appear in Annex (;¬) it looks like I'll need to go to the Arduino IDE (or forget about it entirely - It wasn't for a serious purpose).

Andy. 

[cicciocb]

Are you looking for something like this ? 

[Andy Gadget]

Francesco, you are an absolute marvel! (and yes, I have noticed the 'beta 9' version ;¬)

I'll await the next release, as I assume you'll want to roll it out with other changes.

Many thanks.

[Andy Gadget]

Hi Cicciocb,  I'm confused.

Should your modified WGETASYNC line work with current version (1.4)?

With this code :

CODE: xxxx.bas

---

do
 WGETASYNC("www.seismicportal.eu/standing_order/websocket", 80,2)

  ONWGETASYNC MsgReceived
  wait
loop

MsgReceived:
wlog WGETRESULT$
return

I'm consistently getting this result :

Connection Error

HTTP/1.1 400 Bad Request

Server: nginx/1.13.3

Date: Sat, 11 Jan 2020 11:34:26 GMT

Content-Type: text/html; charset=UTF-8

Content-Length: 34

Connection: close

Set-Cookie: BIGipServerpool-seismicportal=60860608.20480.0000; path=/; Httponly

Can "Upgrade" only to "WebSocket".

[cicciocb]

I still not have published this update, so your code doesn't work

[Andy Gadget]

Ah, so the beta9 was significant.

Many thanks for clarifying. 

Andy.

[Andy Gadget]

Hi CiccioCB, many thanks for the preview of 1.41 beta 1 to evaluate the websockets feature.

(Please bear in mind I am an absolute amateur in the field of http protocols.) 

The command does connect and receive data via the websocket but with a couple of issues:

Running the code below initiates the websocket with the log as shown and it will then receive messages from the server.

Stopping the program and running again returns the response 'disconnected', so a 2nd run appears to toggle the connection (although I may be misinterpreting this).

If no data is received for a period of time (appears to be four or five minutes but hard to pin down more precisely) it will receive no further data.

If I stop and re-run the program I receive the 'disconnected' message, so it appears that the connection has hung rather than closed. 

Sometimes the feed will receive a few messages but then no more, even when well within the timeout I mentioned above.

Stopping and re-running the program gives the 'disconnected' response.

I don't know how far you want to take the websockets implementation in Annex but I would suggest an explicit 'close websocket' command and a way to determine the status of a websocket (open or closed) would be useful.

This is the code I've been using:

CODE: xxxx.bas

---

WGETASYNC("www.seismicportal.eu/standing_order/websocket", 80,2)
ONWGETASYNC MsgReceived

do
  wait
loop

MsgReceived:
Event$ = WGETRESULT$

if instr(Event$,"onnect")<> 0 then wlog Event$
Act$ = json$(Event$,"action")
Tim$ = json$(Event$,"time")
Mag$ = json$(Event$,"mag")
Reg$ = json$(Event$,"flynn_region")

Type$ = "Event  : "
if Act$ = "update" then Type$ = "Update : "
wlog Type$ +"Mag "+Mag$+" in "+Reg$+" at "+mid$(Tim$,12,5)+" on "+left$(Tim$,10)

return

This is the header for the initial connection:

HTTP/1.1 101 Switching Protocols

Server: nginx/1.13.3

Date: Sun, 12 Jan 2020 20:40:51 GMT

Content-Length: 0

Connection: upgrade

Upgrade: websocket

Sec-Websocket-Accept: Kb0zGQOB06KU4OtGPFqPwTgK0d0=

Set-Cookie: BIGipServerpool-seismicportal=44083392.20480.0000; path=/; Httponly

While I was testing I was seeing quite a few tremors from the erupting volcano in the Philipines which is expected to blow on a large scale, and quite a few strong events in California, although I think they're no strangers to 'quakes.

[cicciocb]

This was just experimental and I was not expecting too much from it :-(

I was just curious to see if it could work with very few changes and it was the case.

I think that the websocket must be explored mode but I fear that there is no more place in the esp8266 to implement it.

For sure it will require a separate command as it is different from the WGETasync.

[Andy Gadget]

Thanks for trying it.  

Not a problem for me as this was just a casual project for something which caught my interest.

I'm impressed with how far you've taken Annex and made it freely available to us all.

Keep up the great work.

Andy.