Virtual Input returns no values?

[Rob]

I’m trying to pull my gas consumption data from my Octopus account using their API but am getting a 401 error response and no data. However, if I paste the URL into a browser then I get the data returned as expected.

The URL is:

https://api.octopus.energy/v1/gas-meter-points/[MPRN]/meters/[METERSERIALNO]/consumption/?api_key=[My_Octopus_API_key]

The response in the HTTP monitor is:

HTTP/1.1 401 Unauthorized\r\nDate: Tue, 05 Mar 2024 10:31:10 GMT\r\nContent-Type: application/json\r\nContent-Length: 58\r\nConnection: close\r\nSet-Cookie: AWSALB=Y8DXEH7Hv9G9FsifBJdAEbACLcP8GWJhEW1Vpq97+OX0Y9vofs74UHK+l3z/9Z7Qx5/BYNNrrew86Wk161XbpNFPe9FfvS0D3XlipFIhnW3SBA55ZU3+g0DY+fk0; Expires=Tue, 12 Mar 2024 10:31:10 GMT; Path=/\r\nSet-Cookie: AWSALBCORS=Y8DXEH7Hv9G9FsifBJdAEbACLcP8GWJhEW1Vpq97+OX0Y9vofs74UHK+l3z/9Z7Qx5/BYNNrrew86Wk161XbpNFPe9FfvS0D3XlipFIhnW3SBA55ZU3+g0DY+fk0; Expires=Tue, 12 Mar 2024 10:31:10 GM...

[Jonathan Dixon]

Can you share the http request from the monitor?

And a successful request using command line"curl -v" for comparison?

I just debugged something using similar approach , in my case it was Loxone including a trailing "/" in the Host: header. 

--
You received this message because you are subscribed to the Google Groups "Loxone English" group.
To unsubscribe from this group and stop receiving emails from it, send an email to loxone-englis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/loxone-english/47b43cd1-8d41-4323-b2a0-c71c6ad53b68n%40googlegroups.com.

[Rob]

This is the http request in the monitor:

GET /v1/gas-meter-points/[MPRN]/meters/[METERSERIALNO]/consumption/?api_key=[My_Octopus_API_key] HTTP/1.1\r\nHost: api.octopus.energy\r\nUser-Agent: [en]\r\nConnection: close

and this is the curl result (although this too has failed, unlike when I use Chrome!!) which seems odd:

 >curl -v https://api.octopus.energy/v1/gas-meter-points/[MPRN]/meters/[METERSERIALNO]/consumption/?api_key=[My_Octopus_API_key]

*   Trying 54.77.136.51:443...

* Connected to api.octopus.energy (54.77.136.51) port 443

* schannel: disabled automatic use of client certificate

* ALPN: curl offers http/1.1

* ALPN: server accepted http/1.1

* using HTTP/1.1

> GET /v1/gas-meter-points/[MPRN]/meters/[METERSERIALNO]/consumption/?api_key=[My_Octopus_API_key] HTTP/1.1

> Host: api.octopus.energy

> User-Agent: curl/8.4.0

> Accept: */*

> 

< HTTP/1.1 401 Unauthorized

< Date: Tue, 05 Mar 2024 11:45:24 GMT

< Content-Type: application/json

< Content-Length: 58

< Connection: keep-alive

< Set-Cookie: AWSALB=6nxAjmmIfu9O1Eugtl3rJK7fT/NJjy+1FZsIXv85T5z0E/P/XY+mfX5zM/IELUoBTGU+6TimOlKb8tK1a2IX9Sv8A0fIHc3JgufckAav1M1ElId+wVXsKZL0OdIA; Expires=Tue, 12 Mar 2024 11:45:24 GMT; Path=/

< Set-Cookie: AWSALBCORS=6nxAjmmIfu9O1Eugtl3rJK7fT/NJjy+1FZsIXv85T5z0E/P/XY+mfX5zM/IELUoBTGU+6TimOlKb8tK1a2IX9Sv8A0fIHc3JgufckAav1M1ElId+wVXsKZL0OdIA; Expires=Tue, 12 Mar 2024 11:45:24 GMT; Path=/; SameSite=None; Secure

< Server: nginx

< WWW-Authenticate: Basic realm="api"

< Allow: GET, HEAD, OPTIONS

< X-Kraken-Correlation-ID: 1-65e705d4-68d041a368c05e415a331f91

< Strict-Transport-Security: max-age=31536000

< X-Content-Type-Options: nosniff

< Referrer-Policy: same-origin

< Cross-Origin-Opener-Policy: same-origin

< X-Frame-Options: SAMEORIGIN

< Vary: Accept-Language, Cookie, origin

< Content-Language: en-gb

< 

{"detail":"Authentication credentials were not provided."}* Connection #0 to host api.octopus.energy left intact

[Jonathan Dixon]

tip: if the command works in Chrome, you can get the corresponding curl command by  using the chrome network tab (F12 -> Network), find the request and do right click -> Copy -> copy as cURL

that should include all cookies / params needed to hermetically replay that request.

Looking at the octopus docs  https://developer.octopus.energy/docs/api/#authentication

I think you need to include the API key as a username (no password) so it is sent via HTTP Basic Auth.

In the virtual output, try setting the HTTP address to "https://[API_KEY]@api.octopus.energy"

To view this discussion on the web visit https://groups.google.com/d/msgid/loxone-english/0d37ce2e-03c0-4244-9247-b2fb10c12908n%40googlegroups.com.

[Rob]

Thanks. Have tried those options and still couldn't get any data back. Very weird, I have other virtual inputs pulling Agile data from Octopus and they work fine!!

It must be something to do with authentication but I have tried quite a few combinations now and no matter where I put the api-key it still does not work.

I did notice from another working Octopus VI that there was a / at the end of the URL, so tried that and in HTTP monitor it showed successful requests with responses coming back. Yay I thought but then when I tried to use command recognition on a VI input command it then said no data and going back to the monitor it was now failing again, even though I hadn't altered the URL.

I also can't get the URL to work in postman so definitely think it's an auth issue using the API key.

Very frustrating!!

[Rob]

UPDATE:

I've managed to get curl to pull the correct data by using the following with api key at the front. Missed the : before so didn't work!!

curl -u "[MY-API-KEY]:" https://api.octopus.energy/v1/gas-meter-points/[MPRN]/meters/[METERSERIALNO]/consumption/

I've also now got the VI to at least get a 200 response using the following similar URL:

https://[MY-API-KEY]:@api.octopus.energy/v1/gas-meter-points/[MPRN]/meters/[METERSERIALNO]/consumption/

However, the Input command still doesn't work. When i click on command recognition it just tells me this:

---------------------------

Download failed

---------------------------

Failed to retrieve data from 'https://[MY-API-KEY]:@api.octopus.energy/v1/gas-meter-points/[MPRN]/meters/[METERSERIALNO]/consumption/' (Status code: 401)

See HTTP monitor for more information.

---------------------------

yet the http monitor shows success....!!!!

[Jonathan Dixon]

I had a quick look at this - unfortunately I only have a gen1 MS to test with, and the authentication seems to get further messed up by the loxberry https2http so not much useful to add. The HTTP virtual input is more limited than http virtual output (can't set headers, can't chose GET/POST method, etc) so generally less amenable to experimentations

FWIW -  in my own install I'd try and avoid coding in too much Octopus specific info in my Loxone config, as I like to keep loxone focused on the devices inside the house, whereas energy supplier is something I'd like the freedom to shop around for as deals come and go without fearing breaking my home management system. But it's a very personal choice where to place the line between building vs personalization logic.

To view this discussion on the web visit https://groups.google.com/d/msgid/loxone-english/14ca6950-ec72-4f4a-b664-da00049fd9abn%40googlegroups.com.

[Rob]

SOLVED:

Thanks Joth, Yes the http VI is more limited than the http VO so I thought I'd try the VO to pull data into the miniserver and, after a couple of fails with authentication syntax, have now got it working.

I use the VO to pull data into a html file saving to the miniserver and then a VI to pull the file data and command recognition as normal to get what I need.

I'm only pulling in previous day's gas usage, rather than building any complex logic, so that I can get a complete picture of the house energy usage. Electricity usage already comes from my inverter integration into the energy monitor.