A little more SstHttpClient help please

[Louis LaBrunda]

Hi,

With the real password this works with curl:

curl -X POST https://enlighten.enphaseenergy.com/login/login.json? -F "user[email]=LouLaB...@gmail.com" -F "user[password]=xxxxxxxxx"

and returns what looks like json with values I expect.  The code below sorta works but returns what looks like the html of a login screen.  So, it didn't really login, it just went to the server's login page.  Any ideas?

Lou

| httpClient loginUrl headers enphaseReply enphaseData enphaseStream reader user password |

user := 'LouLaB...@gmail.com'.
password := 'notreal'.
loginUrl := 'https://enlighten.enphaseenergy.com/login/login.json?'.
loginBody := '{''user[email]'':%1, ''user[password]'':%2}' bindWith: user with: password.
headers := LookupTable new.
headers
at: 'accept' put: 'application/json';
at: 'Content-Type' put: 'multipart/form-data'.
httpClient := (SstHttpClient forTransportScheme: 'httpsl') startUp.
enphaseReply := [
[httpClient post: loginBody typed: nil at: loginUrl using: auth withHeaders: headers] whenTimeoutInSeconds: 20 do: [:toSig |
toSig exitWith: (SstError for: ExSstNonFatalError with: MxSSTG122)]
] on: Error do: [:sig | sig exitWith: (SstError for: ExSstNonFatalError with: sig exception description)].
(enphaseReply notNil and: [enphaseReply isSstError not]) ifTrue: [
enphaseStream := ReadStream on: enphaseReply contents asString.
reader := STONReader on: enphaseStream.
reader next ifNotNil: [:enphaseT | enphaseData := enphaseT].
enphaseData inspect.
] ifFalse: [enphaseReply inspect].
httpClient shutDown.

[Louis LaBrunda]

Hi Guys,

I'm still banging away on this without much joy.  I think I need to construct the data string in html form and not the json form I started with.  Is there anything in the Sst... world that will help with this?

Lou

[David Gregory]

Since you are setting the Content Type to 'multipart/form-data' that is what format you would need to supply the loginBody in.  You appear to be generating a JSON like string instead that definitely isn’t 'multipart/form-data'.

However looking at https://enlighten.enphaseenergy.com/login, it generates a different request to what you are generating via curl and sends it to https://enlighten.enphaseenergy.com/login/login.  It generates a ‘application/x-www-form-urlencoded’ message but with different parameters to your example.  The fields for the email and password are called ‘user_email’ and ‘user_password’ and there are lots of other fields such as authenticity_token.  The password appears to be encoded somehow.

Anyway, submitting a login to their login page shows the form urlencoded message for that end point (this was for an email of j...@test.com and password of 1234)

utf8=%E2%9C%93&authenticity_token=n4N3AqzBSXXtPvZnJwnoVRhf_1ZyBDsfdHmFX2u2ioQvXkPuekeif3Olal1Dw1_9D5-2Y4N-NlxiU7pAOxzFRw&user%5Bemail%5D=joe%40test.com&user%5Bpassword%5D=ab56b4d92b40713acc5af89985d4b786&secured_user=true&commit=Sign+In

 VAST will generate url encoded strings via String>>#sstUrlEncode

If the login.json end point really does take a 'multipart/form-data' message, it is a lot more complex to generate.  See https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types#multipartform-data for a sample of what it looks like.  VAST has a parser to read ‘multipart/form-data’ messages in SstHttpMultipartContentAssembler, but I’m not aware of a class that can generate them.

[Louis LaBrunda]

Hi Dave,

Thanks for the help.

Big Picture:  I have an Enphase IQ Gateway that controls the combining of the grid, solar panels and a battery.  The gateway has a local API that requires a token or bearer value.  I have one but they are only valid for a year.  I want to be able to get a new one from my program.  The manual way of obtaining one is to login to the web site and enter your gateway serial number.  Enphase documents automated ways of doing the same (see below) one of which is with some curl code.  The curl code does the login, saves the session id in a file and then uses it for the next step to pass the serial number and get the token.

My first step is to get the login to work.  I have stripped out everything else from the curl code and have this:

curl -X POST https://enlighten.enphaseenergy.com/login/login.json? -F "user[email]=LouLaB...@gmail.com" -F "user[password]=LouPassword"

Running the above, with my real password, logs in and even returns my current token.  The "-F" tells curl to treat what follows as html fields.  That is why I use Content-Type of multipart/form-data (HttpContentTypeMultipart).  I have attached a file (with the password changed) that curl made that should show what curl sent to the server.  I have extracted some of it here:

POST /login/login.json? HTTP/1.1..Host: enlighten.enphaseenergy.com..User-Agent: curl/8.7.1..Accept: */*..Content-Length: 300..Content-Type: multipart/form-data; boundary=------------------------PcWBhkMoVLPjCq2JSxhpQY....=> Send data, 300 bytes (0x12c)--------------------------PcWBhkMoVLPjCq2JSxhpQY..Content-Disposition: form-data; name="user[email]"....LouLaBrunda5@gmail.com..--------------------------PcWBhkMoVLPjCq2JSxhpQY..Content-Disposition: form-data; name="user[password]"....LouPassword..--------------------------PcWBhkMoVLPjCq2JSxhpQY--..

It does look like it is going to "/login/login.json?"  The reply looks like json and that is fine as I can handle it.  The Content-Type: "multipart/form-data; boundary=..." is multipart/form-data with a boundary specified.  I can figure out how to do that as I have done it before creating html email.  The double quotes " seem to be wrapping differently from the way they were entered with curl.  There also seem to be a lot of Cr/Lfs mixed in.  It doesn't look like the userid/password is encoded.  I think if they had spaces in them, they would need a base 64 encoded string.  I think curl can do that and we can in VA Smalltalk but it doesn't look like it is needed here.  I'm going to try to bruit force the data to look like what curl sent.  That will be tough as it will be trial and error with not real feed back as to why it doesn't work.  I'm certainly open to using whatever SstHttpClient and friends can offer in the way of help.  Below are the sample curl and python scripts that Enphase supplies to do the whole process of getting a token.

Lou

user=’<UserName>’
password='<Password>'

envoy_serial=’<Envoy_Serial_No>’
session_id=$(curl -X POST https://enlighten.enphaseenergy.com/login/login.json? -F "user[email]=$user" -F "user[password]=$password" | jq -r ".session_id")
web_token=$(curl -X POST https://entrez.enphaseenergy.com/tokens -H "Content-Type: application/json" -d "{\"session_id\": \"$session_id\", \"serial_num\": \"$envoy_serial\", \"username\": \"$user\"}")

import json
import requests
user=’<UserName>’
password='<Password>'
envoy_serial=’< Envoy_Serial_No>’
data = {'user[email]': user, 'user[password]': password}
response = requests.post('https://enlighten.enphaseenergy.com/login/login.json?', data=data) response_data = json.loads(response.text)
data = {'session_id': response_data['session_id'], 'serial_num': envoy_serial, 'username': user}
response = requests.post('https://entrez.enphaseenergy.com/tokens', json=data)
token_raw = response.text

[Louis LaBrunda]

Hi,

I can set breakpoints but when they trigger, they don't let me continue with the code.  Any ideas how to see what is being sent to the server?  I get sst error 153 with: SstHttpResponseHeader{ HTTP/1.1 400 Bad Request Status: 400 Bad Request...

Lou

[David Gregory]

Since you are setting the Content Type to 'multipart/form-data' that is what format you would need to supply the loginBody in.  You appear to be generating a JSON like string instead that definitely isn’t 'multipart/form-data'.

 

However looking at https://enlighten.enphaseenergy.com/login, it generates a different request to what you are generating via curl and sends it to https://enlighten.enphaseenergy.com/login/login.  It generates a ‘application/x-www-form-urlencoded’ message but with different parameters to your example.  The fields for the email and password are called ‘user_email’ and ‘user_password’ and there are lots of other fields such as authenticity_token.  The password appears to be encoded somehow.

 

Anyway, submitting a login to their login page shows the form urlencoded message for that end point (this was for an email of j...@test.com and password of 1234)

 

utf8=%E2%9C%93&authenticity_token=n4N3AqzBSXXtPvZnJwnoVRhf_1ZyBDsfdHmFX2u2ioQvXkPuekeif3Olal1Dw1_9D5-2Y4N-NlxiU7pAOxzFRw&user%5Bemail%5D=joe%40test.com&user%5Bpassword%5D=ab56b4d92b40713acc5af89985d4b786&secured_user=true&commit=Sign+In

 

VAST will generate url encoded strings via String>>#sstUrlEncode

 

If the login.json end point really does take a 'multipart/form-data' message, it is a lot more complex to generate.  See https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types#multipartform-data for a sample of what it looks like.  VAST has a parser to read ‘multipart/form-data’ messages in SstHttpMultipartContentAssembler, but I’m not aware of a class that can generate them.

 

 

 

From: va-sma...@googlegroups.com <va-sma...@googlegroups.com> On Behalf Of Louis LaBrunda
Sent: Thursday, August 15, 2024 6:03 AM
To: VAST Community Forum <va-sma...@googlegroups.com>
Subject: A little more SstHttpClient help please

 

EXTERNAL: Do not click links or open attachments if you do not recognize the sender.

--
You received this message because you are subscribed to the Google Groups "VAST Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to va-smalltalk...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/1ce8cb3c-1abc-4e15-ad27-7ae7e57d626dn%40googlegroups.com.

[Louis LaBrunda]

Hi Dave, Everyone,

I have the login half working.  I used multipart/form-data, built the login body with what it looked like curl was doing.  Thinks like 'Content-Disposition: form-data; name="user[email]"' and html boundary and dashes where they are wanted.  All very much like html email.

Now for the next step.  Keeping the session going and asking for the token.  Wish me luck.

Lou

[Louis LaBrunda]

Hi All,

Below is a workspace with my latest.  I know you can't test it because I changed the password.  The login part works.  Then it traps '(ExMessageNotUnderstood) An exception has occurred' doing the post to https://entrez.enphaseenergy.com/tokens.  It look like a message is being sent to nil.

Maybe someone will spot where I'm doing something dumb.

Enough for today, time for pizza.

Lou

"
EtWindow defaultEvaluationPools: nil.
EtWindow defaultEvaluationPools: (EtWindow defaultEvaluationPools asSet addAll: #(#SstConstants #SciSslConstants #SstHttpConstants #NlsCatSSTG)) asArray.
"

| httpClient loginUrl tokenUrl serialNumber bearer loginHeaders tokenHeaders enphaseReply
enphaseData enphaseStream reader sessionId user password auth boundary twoDashes message
managerToken |

user := 'LouLaB...@gmail.com'.
password := 'LouPassword'.
serialNumber := '202406001616'.
twoDashes := '--'.
boundary := '+++++ZZZZZ+++++'.

loginUrl := 'https://enlighten.enphaseenergy.com/login/login.json?'.

loginBody := WriteStream onStringOfSize: 100.
loginBody
nextPutAll: twoDashes; nextPutAll: boundary; cr;
nextPutAll: 'Content-Disposition: form-data; name="user[email]"'; cr; cr; nextPutAll: user; cr;
nextPutAll: twoDashes; nextPutAll: boundary; cr;
nextPutAll: 'Content-Disposition: form-data; name="user[password]"'; cr; cr; nextPutAll: password; cr;
nextPutAll: twoDashes; nextPutAll: boundary; nextPutAll: twoDashes; cr.

loginHeaders := LookupTable new.
loginHeaders
at: 'accept' put: HttpContentTypeJson;
at: 'Content-Type' put: (HttpContentTypeMultipart, '; boundary=', boundary).
httpClient := SstHttpClient forTransportScheme: 'httpsl'.
httpClient startUp.
enphaseReply := [
[httpClient post: loginBody contents typed: nil at: loginUrl using: nil withHeaders: loginHeaders] whenTimeoutInSeconds: 20 do: [:toSig |

toSig exitWith: (SstError for: ExSstNonFatalError with: MxSSTG122)]
] on: Error do: [:sig | sig exitWith: (SstError for: ExSstNonFatalError with: sig exception description)].

(enphaseReply notNil and: [enphaseReply isSstError not]) ifTrue: [
enphaseStream := ReadStream on: enphaseReply contents asString.
reader := STONReader on: enphaseStream.
reader next ifNotNil: [:enphaseT | enphaseData := enphaseT].
enphaseData inspect.

message := enphaseData at: 'message' ifAbsent: [].
] ifFalse: [enphaseReply inspect].

(message = 'success') ifTrue: [
sessionId := enphaseData at: 'session_id' ifAbsent: [].
managerToken := enphaseData at: 'manager_token' ifAbsent: [].

tokenUrl := 'https://entrez.enphaseenergy.com/tokens'.
tokenBody := '{"session_id": "%1", "serial_num": "%2", "username": "%3"}' bindWith: sessionId with: serialNumber with: user.

tokenBody inspect.

tokenHeaders := LookupTable new.
tokenHeaders
at: 'accept' put: HttpContentTypeJson;
at: 'Content-Type' put: HttpContentTypeJson.

enphaseReply := [
[httpClient post: tokenBody contents typed: nil at: tokenUrl using: nil withHeaders: tokenHeaders] whenTimeoutInSeconds: 20 do: [:toSig |

toSig exitWith: (SstError for: ExSstNonFatalError with: MxSSTG122)]

] on: Error do: [:sig | sig inspect. sig exitWith: (SstError for: ExSstNonFatalError with: sig exception description)].

(enphaseReply notNil and: [enphaseReply isSstError not]) ifTrue: [
enphaseStream := ReadStream on: enphaseReply contents asString.
reader := STONReader on: enphaseStream.
reader next ifNotNil: [:enphaseT | enphaseData := enphaseT].
enphaseData inspect.

" message := enphaseData at: 'message' ifAbsent: []."
] ifFalse: [enphaseReply inspect].

].
httpClient shutDown.

[Louis LaBrunda]

Hi,

I removed the capturing and ignoring of errors so that I could see the actual problem.  First dumb mistake, #tokenBody is a string not a stream I don't want its #contents, just the string.

Now I get an 'Invalid server address' error.  The sample curl and python code both show logging into: 'http://enlighten.enphaseenergy.com/login/login.json?' (which I now have working with https:..) and then going to: 'http://entrez.enphaseenergy.com/tokens', passing it the session id from the login.

It seems odd to me that you can login to one site and than jump to another with the session id?  Also, the original login was with http:.. and I kept getting permanent forwarding complaints to the https:.. version of the site.  I changes to going strait to the https:.. version.  I will try letting the Smalltalk code do the forwarding but I can't think of how that extra work would help.

Lou

[Louis LaBrunda]

Hi All,

Success!  I went back to using the https://... urls and I had to modify base Smalltalk code to stop it from throwing and error that I guess really isn't an error.  I will make another thread post to explain that in the hope that someone from Instantiations will see it and take note.

Lou

[David Gregory]

" It seems odd to me that you can login to one site and than jump to another with the session id?"

It is not really a session id.  Using signed tokens is now very common and is the basis for all the "Sign in with Google" or "Sign in with Apple" and anything OAuth.  You signin to the identity provider (eg google or apple or ...) which issues you an application specific digitally signed token which you then supply to the server you want to access as 'proof' of authorization.  The server checks the signature of the token is correct (to ensure it is a token issued by an approved identity system) and allows access.  Most tokens are time limited in validity so you have to get a new access token every now and then.

While tokens are signed, they are not encrypted, so it is quite insecure to send them over http.  Most sites using tokens will require https.

[Louis LaBrunda]

Hi David,

Thanks for the info.  I changed everything to https because the http site was forwarding it and going with https was just easier and of course safer.  As for calling it "session id" (which didn't make sense to me in the first place) that is what the Enphase site calls it in their return json dictionary.  Your explanation make more sense.

Lou

[Louis LaBrunda]

Hey Dave,

Given your explanation of what the so called  "session id" is used for, would you say my reasoning about the test in #SstHttpClient>prepare:andConfigureFor: of the endpoints being the same is a problem?

Lou

[David Gregory]

I would not consider SstHttpClient 'reusable' between URLs to different hosts.  If you use the 'server' transport schemes (ie 'https' rather than 'httpsl') then each SstHttpClient takes a connection from the connection pool for the duration of its life and thus you create an SstHttpClient, use it, then shut it down to release the connection back to the pool so that the connection can be used again by a different SstHttpClient.

In the 'light' transport schemes, the life of the SstHttpClient is the same as the socket it holds (each SstHttpClient has its own socket in 'light' mode).

Anyway, I'd create separate SstHttpClient objects for the authentication and then for the subsequent access to the resource server.

[Louis LaBrunda]

Hi Dave,

I liked what you said in your last post very much.  No one likes being wrong.  If I'm wrong, I like being told I'm wrong and will change my mind if the facts warrant it.  Given what we both said about the seemingly misnamed "session id", I thought that it led me to think that the two steps were part of one session.  And, one session should be one connection.  So, I made the second request with the same SstHttpClient.  That led to the error of the second endpoint not being the same as the first.  That led me to remove the test.  That works.

After reading your post, I thought okay, I got this wrong.  Make one connection, get the wrongly named "session id", make another connection, get the result.  Well that doesn't work.  I get the error below.  I tried shutting down the first, not shutting down the first, wrapping it all in #critical, using 'httpsl' and 'https'.  Some attempts did get other errors but this one is with the smallest change to what I was doing.  Namely, use two 'httpsl' SstHttpClients.

Note: This is a Seaside program and the call to this code is triggered by clicking a button on the Seaside drawn web page.

Any idea what I may be doing wrong?

Lou

Error String: 'SstSendError(''SSL handshake failed: INTERNAL_ERROR (167772427): Unknown error
OpenSSLError
Error Code: 167772427
Error Object: (''''wrong version number'''')
Error String: ''''error:0A00010B:SSL routines::wrong version number''''
Error Hint: ''''SSL routines:wrong version number''''
AuxiliaryData: nil'')'
Resumable: false

Process
Name: Dispatch worker: 17478
Process State: suspended
Priority: 3

[Mariano Martinez Peck]

Hi Louis,

Please send the full stacktrace. 

To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/ff822136-2abf-488b-8d20-4ca311ae7b36n%40googlegroups.com.

--

Mariano Martinez Peck VAST Team Lead Senior Software Engineer  mp...@instantiations.com  @MartinezPeck  /mariano-martinez-peck  instantiations.com

[Louis LaBrunda]

Hi Mariano,

Attached.

[David Gregory]

In this case you have requested an https based transport but are using an http url (http://entrez.enphaseenergy.com/tokens)

SSL is reporting 'wrong version' because the other end didn't actually respond with a valid SSL handshake (as it is a non-ssl URL).

[Louis LaBrunda]

Hey Dave,

That was indeed the problem.  Thank you very much.  I need to be more careful.  Part of the problem, besides me not paying close enough attention, is that all of the examples I have (curl and Python) are of the non-secure variety.  Funny thing, the code where I removed the endpoint test, worked with the first call being https and the second being http.  I guess it just has to do with the order of what and when things get checked.  Anyway, thanks again.

Lou