New ArduinoHttpClient library

[Sandeep Mistry]

Hi Everyone,

I've been working on a new library called ArduinoHttpClient, that is based on Adrian McEwen's HttpClient library. The library supports HTTP (GET, POST, PUT, DELETE) requests and also includes a WebSocket client. Both can be used with any networking library that provide a device specific Client, such as WiFi101 and Ethernet.

There is an initial beta release available in Arduino IDE Library Manager (search for "ArduinoHttpClient") and source code can be found on Github. Massimo and Tom thought the developer mailing list would be a good place to get early feedback. Please try it out if you are interested, and provide any feedback or suggestions you have on Github.


Thanks,
Sandeep

[Paul Stoffregen]

I'm looking at it now, just a quick peek.  Hope to get some time to actually use it next week.

Is there an API reference or guidance on usage?  The header file has comments which appear to be meant as the documentation and I see there's several examples, but is there anything else like a file that's meant for novice users to understand how to use it?

Looking at the implementation very briefly, my main concern is the use of busy loops and delay() to wait for incoming data.  The blocking approach is simple, and probably makes a lot of sense for this library to be used by sketches.  But consider the case of this library being used by other libraries, to implement protocols which are built on top of HTTP.  Any blocking inside the library will force any other higher level protocol libraries build on top of this to also be blocking.

My hope is you might adopt Arduino's usual convention of available() and read(), rather any code which causes blocking for incoming data.  For the status code, rather than waiting, available would return 0 or 1 depending on whether it's arrived, and read would return -1 if the status code hasn't arrived yet.

Likewise for skipping past the unwanted headers, perhaps an available & read pair could be provided to read the response body.  If the user's desire is to ignore the rest of the headers, instead of the blocking skipResponseHeaders(), the available function for the body would look at the internal state and discard the rest of the headers, without blocking, and return 0 if none of the body data is available yet.  Following the normal convention, a read function for the body ought to return what's currently received.

The responseBody() function seems to try to do this now.... but another possible issue is an attempt to allocate a String large enough to hold the entire expected document.  That probably makes sense for a very limited class of applications where a server returns a short body... but maybe not a good plan for a general purpose HTTP library where most Arduino compatible boards have less RAM than the typical size of most HTML documents.  I see there's also read() functions, so maybe responseBody is meant to be used only as a convenience function?  Maybe I'm overly paranoid, but this worries me.  Maybe it could lead to Arduino users unwittingly crafting projects with security vulnerabilities, where a server could return a longer response that causes a crash or incorrect behavior?  I see several of the examples use responseBody in the simplest way possible, without checking how much data is expected, or whether their one and only call to responseBody actually returned all the body's data.  Or maybe this is only used for cases where the client fully trusts the server?

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

[Gabriel Staples]

I agree with Paul about the blocking concern. As I write new libraries I'm going through great effort to ensure they are not blocking. So many Arduino libraries out there become unusable in many applications if they are blocking, and they have to be rewritten to become nonblocking and usable with other code.

As a standalone single-purpose device blocking is often ok, but the second you start merging in a lot of functionality and multiple libraries, it becomes very important that each is nonblocking.

Sincerely,

Gabriel Staples
www.ElectricRCAircraftGuy.com

(sent from my Android)

[Massimo Banzi]

Paul

Great feedback as usual! we’ll try to incorporate as much as possible.

We announced the library here exactly because we wanted to work with you guys to create a standard http library for Arduino that can become a solid building block for the whole community.

The library itself is derived from the heroic work that Adrian McEwen did for the Arduino Ethernet shield so some solutions are still from an era in which 1k of ram was luxury :)

We believe that nowadays trying to do real IOT applications with 1K of ram is a little bit painful and we wanted to make some API easier to use at the expense of more RAM and code space.

Please help us by sending us pull requests and Sandeep will incorporate them asap.

We’ll work on the docs in the meantime :)

thanks!

m

--

Massimo Banzi <m.b...@arduino.cc>

Arduino LLC

[Andrew Kroll]

Same reason I still use my own networking libs, which follow standards that everyone uses, and doesn't block unless you pass the option to do so. This actually makes more sense when you think about it. Why make 2 calls
Sure, you could stil isAvaiable() call, it may have a few corner case uses...

Instead of:
1: is there something there?

2: Ok, there is go get it! (or not...)

v.s.

1: Return zero length if nothing is there

or

1: Return after something arrives

Lets say it currently looks like:
int readstuff(char *buffer, int bufferlength) { ... }

You modify the method call to look something like:

int readstuff(char *buffer, int bufferlength, bool block=true) { ... }

That way you know exactly what you will get when you ask for it... plus you could provide a default on the blocking as true to the existing method, and not break old code.

How's that for having your cake and eating it too?

Visit my github for awesome Arduino code @ https://github.com/xxxajk

[Massimo Banzi]

Thanks for the comments and the suggestions.

Who would like to help us out and test it / help document it/ send pull requests?

m

--

Massimo Banzi <m.b...@arduino.cc>

Arduino LLC

[Paul Stoffregen]

On 07/07/2016 08:12 AM, Andrew Kroll wrote:

Why make 2 calls

Regarding decisions about how APIs should be designed for Arduino libraries....

https://www.arduino.cc/en/Reference/APIStyleGuide

[Andrew Kroll]

Sure, I know this...but why not expand on it... for libraries you could exploit the more efficient way, that's my point. ;-)

[Sandeep Mistry]

Thanks for the feedback so far!

Regarding blocking concerns, what are your thoughts on using httpClient.setHttpResponseTimeout(0) or httpClient.setTimeout(0) to activate non-blocking mode? This would allow more complicated use cases to manage their own timeout mechanism externally, while maintaining the simpler API. As Paul suggested a negative value would be returned to indicate a timeout.

I like Paul's suggestion of using available() to skip response headers as well. I can work on this after there's a consensus on the non-blocking topic. 

Yes, responseBody() is mean't to be used as a convenience function. I'll work on a pull request to make it safer, and return an invalid string if memory allocation fails. All the example sketches are pretty basic at the moment and use responseBody(), we need add some more advanced examples that use the Stream API's to process the response body.

[Peter Goderie]

Regarding blocking concerns:

I would prefer the use of a well defined callback interface class (let's call it iHttpClientCallbacks)

and a function like:

void setCallbacks(iHttpClientCallbacks* callbacks, uint32_t enabledCallbacksMask);

After calling setCallbacks all functions with corresponding callback(s) enabled would automatically become non-blocking and would call the corresponding callback(s) instead.

This would also still allow you to set a timeout value, only now it would not block but the callback would be called on timeout.


Op dinsdag 12 juli 2016 19:00:05 UTC+2 schreef Sandeep Mistry: