httpc:request pipelining example and blocking problem

[Anthony Howe]

I'm trying to exclusively use inets httpc for some simple REST request. Please
do NOT suggest other Erlang HTTP clients; the exercise is to only use inets
httpc. The docs for the httpc client are pretty poor and online searches
haven't helped.

Two questions:

1. Does anyone have an example of using inets httpc to do pipeline requests?

2. I've managed a single asynchronous request to a Ruby based service which
worked fine. Now when I try make a request to an Erlang Cowboy based service
(an old Erlang 19 3rd party application), the request goes through and the reply
sent (as observed using tcpdump), but the process blocks until the timeout,
never returning the response.

--
Anthony C Howe SnertSoft
ach...@snert.com Twitter: SirWumpus BarricadeMX & Milters
http://snert.com/ http://nanozen.snert.com/ http://snertsoft.com/

[Yao Bao]

Hello,

It is good to see users of inets still exists even though for exercises. And
luckily, it still exists within Erlang/OTP by default.

Tried inets httpd to build a simple http server, it works fine. Someday I
might finally realize some advanced features of a modern http server
provides is essential to my application, but I prefer starting from basic/poor
things.

For your question:
1. Haven't use inets httpc, so no advice personally.
2. If it worked with Ruby based service but not Erlang 19 based Cowboy based
service, the problem might be the communication. If the http request and response
works fine, and the process just blocks, I would guess the problem comes from the
"driver" of the inets httpc, it receives the http message, and turn it into an Erlang
message, might be the Erlang message which been putted into process's mailbox
does not match any of the designed pattern, so the "receive" expression blocks and
timed out. Or, the http driver does not send message back to the client process for
some reason.

More information about your situation would be helpful for us to find the problem:
- which version of inets httpc (and version of Erlang/OTP) do you use?
- which version of Cowboy (you mentioned Erlang 19) do you use?

The source code of inets httpc is not huge, if you want to know where exactly the
message is lost, give it a try to dig the source code.

Cheers,
Yao

[Vance Shipley]

Anthony,

The trick is to use a unique Profile (e.g. foo) for all the requests
to the pipelined server.

inets:start(httpc, [{profile, foo}]),
httpc:set_options([{keep_alive_timeout, 4000}], foo).

You MUST set the keep alive timeout. The server settings must also be
compatible.

--
-Vance

[Anthony Howe]

On 2021-12-17 20:19, Yao Bao wrote:
> Hello,
>
> It is good to see users of inets still exists even though for exercises. And
> luckily, it still exists within Erlang/OTP by default.

Well partially an exercise to remain light weight and keep it basic because that
should be all that is needed, but it is also a work thing.

> Tried inets httpd to build a simple http server, it works fine. Someday I
> might finally realize some advanced features of a modern http server
> provides is essential to my application, but I prefer starting from basic/poor
> things.

Well this particular task I need both httpd and httpc. The small REST server
part worked fine, despite the poor documentation. Took me a while to realise
you need to add your own modules to the list, but once past that it was fine.

> For your question:
> 1. Haven't use inets httpc, so no advice personally.
> 2. If it worked with Ruby based service but not Erlang 19 based Cowboy based
> service, the problem might be the communication. If the http request and response
> works fine, and the process just blocks, I would guess the problem comes from the
> "driver" of the inets httpc, it receives the http message, and turn it into an Erlang
> message, might be the Erlang message which been putted into process's mailbox
> does not match any of the designed pattern, so the "receive" expression blocks and
> timed out. Or, the http driver does not send message back to the client process for
> some reason.

Was working on this more last night.

* One part of the issue was the server was sending a HTTP chunked response and I
needed to switch to using `stream`.

* The other issue is I'm using a `gen_server` to drive part of the client aspect
and I needed to restructure how I setup the receive loop. Now I get the 1st
chunk (more often), but still appears to block on occasion on the chunk.

> More information about your situation would be helpful for us to find the problem:
> - which version of inets httpc (and version of Erlang/OTP) do you use?

This testing tool is using Erlang 21 (prebuilt Ubuntu 20.4 package).

> - which version of Cowboy (you mentioned Erlang 19) do you use?

The backend is using Erlang 19 and a similarly old version of Cowboy. Not sure
of the version. The backend I don't have much influence over (yet), but it
works in production and the client doesn't want to fiddle.

> The source code of inets httpc is not huge, if you want to know where exactly the
> message is lost, give it a try to dig the source code.

I was hoping to avoid that, but alas.

Anthony

[Yao Bao]

> * One part of the issue was the server was sending a HTTP chunked response and I
> needed to switch to using `stream`.
>
> * The other issue is I'm using a `gen_server` to drive part of the client aspect
> and I needed to restructure how I setup the receive loop. Now I get the 1st
> chunk (more often), but still appears to block on occasion on the chunk.

What if turn the “switch to using stream” off, can you consistently get the chunk?

Cheers,
Yao