Streaming upload via fetch()

[Yutaka Hirano]

Recently we had an internal discussion about streaming upload with fetch API. It seems there are some use-cases, but we think it's good to have more use-cases / requests. Hence I'm starting this public thread.

We'll appreciate your feedback.

To be clear, what we are talking about does not provide a bi-directional streaming (i.e., the request body stream needs to be finished before getting a response) due to implementation difficulties.

Eric, on the GitHub issue David Fowler, a person from Microsoft, seemed interested in the feature. Could you cc that person please?

Thanks,

[nwt...@gmail.com]

Hi, developers. I'm looking forward to using streaming upload with fetch API.

My use-case is to use it with Piping Server, which is a streaming data transfer server over HTTP without WebSocket or WebRTC. The server is designed by using stream, so streaming upload with fetch API is a perfect fit for the server.

The following README and posts describe what Piping Server is. They are almost the same, but the Japanese one is more informative.

• GitHub README 
• English post: Data Streaming between Every Device over HTTP/HTTPS
• Japanese post: https://qiita.com/nwtgck/items/78309fc529da7776cba0

In the posts, curl command is used frequently for streaming upload via Piping Server. However, if streaming upload with fetch API is available, you can upload data with the following processings in browser-side.

• streaming compression
• streaming-encryption
• archiving some files

They are not only for Piping Server, but those kinds of streaming processings can also be utilized in many applications and the processing can be composed together easily.  I think streaming processings are highly efficient in terms of both time and space (memory). In addition, some streaming features below in browsers can be combined together.

• screen capture
• voice recording

I think the streaming processings above and streaming data collection can be also composable. Thus, streaming upload with fetch API provides high composability and efficiency.

Best regards,

Ryo Ota

2019年9月3日火曜日 18時41分16秒 UTC+9 Yutaka Hirano:

[br...@audiopump.co]

Internet Radio Source Streaming

SHOUTcast and Icecast servers have been supporting internet radio streams for more than 20 years, with video support added since then.  For Icecast and compatible servers, the source stream for these broadcasts come from normal HTTP PUT requests.  The request headers are sent, the server sends an HTTP/1.1 100 Continue, and the encoded media data is sent live as it's being recorded.

Since the addition of the Media Devices API to browsers I (and others) have created web-based "source clients" which allow captured to be sent to these streaming servers.  The problem is that browsers do not allow a streaming request body.  Therefore, the only way to get the data out is to have a Web Socket server that proxies the data from the browser on to the streaming server.  It would be much better if the client could connect directly to the streaming server, for reliability and efficiency.

Segmented streaming to regular HTTP servers could benefit as well.  Suppose you are encoding video to DASH or HLS segments in-browser with the Media Recorder API, and uploading them via a series of HTTP PUT requests.  If you are creating longer segments, such as 8-seconds duration, it would be beneficial to upload these segments as they are being encoded.  If the segment must be finished encoding before the upload, 8 seconds of latency is added end-to-end on top of the latency already required for this style of streaming.

We could do away with a lot of hacks and unnecessary infrastructure if we could make a standard HTTP request with a streaming request body from a browser.

Thanks,

Brad Isbell

AudioPump, Inc.

[wen...@google.com]

Streaming upload support via fetch() will be extremely useful for grpc-web (https://github.com/grpc/grpc-web)

gRPC is a general-purpose RPC framework with cross-language support. Today, we can't have a browser client directly communicate with an end-point (API) that includes any client-to-server streaming method.

Using websockets will require extensive protocol translation, and will also make the client side implementation much more complicated, e.g. HTTP fallback, flow-control etc.

[jaffat...@gmail.com]

Doesn't grpc depend on bi-directional streaming?

[Wenbo Zhu]

On Sun, Sep 15, 2019 at 12:28 AM <jaffat...@gmail.com> wrote:

Doesn't grpc depend on bi-directional streaming?

https://github.com/grpc/grpc-web currently supports only server-streaming (via xhr or fetch).

[konrad.m...@gmail.com]

Hi,

just stumbled across this thread in a StackOverflow question.

I have a use-case at the company I work at where we have a JavaScript client that is supposed to update the firmware of industrial color sensors devices in the local network. This client has to talk to two APIs, the firmware delivery server API and the firmware API of the individual sensors. The firmware images are around 100MB so it’s not feasible to put them in memory. Basically what I would want to work is:

fetch('FIRMwARE_IMAGE').then(res => fetch('SENSOR_FIRMWARE_UPDATE_ENDPOINT', { method: 'POST', body: res.body }))

As far as I can see from the spec this should work as-is (despite the fact, that we would probably implement some kind of progress logic in between), but does not because streaming uploads are not implemented yet.

In some cases we were able to work around this, by implementing the update logic in a separate backend software on the machine running the client (imagine a dedicated sensor-management computer). But sometimes this machine is actually the sensor itself, as they have their own web interface which users can use to administrate the sensor. In these cases we cannot offload the update logic to the sensor because the network connectivity of the sensor may not allow access to the firmware server, but the machine the browser is running on does. The problem get’s even more complex if the sensors have enabled user management and restrict API access, as the update-backend now needs credentials to perform updates that have be shared from the client.

Having streaming upload would drastically reduce this complexity.

Kind regards

Konrad Mohrfeldt

[guest271314]

This provides a means to perform fetch streaming upload using Nodejs with Express https://glitch.com/edit/#!/fetch-request-stream. Is there equivalent code for Python and PHP?