video streaming with Firefox behind a load balancer

[Austin]

Hello Sakai Devs,

Just wondering if anyone has seen any issues with viewing/streaming video via the content resources tool URL in Firefox 60+ behind a load balancer? e.g.

/access/content/group/abc1234d-a123-456c-7e89-123b552b6733/MyVideo.mp4

What's happening is that "some" .mp4 videos will fail with a "Connection Failed" error in Firefox after playing for around 45s.

This problem started happening on our test system after we configured our load balancer to route incoming and outgoing traffic through the load balancer.  Previously, outgoing traffic was sent directly back to the client and the problem did not happen.

I suspect that there's some combination of Firefox, Sakai, and the new Load Balancer config. that's causing the connection to fail mid stream.

However, what's complicated is that in the past when Firefox 60 first came out, this problem also happened when the .mp4 file was served by apache only and Sakai was not involved at all.  But with the most recent Firefox 65.0.2, when the file is served by apache, the problem went away, so Firefox must have improved something in newer versions.  But... the problem still happens with the lastest FF when the file is served up by Sakai through the load balancer.

Also, we're using Apache / modjk / tomcat but I've confirmed that those are not involved by running a vanilla tomcat project (no Sakai code at all) with the same apache / modjk config and the videos don't fail.  Also note that with this basic setup, the older FF 60.0 also still fails, so that indicates as I mentioned before that FF must have improved something with 65.0.  But unfortunately that leaves the Sakai code as being involved.

What's even stranger is that the problem happens with some .mp4s and not others.  In particular .mp4s downloaded from youtube fail.  But mp4s I created with DaVinci Resolve did not fail.   But what I noticed is that the failing .mp4s seem to fail while streaming (I think), but the videos I created, appear to be completely downloaded before it plays, which I suspect is why they don't fail.

What looks like is happening, is that both files are sent to

BaseContentService.java

protected IOException copyRange(InputStream istream, OutputStream ostream, start, long end)

and at some point the mp4 from youtube fails in

if (bytesToRead >= len) {

    ostream.write(buffer, 0, len);

    bytesToRead -= len;

}

where the ostream.write() fails and throws an error

org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe

Also, further up the chain (in BaseContentService.java) there's some code that sets the response headers to Accept-Ranges: none 

e.g. there's a comment

// If there is a direct link to the asset, no sense streaming it.

// Send the asset directly to the load-balancer or to the client

which I think is also involved.  But what's stranger, is that despite this header, the failing mp4 still appears to stream... at least it plays right away when it's loaded.  vs. the mp4 I created will show a spinner icon while the video is loading before it plays.

(Another note, in Chrome, it will make an initial request where Sakai will set the Accept-Ranges: none, but then Chrome makes another request right away that contains a "Range" header, so Sakai will instead set the response header to Accept-Ranges: bytes and the video streams just fine.)

Thanks and sorry that this is so complicated, but it's been quite an interesting problem!  But I'm still confused about where the main point of failure is... the Sakai streaming code, Firefox, or the Load Balancer?

- Austin

[Austin]

Hello Sakai Devs,

Just wondering if anyone was able to look into this?  I still believe that there's some incompatibility between Sakai's streaming code, Firefox, and our Load Balancer.  But I still don't know what exactly the root cause is yet.  I still think it has do with this code in BaseContentService.java and in LessonBuilderAccessService.java because it looks like lesson builder's code is copied from BaseContentService

// If there is a direct link to the asset, no sense streaming it.

// Send the asset directly to the load-balancer or to the client

URI directLinkUri = m_storage.getDirectLink(resource);

ArrayList<Range> ranges = parseRange(req, res, len);

if (directLinkUri != null || req.getHeader("Range") == null || (ranges == null) || (ranges.isEmpty())) {

res.addHeader("Accept-Ranges", "none");

Because if I run it through lesson builder and I "embed" the video on a lesson builder page, the video works fine, where it does NOT set the Accept-Ranges to "none".  Instead, further down the in the code, it sets it to "bytes".  But if I take the same file and "link" to it in a lesson builder page, then click on the link, the Accept-Ranges gets set to "none" and it fails.  However, it's still strange that only some videos exhibit the bad connection problem.  For the videos that don't fail, they also get their Accept-Ranges set to none, but again, it appears that those videos get fully downloaded before playing vs. streamed in parts for the failing videos.  I couldn't see any obvious differences in the Sakai code, so maybe the coyote streaming library does something funky depending on the length, bitrate, or encoding?

Also, what is the purpose of setting the Accept-Ranges to "none"?  I mean besides what this mozilla doc says

"No range unit is supported, this makes the header equivalent of its own absence and is therefore rarely used, though some browsers, like IE9, it is used to disable or remove the pause buttons in the download manager."

Is there any particular reason it's being set to 'none' in Sakai?  was that supposed to make the browser download the file instead of play it?  maybe newer browsers will always try to play the video despite that header?

Thanks,

Austin

[Sam Ottenhoff]

Because if I run it through lesson builder and I "embed" the video on a lesson builder page, the video works fine, where it does NOT set the Accept-Ranges to "none".  Instead, further down the in the code, it sets it to "bytes".  But if I take the same file and "link" to it in a lesson builder page, then click on the link, the Accept-Ranges gets set to "none" and it fails. 

Accept-Ranges only gets set to "none" when the user request does not contain a Range.

 

However, it's still strange that only some videos exhibit the bad connection problem. 

Narrowing this down to a repeatable pattern so it can be filed as a JIRA is key.

 

For the videos that don't fail, they also get their Accept-Ranges set to none, but again, it appears that those videos get fully downloaded before playing vs. streamed in parts for the failing videos.  I couldn't see any obvious differences in the Sakai code, so maybe the coyote streaming library does something funky depending on the length, bitrate, or encoding?

Sakai doesn't stream, but it does respond to Range requests. I don't think Tomcat is relevant here. Your load balancer may be relevant.

 

Also, what is the purpose of setting the Accept-Ranges to "none"?  I mean besides what this mozilla doc says

If the client doesn't make a range request, Sakai code will send the content in one shot. Handling range requests is intensive: it requires loading a file into memory to find a specific range of bytes before another HTTP request is sent for a distinct set of bytes.

Is there any particular reason it's being set to 'none' in Sakai?  was that supposed to make the browser download the file instead of play it? 

Inlining vs downloading is a separate subject: content.mime.inline https://github.com/sakaiproject/sakai/blob/master/lessonbuilder/README#L598

[Stephen Marquard]

Sakai's range support originally comes from the Tomcat code:

https://jira.sakaiproject.org/browse/SAK-39673

It may be worth checking to see how that code in Tomcat has evolved, and if there are changes or bugfixes that should be ported into Sakai.

Cheers

Stephen

---

From: 'Sam Ottenhoff' via Sakai Development <saka...@apereo.org>
Sent: Friday, 29 March 2019 4:17 AM
To: Austin
Cc: sakai-dev
Subject: Re: [sakai-dev] Re: video streaming with Firefox behind a load balancer

 

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sakai-dev+...@apereo.org.
To post to this group, send email to saka...@apereo.org.
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.

Disclaimer - University of Cape Town This email is subject to UCT policies and email disclaimer published on our website at http://www.uct.ac.za/main/email-disclaimer or obtainable from +27 21 650 9111. If this email is not related to the business of UCT, it is sent by the sender in an individual capacity. Please report security incidents or abuse via https://csirt.uct.ac.za/page/report-an-incident.php.

[Austin]

Hello Sakai-Devs,

Sorry, to bring up an old thread.

After the original conversation, we found that the Load Balancer was failing on a "Zero Window Timeout".  We increased the timeout and the problem went away for a while, but we've recently run into the problem again.  This time, it seems like the error happens with longer videos 10+ minutes. (previously we saw it with < 5m videos)

It was originally asked if it was reproducible.  At the time, it might have been difficult if you weren't running the same LB as us.  Today, you'd still need the LB to reproduce the error exactly, but now, I've found that a similar error happens without a LB.  The browser will overlay a "network connection error" message over the video, but the audio will continue to play.

I still feel that "Accept-Ranges: none" has something to do with it.

"Accept-Ranges only gets set to "none" when the user request does not contain a Range."

"If the client doesn't make a range request, Sakai code will send the content in one shot"

I still think that's a factor because.  If a video is served directly by apache or tomcat (e.g. not through Sakai), and the initial request does not contain a range, those web servers respond with "Accept-Ranges: bytes"... perhaps that's a new response in newer versions of tomcat/apache? Should Sakai now also respond with 'bytes'?... maybe it depends on the content-type? e.g. video?

As far as Chrome and Safari, the problem still doesn't happen there because I'd guess that it's ignoring the "accept-ranges: none" and makes subsequent requests using ranges anyway.

also we are using

content.mime.inline.count = 7
content.mime.inline.1=video/mp4

.

.

.

Should I JIRA this?  Or is the error is on the LB or browser (Firefox) side?  Maybe firefox ought to ignore 'accept-ranges: none', the way chrome and safari ignore it?

Thanks,

Austin

[Stephen Marquard]

Hi Austin

The Sakai code that handles accept-ranges originally comes from Tomcat, and it's possible that the Tomcat code has evolved to handle browser changes or become more correct over time, and that Sakai's equivalent code should also be updated to match.

I'd suggest reopening SAK-41585 with this new info.

Cheers

Stephen

---

From: Austin <aust...@hawaii.edu>
Sent: Friday, 25 September 2020 04:07
To: Stephen Marquard <stephen....@uct.ac.za>
Cc: Sam Ottenhoff <otte...@longsight.com>; sakai-dev <saka...@apereo.org>

[Austin]

Thanks, Stephen!  I've reopened https://jira.sakaiproject.org/browse/SAK-41585 with the latest info.

[Miguel Pellicer]

I remember including support to these requests for the name pronunciation feature, using Spring:

https://github.com/sakaiproject/sakai/pull/7211/files

Do you think is related to this issue?

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/sakai-dev/CAKL84%3DkjtFHbx%2BsMgrDfJjBGTSpNKpJp9av%3DFWXZsiGt3AHdGA%40mail.gmail.com.

-- 
-----------------------------------------------
Miguel Pellicer
CTO at EDF

Website: https://www.edf.global
LinkedIn: https://www.linkedin.com/in/mpellicer-edf
Office Phone: +34 - 96 381 35 75
Request a meeting here: https://calendly.com/mpellicer
-----------------------------------------------

[Austin]

Thanks for the suggestions Earle!

I was wondering... I'm still not able to reproduce this on the nightly servers.  I don't know if they are behind a Load Balancer, but actually, I don't think that matters.  Without the LB, I'm still seeing a similar error (SAK-41585).   But I don't see that error on the nightly servers.  What is the hardware setup on the nightly servers?  LB  -> apache -> tomcat?  I tried running with tomcat only, but the problem still happened.  So I was wondering if there are other hardware config differences that could affect that?  Or maybe our servers are just really fast (hahaha) and pushing the data to the Firefox faster than Firefox can handle it?  Or maybe my location from the nightly servers is far enough away (I'm in Hawai'i) that the stream isn't too fast for Firefox?

On Thu, Oct 15, 2020 at 10:55 AM Earle Nietzel <ea...@longsight.com> wrote:

This was on my list of todo's!

No matter how many times we say "Sakai is not a streaming service" people still upload media. IMHO for well known media types we should offer minimal streaming capabilities.

Here is what I was thinking of, For media types like audio and video by default when a user clicks them is to stream them and offer the option for downloading, and yes we would want to use springs streaming api vs rolling our own, which the PR that Miguel references does.

There are likely some downsides to doing this that are not well known like: the overhead that Sam already mentioned, tying up threads for long periods of time so it needs to be well understood and probably configurable.

-earle

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/sakai-dev/e6e4c5b0-3095-e818-a6f7-900ede64110f%40entornosdeformacion.com.

[Earle Nietzel]

This was on my list of todo's!

No matter how many times we say "Sakai is not a streaming service" people still upload media. IMHO for well known media types we should offer minimal streaming capabilities.

Here is what I was thinking of, For media types like audio and video by default when a user clicks them is to stream them and offer the option for downloading, and yes we would want to use springs streaming api vs rolling our own, which the PR that Miguel references does.

There are likely some downsides to doing this that are not well known like: the overhead that Sam already mentioned, tying up threads for long periods of time so it needs to be well understood and probably configurable.

-earle

On Mon, Sep 28, 2020 at 3:44 AM Miguel Pellicer <mpel...@entornosdeformacion.com> wrote:

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/sakai-dev/e6e4c5b0-3095-e818-a6f7-900ede64110f%40entornosdeformacion.com.

[Sam Ottenhoff]

Nginx proxying Tomcat. No other load balancer.

$ curl -I https://trunk-mysql.nightly.sakaiproject.org/portal/
HTTP/2 200
server: nginx/1.18.0 (Ubuntu)
date: Thu, 15 Oct 2020 23:20:33 GMT
content-type: text/html;charset=UTF-8
set-cookie: SAKAIID=4f387a6e-0d9e-483a-b23c-feaddbd2e704.ip-172-31-7-189; Path=/; Secure; HttpOnly; SameSite=Lax
expires: Wed, 16 Oct 2019 23:20:33 GMT
last-modified: Thu, 15 Oct 2020 23:20:33 GMT
cache-control: no-store, no-cache, must-revalidate, max-age=0, post-check=0, pre-check=0
pragma: no-cache
strict-transport-security: max-age=314159265359

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/sakai-dev/CAKL84%3Dkm_xGcT5yY1a0cUUR9tWiUU_-ZLpUNAAQzT%3DHtB8Jg3Q%40mail.gmail.com.