Inquery about trigger setting

[안봉봉]

Hello.

I have a question about using mist server.
I would like to configure CDN by configuring the origin and edge servers.

The triggers I want to create are as follows. When transmitted to the origin server, the trigger detects it and pushes it to the edge server.

I used the "STREAM_PUSH" trigger and put rtmp://[Edgeserver IP]:1935 in the Handler part.

But it's not being transmitted normally, should the same stream be created in the stream item of the edge server?

If so, I wonder how I should input the stream source part.

Thank you.

[Balder Vietor]

Hello, 

The trigger STREAM_PUSH is a trigger meant to verify whether an incoming push should be allowed to continue. It isn't meant to redirect the stream. The trigger that would be able to redirect a stream is the PUSH_REWRITE trigger, however this one only redirects to a different stream name within the MistServer receiving the push. The usual functionality here is to accept a stream name using a Token and then rewrite the stream to a more human friendly name instead of the token being the stream name. Example here: https://news.mistserver.org/news/114/Simple+token+support+for+live+streams

The only thing you would need for the origin server to automatically push towards the edge server is to set up an automatic push. You can do so at the push panel & just need to select the stream and the target it needs to replicate the stream towards.

The alternative is to pull from the origin server, which works as well. At that point you might want to use the STREAM_BUFFER trigger to check if a stream is online and allow watching only when the status says the stream is live. When FULL comes by it means the stream is live, EMPTY means the stream is offline. (DRY/RECOVER can indicate stability/quality, but essentially aren't needed to watch live/offline only). 

With kind regards,

Balder Viëtor
Head of Testing

MistServer

--
You received this message because you are subscribed to the Google Groups "Mistserver.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mistserver+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mistserver/d7e117ad-ec6d-4052-84a1-d9069ad5d0c8n%40googlegroups.com.

[안봉봉]

Hello.

Thank you for your detailed reply.

We are currently providing streaming services through the Wowza streaming engine.
It is used in the Origin-Edge configuration, and we are conducting a test to review the change to Mistserver.

Since Mistserver has a light engine and supports various functions, we would like to use docker and container structures to achieve service stability and scalability.
Therefore, please understand that there are a lot of questions during the test process.

1. Enable Stream Protocol
- Enable default protocols enabled for the entire protocol
- Only DTSC, HTTP, RTMP, RTSP, TS over SRT are active and the rest of the protocol is enabled only
- Execute EXPOSE 80443 1935 4242 808088 to activate ports in Docker
=> Is there a way to activate the enabled protocol? (Especially, I want to enable the HLS protocol)

2. DTSH error when playing live
- Playing in the order of "OBS > Origin Server > Edge Server > Viewers" (stream name: test)
- The edge server pulls and sets the origin server's HLS URL (http://1.201.172.154/hls/test/index.m3u8) as the source
- When playing with HLS, the interruption occurs frequently, and the following error occurs on the edge server
. FAIL Error getting URI info for 'http://1.201.172.154/hls/test/index.m3u8.dtsh': 404 URL mismatch
. ERROR Could not connect to 'http://1.201.172.154/hls/test/index.m3u8.dtsh', since we do not have a configured external writer to handle 'http' protocols
=> I want to play it so I don't have .dtsh after m3u8.
=> Is it because the HLS protocol is not activate on the edge server?

3. Reduce Live Latency
- The current setting is as follows
. Setting photo are attached
- Different delay times for playback when played with multiple edges (8 seconds, 12 seconds, 20 seconds)
=> I want to play the most recent ts file when I play stream
=> Is there any additional setting to minimize live latency?

* Origin_Optional parameters

.

* Origin_Stream processes

.

*Edge_stream


*Edge_Stream processes

4. Live automatic push
- After automatic push settings are made from mist server 1 to mist server 2, check that the delay is as short as 4 to 5 seconds when playing on mist server 2
- Also, when played on mist server 2, it plays reliably compared to HLS URL Pulling (http://1.201.172.154/hls/test/index.m3u8)
=> Can mist server 1 be used as origin server and mist server 2 as edge server?
=> By serving users as edge servers, we want to reduce the load on the origin server and increase streaming stability

Please let me know if you need anything else or have any advice.

Thank you

2023년 8월 7일 월요일 오후 6시 20분 6초 UTC+9에 Balder Vietor님이 작성:

[Balder Vietor]

Hello,

I'll try and answer your questions to the best of my abilities, seeing as it's quite a lot please do ask again if something isn't clear. 

1. Enable Stream Protocol

This is one of the things we plan to tackle on making it more clear with our new interface. 

Only protocols that require a specific port are affected input wise, as they'd need a port to be active to work.

All protocols that mention "X over HTTP" are actually output only and require HTTP to be active in order to work. So if you want HLS, you just need to activate HTTP and set up Apple segmented over HTTP (HLS) to be enabled.

Things that aren't quite clear here is that input wise you'd still be able to use MP4 urls/links or HLS links even if they are not enabled. Since input is something you control whether you do it or not it's never inactive.

2. DTSH error when playing live

This is a default message that comes up whenever an end location does not have DTSH, which MistServer uses to both speed up reading in as well as allowing trick play on protocols that would otherwise not support it. The message can be safely ignored as it mentions it cannot find one and then that it cannot create one, which is normal for connecting to a different server. 

HLS pull would work with or without HLS being set up, however it would need to read in the entire HLS stream every time it is considered inactive, which would be a hassle. If this is a VOD stream we would recommend making the vod asset available to all servers using it for example putting it in a storage all of them can access.

If the goal here is sharing a live stream between MistServer servers the easier set up would be using the DTSC protocol, which is our own internal media format and a lot speedier in terms of latency when sharing media between MistServers. 

If it's pulling from a completely different server and only HLS is available, well then you'll just see this message more.

3. Reduce Live Latency

The settings you're using are quite wrong. The buffer time is the amount of live DVR buffer available for the stream. This will get raised to an amount that every protocol output can have decent playback so it doesn't matter too much. It's not a latency introduced or anything. 

Latencywise MistServer would automatically try to put players as close to live as they would be able to play with stable playback. There's protocols that are way better for low latency than others of course. Best in terms of latency are for the browser:

WebRTC > WS/MP4 > MP4 = WebM > HLS(CMAF) = DASH(CMAF) > HLS(TS)

There's tricks in lowering the latency like:

- Make sure there's 1 keyframe per second

- Remove the bframes so WebRTC can work

- Do not hook up encoders, or if you do choose gstreamer (as it's faster than ffmpeg) or use a hardware encoder

- WebRTC does work faster if you configure the STUN/TURN

- For HLS/CMAF you can limit the amount of segments available from live by editing the protocol in the protocol panel

Streamprocesses

These are meant for you to run/activate an encoder to add different qualities. They are completely optional and will always increase latency. These are definitely not recommended if you want low latency. You've also got them set to invalid settings so they're probably starting/stopping continuously on the background, a resolution is at least the minimum that it'll need to work.

If your stream is already in a quality that you won't need any transcoding, I would recommend removing these completely or at the very least only encode on the origin ( or edge), not both.

Sharing live streams between servers

Using DTSC is the fastest method between servers, you just have the origin push dtsc or edge pull dtsc. It's as simple as having it push towards the edge using:

dtsc://edgeserver/streamname

or pull using

dtsc://originserver/streamname

If you want to manually add delay I would push from the origin server, you can add the flag "?pushdelay=seconds" where seconds is the amount of seconds you want the push to be delayed by. If you want to use this, you'll need to use RTMP, SRT or RIST however. DTSC will ignore this as it's made to send over stream contents as fast as possible. 

You can also simply push towards the edge using RTMP, SRT or RIST. Which is the best fit depends on the network conditions.

4. Live automatic push

Yeah a traditional setup with origin/edge servers is possible through MistServer. We also have a load balancer that can make things easier, but it's not provided with the open source version of MistServer. 

Whether push or pull is better depends on your usage and preference. Technically there's no big difference as they should get similar latencies, but only if you use similar protocols. HLS is not recommended as an output to share between servers. The main con to HLS is that compared to other formats it has a lot of overhead, so I would go to the more traditional push formats for live streams:

RTMP, SRT, RIST or DTSC (if it's between MistServers). 

In general you can expect similar latencies between the protocols, but DTSC would be optimized for low latency. It requires a TCP connection and goes over port 4200 by default. You can set it up in the protocol panel. 

Most of our users tend to go with a push set up, mostly because it's easier to understand and thus set up. Pull tends to get better when you have too many servers to remember however. 

With kind regards,

Balder Viëtor
Head of Testing

MistServer

To view this discussion on the web visit https://groups.google.com/d/msgid/mistserver/227451eb-4299-4f6c-a39c-a99b9f407e0an%40googlegroups.com.

[안봉봉]

Hello.

Thank you for your reply and I'm conducting the test by applying the parts you mentioned.

However, I am inquiring because there are additional issues as below.

1. Save Error Log
- Logs for access in "General > Access log" can be stored on the server as a file
- The log for Error can be checked in the logs on the UI, is it possible to save it as a file on the server like the access log?

2. DDSC playback
- Origin Server > Stream to Edge Server via DTSC
- At this time, HLS and CMAF cannot be played because the edge server did not receive stream information as shown in the image below
. Error log not seen even if debug level is 10
. Playable through Preview, html

3. Origin Server Error
- If the stream is being transmitted, the following error occurs intermittently and the transmission is interrupted
- Stream processes setting on origin server is in None state
. Wed 09 Aug 2023, 15:06:08 FAIL Waiting for RelAccX structure to be ready timed out, aborting
. . Wed 09 Aug 2023, 15:06:08 FAIL Video encode command failed
. . Wed 09 Aug 2023, 15:06:08 FAIL timeout, resolution is not set!

4. Origin Edge
- Load balancing of edge servers is performed by setting internal L4 equipment as a domain.
- Since it is set to Round-robin, the user requests it from Edge Server 1 and watches the video, and the next m3u8 file can be requested from Server 2 to Edge Server.
- I checked that the playback fails at this time, but in the example below, can the URL after the stream key be different affect the playback failure?
.  A value such as 3_2, 1_0, or a tkn value in the example below

Edge Server 1)

http://1.201.173.43/hls/dslee/3_2/index.m3u8?tkn=2675308284
http://1.201.173.43/hls/dslee/3_2/1096561_1098481.ts?tkn=2675308284

Edge Server2) 

http://1.201.163.102/hls/dslee/1_0/index.m3u8?tkn=1174353817
http://1.201.163.102/hls/dslee/1_0/1166401_1168321.ts?tkn=1174353817

Thank you so much for always answering kindly.

2023년 8월 9일 수요일 오후 8시 37분 48초 UTC+9에 Balder Vietor님이 작성:

[Balder Vietor]

Hey again, 

Like last time I'll answer them in the same fashion. 

1. Save Error Log

The best way to grab the MistServer log is actually through journalctl. The log in the interface is not recommended at all, because it's the last 100 and really barebones informationwise. 

To get to this log you can just use journalctl syntax:

journalctl -u mistserver

This would get you the journal, which can get quite big. We'd usually recommend adding a time filter to it, like:

journalctl --since=DATE -u mistserver

The DATE part could be a specific time, or as simple as -1d for everything since a day ago.

You can also use --until=DATE to set an end time, if you don't it'll just keep filling as is. 

If you want to filter specifically on ERROR/FAIL/WARN I would grep on it, for example:

journalctl --since=-1d -u mistserver | grep 'FAIL\|ERROR\|WARN'

Would only show lines that contain FAIL, ERROR or WARN. Though if you're using loglevel 3 your log messages should only be these messages anyway.

2. DTSC playback & 3. Origin Server Error

These look to be the same issue, so I'm combining them.

The stream process that you set up is blocking the DTSC replication. DTSC is made to wait for all tracks to be available and then send, because you haven't set a resolution the encoder will not start & block the stream from being shared. MistServer expects a new video track to become available, after all there's a video transcode set up, but the transcode is failing & blocking replication as it never becomes ready.

What you need to do is either set up the transcode properly, by giving it at least a resolution format like: 1920x1080  or 1280x720  or even 480x360

The bare minimum is setting up the type, codec and resolution:

Software encoding is CPU heavy however, so I do not recommend this unless you absolutely need it.

4. Origin Edge

Instead of load balancing in this fashion I would recommend redirecting viewers to a single server instead. So have the entire stream connection come from one server per viewer. This would allow you to actually use the statistics as well. Using this method would have a viewer active on every server it requests, so the viewer count would be inaccurate. 

MistServer also optimizes for viewers expecting to receive the request of all segments once a viewer starts viewing. You'd see this especially at VoD content where MistServer starts pre-loading ahead so it can serve the stream at a good quality for playback. If you'd send VoD requests of bits and pieces they would all start to pre-load parts that end up not being requested.

As for what the pieces of the url mean:

/3_2/ this is the track selection of the url. It means tracks 3 and 2 are used for playback. If the next server uses 1_0 that means you're most likely getting different audio and resolution from the servers. Most players would probably trip over something like this.

?tkn=2675308284 This is a token set by MistServer to identify the viewer over multiple protocols. You can remove this in the general settings (by removing the token based session identification). This is completely harmless to playback, but useful for statistics. 

1096561_1098481.ts  This is the segment to be used. It's basically a time in milliseconds where it starts and where it ends. This segment is about 2 seconds long as the value difference is about 2000. If these do not match to the next segment then the playback will have gaps/errors.

Using DTSC push/pull should mean that the tracks and data is the same over all servers, so it should be possible to grab a new m3u8/segment every time and they should match up. It's really not recommended though. You'd see best playback if you keep the stream connection to one server and do the round-robin assignment at the first connection by just redirecting them to a server to use.

With kind regards,

Balder Viëtor
Head of Testing

MistServer

To view this discussion on the web visit https://groups.google.com/d/msgid/mistserver/6c1939b7-8f96-4e42-b287-efc4bf613ca1n%40googlegroups.com.

[안봉봉]

Hello.

As you said, in order to eliminate encoding, we made the Stream processes in None state on the origin and edge servers. However, the following error occurs not only when requesting to domain but also when requesting to server IP.

The same problem occurs for both HLS and CMAF, and CMAF has a short delay time, so I'm conducting tests using CMAF.

** Test domain
1) Edge1
  - http://1.201.173.43/cmaf/test/index.m3u8
2) Edge2
  - http://1.201.173.47/cmaf/test/index.m3u8
3) Edge3
  - http://1.201.163.102/cmaf/test/index.m3u8
4) Domain
 - http://mistserver-edge2.xst.kinxcdn.com/cmaf/test/index.m3u8
  . Edge1, Edge2, Edge3 are configured as round robins

There are some problems here.

Problem 1) In browser developer mode, tracks 0 and 1 are requested alternately, but they don't actually play.

http://mistserver-edge2.xst.kinxcdn.com/cmaf/test/0/index.m3u8?mTrack=1&iMsn=1946&tkn=693951758
http://mistserver-edge2.xst.kinxcdn.com/cmaf/test/1/index.m3u8?mTrack=1&iMsn=1946&tkn=693951758
http://mistserver-edge2.xst.kinxcdn.com/cmaf/test/0/index.m3u8?mTrack=1&iMsn=1946&tkn=693951758
http://mistserver-edge2.xst.kinxcdn.com/cmaf/test/1/index.m3u8?mTrack=1&iMsn=1946&tkn=693951758
http://mistserver-edge2.xst.kinxcdn.com/cmaf/test/0/index.m3u8?mTrack=1&iMsn=1946&tkn=693951758
http://mistserver-edge2.xst.kinxcdn.com/cmaf/test/1/index.m3u8?mTrack=1&iMsn=1946&tkn=693951758

Problem 2) This error occurs when the above error occurs. Occurs on both origin and edge servers.

onFail '': Invalid packet header received. Aborting.

Dropping input track 0,1: disconnec request from buffer

** The settings for each server are as follows. I'm trying to minimize the setup and do a normal playback test, is there any problem? 

The stream is being pulled from the edge server to the origin server using the DTSC protocol.

**Origin Server Settings
{"account":{"admin":{"password":"3ca87422d915115c8c4f040d87eff7c2"}},"autopushes":null,"bandwidth":null,"config":{"accesslog":"/etc/mistserver/config/access.log","controller":{"interface":null,"port":null,"username":null},"debug":null,"defaultStream":null,"limits":null,"prometheus":"","protocols":[{"connector":"AAC"},{"connector":"CMAF","mergesessions":false,"nonchunked":false},{"connector":"DTSC","port":4200},{"connector":"EBML"},{"connector":"FLAC"},{"connector":"FLV"},{"connector":"H264"},{"connector":"HDS"},{"connector":"HLS"},{"connector":"HTTP","port":80,"pubaddr":[]},{"connector":"HTTPTS"},{"connector":"JSON"},{"connector":"MP3"},{"connector":"MP4"},{"connector":"OGG"},{"connector":"RTMP"},{"connector":"RTSP"},{"connector":"SDP"},{"connector":"SubRip"},{"connector":"TSRIST"},{"connector":"TSSRT"},{"connector":"WAV"},{"connector":"WebRTC"}],"serverid":"Origin Server2","sessionInputMode":15,"sessionOutputMode":15,"sessionStreamInfoMode":"1","sessionUnspecifiedMode":0,"sessionViewerMode":15,"tknMode":15,"triggers":{"PUSH_OUT_START":[]},"trustedproxy":[]},"extwriters":null,"push_settings":{"maxspeed":0,"wait":0},"streams":{"test":{"name":"test","processes":[],"resume":"1","segmentsize":1000,"source":"push://211.196.205.72","stop_sessions":false}},"ui_settings":{"graphs":{"Graph 1":{"datasets":[{"datatype":"clients","origin":["stream","test"]},{"datatype":"downbps","origin":["stream","test"]},{"datatype":"upbps","origin":["stream","test"]}],"id":"Graph 1","xaxis":"time"}},"viewmode":"list"},"variables":null}


**Edge Server Settings
{"account":{"admin":{"password":"3ca87422d915115c8c4f040d87eff7c2"}},"autopushes":null,"bandwidth":null,"config":{"accesslog":"/etc/mistserver/config/access.log","controller":{"interface":null,"port":null,"username":null},"debug":null,"defaultStream":null,"limits":null,"prometheus":null,"protocols":[{"connector":"AAC"},{"connector":"DTSC","port":4200},{"connector":"EBML"},{"connector":"FLAC"},{"connector":"FLV"},{"connector":"H264"},{"connector":"HDS"},{"connector":"HLS","nonchunked":false},{"connector":"HTTP","port":80,"pubaddr":[]},{"connector":"HTTPTS"},{"connector":"JSON"},{"connector":"MP3"},{"connector":"MP4"},{"connector":"OGG"},{"acceptable":"0","connector":"RTMP"},{"connector":"RTSP"},{"connector":"SDP"},{"connector":"SubRip"},{"connector":"TSRIST"},{"connector":"TSSRT"},{"connector":"WAV"},{"connector":"WebRTC"},{"connector":"CMAF","mergesessions":false,"nonchunked":false}],"serverid":"Edge Server2","sessionInputMode":15,"sessionOutputMode":15,"sessionStreamInfoMode":"1","sessionUnspecifiedMode":0,"sessionViewerMode":15,"tknMode":15,"triggers":null,"trustedproxy":[]},"extwriters":null,"push_settings":{"maxspeed":0,"wait":0},"streams":{"test":{"always_on":false,"name":"test","processes":[],"realtime":false,"resume":"1","segmentsize":1000,"source":"dtsc://1.201.172.154/test","stop_sessions":false}},"ui_settings":{"graphs":{"Graph 1":{"datasets":[{"datatype":"clients","origin":["stream","test"]},{"datatype":"upbps","origin":["stream","test"]},{"datatype":"downbps","origin":["stream","test"]},{"datatype":"perc_lost","origin":["stream","test"]},{"datatype":"memload","origin":["total"]}],"id":"Graph 1","xaxis":"time"}}},"variables":null}

Thank you very much for your kind reply.

2023년 8월 10일 목요일 오후 6시 6분 23초 UTC+9에 Balder Vietor님이 작성:

[Balder Vietor]

Hey there,

When I try to connect to the CMAF playlist I'm seeing that the segments it's requesting are 5 seconds "behind" to where they should be for real time. Could it be that your computer clock isn't set up correctly?

What I'm seeing is that it's trying to download the partial segment of 500ms, but it's also over 5 seconds in the past. Which I believe will trigger logic it should just download the full segment as it's 1 second in length but then gets this 5 seconds too late and considers it "too far away" for live streaming.

[hls @ 0x7f1abc000c80] skipping 33 segments ahead, expired from playlists

Increasing the segment duration would probably help here as well. Segments of 1000 are small, especially considering the LL-HLS spec will already divide it in partials to allow for quicker playback. We'd recommend a minimum of 2000ms, Apple themselves recommend ~6000. 

The HLS playback works here on our end, but that's unsurprising as it would flat out ignore the computer clock and just go with whatever is "newest" in the playlist.

With kind regards,

Balder Viëtor
Head of Testing

MistServer

To view this discussion on the web visit https://groups.google.com/d/msgid/mistserver/4333b88d-5ab2-4938-bd7d-5ccdca738337n%40googlegroups.com.

[안봉봉]

Hello.

First, the time is set by receiving real-time time.
Therefore, it has been confirmed that the transmission time and computer time are the same.

For CMAF, the following server logs occurred when playing as a domain (round robin) as well as individual IPs, causing buffering to the player.

When this error occurs, I checked that m4s file cannot be imported and only m3u8 is imported continuously.

At this point, the player checked that the buffering continues and the playback stops, but the m3u8 file for the next playback area continues to be imported.

After buffering for a certain period of time, the image is played by importing the m4s file again.

Given that the latency remains at 3-4 seconds when there was no problem, it doesn't seem to be an issue for the network.

Therefore, when I played with HLS, it was confirmed that both IP and domain playback were played without any problems.

However, in the case of HLS, the delay time is about 12 seconds, so we are testing to reduce it. When I opened the file of m3u8, I found that there are 9 TS files, but I wonder if I can adjust the number of TS files that will be included in m3u8.

Thank you always.

2023년 8월 14일 월요일 오후 9시 49분 53초 UTC+9에 Balder Vietor님이 작성:

[Balder Vietor]

Hey, 

Limiting the amount of segments in the M3U8 playlist

This can be done at the protocol setting. 

Live playlist limit is the setting you're looking for. 

Keep in mind that HLS requires at least 3 segments to even allow playback, going under that will most likely break playback for any player. I believe MistServer will grow the playlist to a minimum required for stable playback, but I figured it's worth mentioning.

CMAF warnings

I'll have a look at those warnings, though we are also preparing a new release of CMAF after having reworked it. I've already seen those warnings are gone within that rework, so there's a good chance we'll just push the rework along faster. 

With kind regards,

Balder Viëtor
Head of Testing

MistServer

To view this discussion on the web visit https://groups.google.com/d/msgid/mistserver/7fab0957-efcc-49b3-ba12-0f10b76262c7n%40googlegroups.com.

[안봉봉]

Hello.

Thank you for your reply.

I checked the playlist you told me about as it affects the playlist that you bring when you play the video.

After setting the corresponding value to 3, when playing the image, we checked to get three m3u8s each.



The part I inquired about is that when I open the m3u8 file that I import, I wonder if I can change the number of ts files that 1 m3u8 has.

I am inquiring because the number of ts that I bring at once is 9.



Thank you.

2023년 8월 16일 수요일 오후 5시 29분 50초 UTC+9에 Balder Vietor님이 작성:

[Balder Vietor]

Hey there, 

The Protocol setting for segments in a live playlist should be. I'm thinking since the segments are each 1 second they're limited to 9 segments to keep things playable. 

The absolute minimum possible in this case would be 6, you need 3 times the target duration (in the case that the target duration doesn't match the segment size). 

The 3 M3U8s at the start are a bit weird, that shouldn't be needed. I would've expected a M3U8, then a segment then a new M3U8. Not 3 of them, are they all exactly the same when you try to view what's in there?

With kind regards,

Balder Viëtor
Head of Testing

MistServer

To view this discussion on the web visit https://groups.google.com/d/msgid/mistserver/ef7781db-81f1-40eb-81ba-73982c8eb14fn%40googlegroups.com.