Creating rules for URL based cid's

Geoff

unread,

Dec 2, 2011, 11:21:17 PM12/2/11

to dripls-dev

So I am attempting to pull down an m3u8 from a CDN, and then shape it
with a simple 404 for a specific bitrate. Getting things to work with
the wt_suite is easy, as would be expected, however I (having no
former python skills) have been unable to figure out where I am
supposed to be building my rules for these types of requests. Can
anyone help point me in the right direction? I have read both the
readme and the install file many times, and am really looking for a
shove in the right direction at this point. Any help is greatly
appreciated.

Cheers,

/GC

Ludo Antonov

unread,

Dec 5, 2011, 11:16:40 PM12/5/11

to dripl...@googlegroups.com

Hi Geoff,

thanks for reaching out. Apologies for the delayed reply, i was out during the weekend. You can actually achieve what you want to do without modifying any code.

Here is how. /master.m3u8 supports a url_cid parameter. If you set cid=url , and provide cid_url="<original_master_url>" , DripLS will fetch the master playlist from the cid_url and perform the transformations needed.

Sounds like what you want to do is :

http://dripls-host/master.m3u8?authkey=[authkey]&cid=url&cid_url=[cdn-url-to-master-m3u8]&r=[rules]

Let me know if this helps out. I will update the documentation to explain this scenario.

Ludo

Ludo Antonov

unread,

Dec 5, 2011, 11:34:32 PM12/5/11

to dripl...@googlegroups.com

Geoff,

I updated the documentation on GitHub to explain the scenario you describe :

https://github.com/hulu/dripls

http://dripls-host/master.m3u8?authkey=[authkey]&cid=url&cid_url=[url-to-original-master-m3u8]&tag=[tag]&r=[rules]

DripLS supports fetching and reshaping from an already accessible master m3u8. This is supported with all endpoints ( master.m3u8, cache, tag.m3u8). This configuration is useful when you already have a pre-built m3u8 available online ( either on a CDN, or another host accessible by the DripLS node) and you want to use DripLS to traffic shape this m3u8. To use this configuration set the cid parameter value to url and provide an additional parameter cid_url, which points to your master m3u8.

On Dec 2, 2011, at 8:21 PM, Geoff wrote:

Geoff

unread,

Dec 6, 2011, 8:22:17 PM12/6/11

to dripls-dev

Ludo,

I really appreciate you taking the time out to respond, however I
think that I may not have explained myself thoroughly enough. Let me
give it another shot.

So when I curl a wt m3u8, I get this:

$ curl "http://dripls01/master.m3u8?
authkey=sample&cid=wt&r=1700k~e404"
#EXTM3U
#EXT-X-VERSION:2
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=4000000,RESOLUTION=960x540
http://localhost:8080/playlist.m3u8?p=m_wt_9805609c98404be48326e09fa3451b59_4000000__d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1000000,RESOLUTION=960x540
http://localhost:8080/playlist.m3u8?p=m_wt_9805609c98404be48326e09fa3451b59_1000000__d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1700000,RESOLUTION=960x540
http://localhost:8080/ostatus?s=404
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=650000,RESOLUTION=960x540
http://localhost:8080/playlist.m3u8?p=m_wt_9805609c98404be48326e09fa3451b59_650000__d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=250000,RESOLUTION=960x540
http://localhost:8080/playlist.m3u8?p=m_wt_9805609c98404be48326e09fa3451b59_250000__d

So we can clearly see that the playlist manipulation is occurring when
I use cid=wt. It appears that things are working as expected to this
point. However, when I try to use the cid=url method as described, I
am not getting the expected manipulation of the playlist:

$ curl "http://dripls01/master.m3u8?cid=url&cid_url=http://
segmenter.dev.com/dev/SEGMENTER_TESTING/2011/11/21/EVENT21/
master_mobile.m3u8&r=1200k~e404"
#EXTM3U
#EXT-X-VERSION:2
#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2000000
2200K/2200_slide.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1200000
1200K/1200_slide.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=500000
500K/500_slide.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=800000
800K/800_slide.m3u8

I should be seeing something like:

http://localhost:8080/ostatus?s=404

for the 1200K stream in that m3u8, similar to the cid=wt method,
correct? My original question was that I see the rules for the cid=wt
in the driple/tests directory, and I made an assumption that I need to
create rules for 404's and traffic shaping similar to those somewhere
else in the dripls directory. I haven't figured out where that is
supposed to be. I tried to add some additional output via logging to
see what is being called when cid=wt to try to trace the path taken to
created the manipulated m3u8, but I have not been successful in
figuring out where to build my rules for cid=url shaping.

Hopefully my question is a little bit more clear, and hopefully your
answer is just as simple! Ludo, thanks again for taking the time out
to respond.

Cheers

/GC

Ludo Antonov

unread,

Dec 6, 2011, 9:06:48 PM12/6/11

to dripl...@googlegroups.com

Hmm , Geoff , you shouldn't have to do anything special. One thing I noticed is that ", BANDWIDTH=1200000" has a space. You might be hitting a bug .. i will check and let you know . But your second curl should definitely be able to work out of the box, as you would expect with wt.

Ludo

Ludo Antonov

unread,

Dec 6, 2011, 9:17:52 PM12/6/11

to dripl...@googlegroups.com

It was a bug , as I suspected with not trimming the spaces around BANDWIDTH.

I committed a fix to master. Please let me know if you are still having an issue, and thanks for reaching out!

Ludo

On Dec 6, 2011, at 5:22 PM, Geoff wrote:

Geoff

unread,

Dec 6, 2011, 10:18:04 PM12/6/11

to dripls-dev

Ludo,

Thanks so much for taking a look at this, it has literally been
driving me crazy the past week and a half. After testing I am now
getting another error, but I think this has to do with the stack/m3u8
conventions that we use. Here is the error:

$ curl "http://dripls01/master.m3u8?cid=url&cid_url=http://
segmenter.dev.com/dev/SEGMENTER_TESTING/2011/11/21/EVENT21/

master.m3u8&r=1200k~e404"
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8"></meta>
<title>500 Internal Server Error</title>
<style type="text/css">
#powered_by {
margin-top: 20px;
border-top: 2px solid black;
font-style: italic;
}

#traceback {
color: red;
}
</style>
</head>
<body>
<h2>500 Internal Server Error</h2>
<p>The server encountered an unexpected condition which
prevented it from fulfilling the request.</p>
<pre id="traceback">Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/cherrypy/
_cprequest.py", line 645, in respond
response.body = self.handler()
File "/usr/local/lib/python2.7/site-packages/cherrypy/lib/
encoding.py", line 188, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/cherrypy/
_cpdispatch.py", line 29, in __call__
return self.callable(*self.args, **self.kwargs)
File "/opt/dripls/dripls/main.py", line 55, in master_m3u8
cached_cid = self.cache_stream(cid, r, tag, kwargs)["id"]
File "/opt/dripls/dripls/main.py", line 105, in cache_stream
info = shaper.cache_and_shape(master_playlist, seeded_content_id,
rules)
File "/opt/dripls/dripls/shaper.py", line 231, in cache_and_shape
variant_playlist =
httpls_client.pull_variant_playlist( variant_playlist_desc["url"])
File "/opt/dripls/dripls/httpls_client.py", line 27, in
pull_variant_playlist
playlist_response = urllib2.urlopen(url)
File "/usr/local/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/local/lib/python2.7/urllib2.py", line 386, in open
protocol = req.get_type()
File "/usr/local/lib/python2.7/urllib2.py", line 248, in get_type
raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: 1200K/1200_slide.m3u8
</pre>
<div id="powered_by">
<span>Powered by <a href="http://www.cherrypy.org">CherryPy 3.2.0</
a></span>
</div>
</body>
</html>

It seems to not like the 1200K_slide.m3u8. Is this the part of the
install instructions where you are referencing the extension of the
conf/data.py for the HttplsProvider? We use several different naming
conventions here; master_wired.m3u8, master_mobile.m3u8, etc.

Also I noticed that when I run

$ fab dev deploy I

I kept getting the following error:

Fatal error: run() encountered an error (return code 1) while
executing 'cp /tmp/dripls_wt_segments/* playlists/'

I had to modify the fabfile.py in several places to ensure that the
directory was created in the right place before the attempted copy of
the files. I know that there is probably a way to submit a patch to
you, but I am not a developer and it would take me longer to figure
out how to do that then to just coipy and paste what I modified. Here
are the lines:

package_path = os.path.join(env.path,'dripls')
...
# copy any test segments
run('mkdir -p /tmp/dripls_wt_segments/')
run('mkdir -p {0}/playlists'.format(package_path)) ****
if env.include_wt:
run('mv test/wt_suite/segments/* /tmp/
dripls_wt_segments/')

run('touch /tmp/dripls_wt_segments/touch')
run('cp /tmp/dripls_wt_segments/* {0}/
playlists/'.format(package_path)) ****

if env.env == "dev":
run('cp test/wt_suite/local/* {0}/
playlists/'.format(package_path)) ****
else:
run('cp test/wt_suite/wt_dripls/* {0}/
playlists/'.format(package_path)) ****

Forgive me if this was the incorrect way to do this, but it seems the
easiest to me. Please let me know if you see any errors in my
process, and thanks again for getting back to me so quickly.

/GC

On Dec 6, 9:17 pm, Ludo Antonov <ludo.anto...@gmail.com> wrote:
> It was a bug , as I suspected with not trimming the spaces around BANDWIDTH.
>
> I committed a fix to master. Please let me know if you are still having an issue, and thanks for reaching out!
>
> Ludo
>
> On Dec 6, 2011, at 5:22 PM, Geoff wrote:
>
>
>
>
>
>
>
> > Ludo,
>
> > I really appreciate you taking the time out to respond, however I
> > think that I may not have explained myself thoroughly enough. Let me
> > give it another shot.
>
> > So when I curl a wt m3u8, I get this:
>
> > $ curl "http://dripls01/master.m3u8?
> > authkey=sample&cid=wt&r=1700k~e404"
> > #EXTM3U
> > #EXT-X-VERSION:2
> > #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=4000000,RESOLUTION=960x540

> >http://localhost:8080/playlist.m3u8?p=m_wt_9805609c98404be48326e09fa3...
> > #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1000000,RESOLUTION=960x540
> >http://localhost:8080/playlist.m3u8?p=m_wt_9805609c98404be48326e09fa3...

> > #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1700000,RESOLUTION=960x540
> >http://localhost:8080/ostatus?s=404
> > #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=650000,RESOLUTION=960x540

> >http://localhost:8080/playlist.m3u8?p=m_wt_9805609c98404be48326e09fa3...
> > #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=250000,RESOLUTION=960x540
> >http://localhost:8080/playlist.m3u8?p=m_wt_9805609c98404be48326e09fa3...

Ludo Antonov

unread,

Dec 6, 2011, 10:37:27 PM12/6/11

to dripl...@googlegroups.com

Hi Geoff,

the first error is about the fact that the variant playlist "1200K/1200_slide.m3u8" is not a properly formed url. DripLS expects a full url for a variant playlist( starting with http(s)://...). This was the only scenario we had.

I just looked at the IETF draft ( http://tools.ietf.org/html/draft-pantos-http-live-streaming-07#section-8.2) , and indeed allows for relative playlists. I will commit a fix for this soon and let you know ( next day or so). In the meantime, if you specify a full url , things should work.

Thanks for the patch, I will incorporate it with the fix as well!

Ludo

Geoff

unread,

Dec 6, 2011, 10:45:08 PM12/6/11

to dripls-dev

Ludo,

Indeed it does work as expected with the full URL:

$ curl "http://dripls01/master.m3u8?cid=url&cid_url=http://
segmenter.dev.com/dev/SEGMENTER_TESTING/2011/11/21/EVENT21/
master_mobile.m3u8&r=1200k~e404"
#EXTM3U
#EXT-X-VERSION:2

Fantastic! Thanks so much for working through this with me, I
appreciate it very much. I will keep an eye out for the relative path
commit.

/GC

On Dec 6, 10:37 pm, Ludo Antonov <ludo.anto...@gmail.com> wrote:
> Hi Geoff,
>
> the first error is about the fact that the variant playlist "1200K/1200_slide.m3u8" is not a properly formed url. DripLS expects a full url for a variant playlist( starting with http(s)://...). This was the only scenario we had.
>

> I just looked at the IETF draft (http://tools.ietf.org/html/draft-pantos-http-live-streaming-07#sectio...) , and indeed allows for relative playlists. I will commit a fix for this soon and let you know ( next day or so). In the meantime, if you specify a full url , things should work.

Message has been deleted

Ludo Antonov

unread,

Dec 6, 2011, 11:28:19 PM12/6/11

to dripl...@googlegroups.com

yes , actually fab dev deploy , runs off of /tmp . It was a bit of a poor choice at the time, in an attempt to separate source from deployment. If you look under /tmp/dripls/dripls/playlists , they should be there .

Ludo

On Dec 6, 2011, at 8:24 PM, Geoff wrote:

> Ludo,
>
> I have one more question! So now that I am getting the shaped m3u8,
> shouldn't the playlists be getting created and stored in the playlists
> directory? I am getting the output in the m3u8 from my above post,
> but I am not seeing the cached/shaped playlists being created anywhere
> on the filesystem. Any advice?
>
> Thanks
>
> /GC

>
> On Dec 6, 10:45 pm, Geoff <geoff.cardam...@gmail.com> wrote:
>> Ludo,
>>
>> Indeed it does work as expected with the full URL:
>>
>> $ curl "http://dripls01/master.m3u8?cid=url&cid_url=http://
>> segmenter.dev.com/dev/SEGMENTER_TESTING/2011/11/21/EVENT21/
>> master_mobile.m3u8&r=1200k~e404"
>> #EXTM3U
>> #EXT-X-VERSION:2

>> ...
>>
>> read more »

Geoff

unread,

Dec 6, 2011, 11:29:42 PM12/6/11

to dripls-dev

I just used the fabfile to bounce the service, and now he playlists
are appearing in the folder that I was expecting. I didn't change
anything either.

> ...
>
> read more »

Ludo Antonov

unread,

Dec 6, 2011, 11:35:58 PM12/6/11

to dripl...@googlegroups.com

hmm odd indeed. But you are able to see them now , yes?

Geoff

unread,

Dec 6, 2011, 11:51:35 PM12/6/11

to dripls-dev

Ludo,

Yes, they are appearing now. I am experiencing 206's because cherrypy
is trying to pull down the chunks from the CDN and the m3u8 is using
relative path's so it's having some issues. I think the issue with
the playlists was because i never bounced the service after the
initial install when it started it. Maybe it was still trying to do
something with the /tmp/dripls/playlist folder, and that doesn't exist
on my box. The bounce may have let it find the /opt/dripls/dripls/
playlist directory for writing.

> ...
>
> read more »

Ludo Antonov

unread,

Dec 6, 2011, 11:58:15 PM12/6/11

to dripl...@googlegroups.com

I see . ok . Fix for the relative paths coming shortly btw .. if you already have a m3u8 that you can point me to, that is visible, i will submit it to github in the next 30 min or so .

Ludo

Geoff

unread,

Dec 7, 2011, 2:25:47 AM12/7/11

to dripls-dev

I wish I would have seen this earlier. I was walking out of work when
I fired off my last response. I will see if I can dig something up if
you still need it tomorrow.

Geoff

> ...
>
> read more »

Ludo Antonov

unread,

Dec 7, 2011, 3:34:18 AM12/7/11

to dripl...@googlegroups.com

Hi Geoff,

No worries, i just pushed an update to github that should take care of it . With it now you will have consistent master/local playlists .

Relative urls for variant playlists are handled properly and will be rewritten to a dripls based url.

Relative urls for segments will be rewritten to full urls pointing to the original server, unless they match a rule, in which case they will be either replaced to a status code page, or downloaded and rewritten to a local dripls based url, after the ts is downloaded.

Ludo

Geoff

unread,

Dec 7, 2011, 10:57:53 PM12/7/11

to dripls-dev

Ludo,

Have you tested this with any type of iOS device? I keep experiencing
issues when I attempt to open a link on my iPad. Is it possible for
you to add last modifier headers? For example:

$ curl -si http://iphoned5.akamai.com.edgesuite.net/mhbarron/nasatv/nasatv_1500.m3u8
HTTP/1.1 200 OK
Server: Apache
ETag: "2484654dd16ee41b7a2f0fdc71795783:1276283705"
Content-MD5: JIRlTdFu5Bt6Lw/ccXlXgw==
Last-Modified: Fri, 11 Jun 2010 19:15:05 GMT
Accept-Ranges: bytes
Content-Length: 1593
Content-Type: audio/x-mpegurl
Date: Wed, 07 Dec 2011 22:54:33 GMT
Connection: keep-alive

#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:73285
#EXTINF:10,
http://iphoned5.akamai.com.edgesuite.net/mhbarron/nasatv/nasatv_1500/Seg_060110_103747_36/nasatv_1500_060110_103747_73285.ts
#EXTINF:10,

Every other m3u8 request that I have seen includes these, and they may
be braking the iOS player. The rewritten m3u8's work with QT in OS X,
but for some reason they are not working at all on iOS. Have you ever
tested on an iOS device or have you never had the need?

/GC

> ...
>
> read more »

Ludo Antonov

unread,

Dec 7, 2011, 11:08:31 PM12/7/11

to dripl...@googlegroups.com

Hi Geoff,

just to make sure that I get the issue correctly .. are you saying that some headers are not being included and this breaks iOS devices? We've been testing on iOS devices successfully .. haven't tried it in a couple of months, but i believe last time I did , it worked ok. Can you tell me more specifically which headers are missing? I can make sure they are added, yes.... it shouldn't be much of an issue .

Ludo

Geoff

unread,

Dec 7, 2011, 11:32:22 PM12/7/11

to dripls-dev

Ludo,

It doesn't appear that the Last Modified header is being passed:

$ curl -si "http://dripls01/master.m3u8?cid=url&cid_url=http://
segmenter.dev.com/dripls/master_rel.m3u8&r=1700k~e404"
HTTP/1.1 200 OK
Date: Thu, 08 Dec 2011 04:22:44 GMT
Server: CherryPy/3.2.0
Content-Length: 814
Content-Disposition: inline; filename=url.m3u8
Vary: Accept-Encoding
Content-Type: application/vnd.apple.mpegurl
Connection: close

#EXTM3U
#EXT-X-VERSION:2
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=4000000,RESOLUTION=960x540

http://dripls01:8080/playlist.m3u8?p=m_url_60bf544f7e3b46e38f27930e3d4167cd_4000000__d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1000000,RESOLUTION=960x540
http://dripls01:8080/playlist.m3u8?p=m_url_60bf544f7e3b46e38f27930e3d4167cd_1000000__d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1700000,RESOLUTION=960x540
http://dripls01:8080/ostatus?s=404
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=650000,RESOLUTION=960x540
http://dripls01:8080/playlist.m3u8?p=m_url_60bf544f7e3b46e38f27930e3d4167cd_650000__d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=250000,RESOLUTION=960x540
http://dripls01:8080/playlist.m3u8?p=m_url_60bf544f7e3b46e38f27930e3d4167cd_250000__d

You can see that CherryPy isn't returning the Last Modified header in
the request. Actually I just confirmed that I get a stream up and
workign on my iPad. The header wasn't the problem for me, I think the
wifi i was using was just really bogged down. I can play the full
stream down on the iPad now, using the relative path! This is pretty
fantastic. But I am not sure if other players may brick without the
last modified header.

/GC

On Dec 7, 11:08 pm, Ludo Antonov <ludo.anto...@gmail.com> wrote:
> Hi Geoff,
>

> just to make sure that I get the issue correctly .. are you saying that some headers are not being included and this breaks iOS devices? We've been testing on iOS devices successfully .. haven't tried it in a couple of months, but i believe last time I did , it worked ok. Can you tell me more specifically which headers are missing? I can make sure they are added, yes.... it shouldn't be much of an issue .
>
> Ludo
>
> On Dec 7, 2011, at 7:57 PM, Geoff wrote:
>
>
>
>
>
>
>
> > Ludo,
>
> > Have you tested this with any type of iOS device? I keep experiencing
> > issues when I attempt to open a link on my iPad. Is it possible for
> > you to add last modifier headers? For example:
>

> > $ curl -sihttp://iphoned5.akamai.com.edgesuite.net/mhbarron/nasatv/nasatv_1500....

> > HTTP/1.1 200 OK
> > Server: Apache
> > ETag: "2484654dd16ee41b7a2f0fdc71795783:1276283705"
> > Content-MD5: JIRlTdFu5Bt6Lw/ccXlXgw==
> > Last-Modified: Fri, 11 Jun 2010 19:15:05 GMT
> > Accept-Ranges: bytes
> > Content-Length: 1593
> > Content-Type: audio/x-mpegurl
> > Date: Wed, 07 Dec 2011 22:54:33 GMT
> > Connection: keep-alive
>
> > #EXTM3U
> > #EXT-X-TARGETDURATION:10
> > #EXT-X-MEDIA-SEQUENCE:73285
> > #EXTINF:10,

> >http://iphoned5.akamai.com.edgesuite.net/mhbarron/nasatv/nasatv_1500/...

> ...
>
> read more »

Ludo Antonov

unread,

Dec 8, 2011, 12:49:27 AM12/8/11

to dripl...@googlegroups.com

Fantastic! I will look into adding the Last Modified header, but as a low pri, since you proved that this was not the cause of problems!

Let me know if I can help with anything else.

Cheers!

Ludo

Reply all

Reply to author

Forward