larverd judith latyna

[Albina Hickel]

Netflix recently published how it built Timestone, a custom high-throughput, low-latency priority queueing system. Netflix built the queuing system using open-source components such as Redis, Apache Kafka, Apache Flink and Elasticsearch. Engineers state that they built Timestone since they could not find an off-the-shelf solution that met all of its requirements.

One of these requirements is the ability to mark some work items as non-parallelizable without requiring any locking or coordination on the consumer side. This requirement means that Timestone should not release some messages for processing until previous items belonging to the same work set are completed first. Timestone introduces the concept of "Exclusive Queues" to support this notion.

Another requirement is that a message can only be assigned to one worker at any given time. It is important since work that tends to happen in Cosmos is resource-intensive and can fan out to thousands of actions, and one of the goals was to reduce resource waste. This requirement rules out eventually consistent solutions and means that Netflix engineers want linearizable consistency at the queue level.

Netflix engineers achieved this requirement by maintaining a message state per message. When a producer enqueues a message, the message is set to the "Pending" or "Invisible" state, depending on the message's optional invisibility timeout. When a consumer dequeues a pending message, it acquires an exclusive lease on that message, and Timestone sets the message in the "Running" state. At this stage, the producer can mark the message as "Completed" or "Canceled". Each message can be dequeued up to a finite number of attempts, after which Timestone moves it to the "Errored" state. The following diagram illustrates all possible state transitions.

The Timestone server exposes a gRPC-based interface. All API operations are queue-scoped. All API operations that modify the state are idempotent. The system of record is a durable Redis cluster. Redis persists each write request to a transaction log before it sends a response back to the server. Inside Redis, a sorted set sorted by priority represents each queue. Messages and queue configurations are stored as hashes.

Almost all of the interactions between Timestone and Redis [...] are codified as Lua scripts. In most of these Lua scripts, we tend to update a number of data structures. Since Redis guarantees that each script is executed atomically, a successful script execution is guaranteed to leave the system in a consistent (in the ACID sense) state.

Timestone captures information about incoming messages and their transition between states in two secondary indexes maintained in Elasticsearch for observability purposes. When the Timtstone server gets a write response from Redis, it converts it into an event sent to a Kafka cluster. Two Flink jobs, one for each type of index Timestone maintains, consume the events from the corresponding Kafka topics and update the indexes in Elasticsearch.

Netflix built Timestone to support the needs of its media encoding platform, Cosmos. Timestone also backs Conductor, Netflix's general-purpose workflow orchestration engine, and it acts as the scheduler for large-scale data pipelines.

If so my douchie friend from across the pond, you are a shill and you drive to my mothers house every 2 days a reset her unit so she can watch movies. I bought it for her because it was supposed to be simple and I relied on the WD name.

However, yesterday afternoon while I was at work my wife called and told me that the Instant Queue was no longer available. I told her to use the Wii instead and it worked fine. I performed the reset procedure I found here and got it working again.

im having this issue as well now. i wanted to know if everyone here has blu-ray added to their netflix accounts? i just added blu-ray and now im having this issue. looking forward to the firmware update. i have no other complaints except the long startup times when the wdtv starts up. its much longer than the normal wdtv live.

another thing for those of you who are havng probelms reaching the deactivate option within wdtv. make sure you start pushing the arrow sequence once the netflix window opens up and starts to load up the queue. u should see the circular progress indicator when you push the buttons. if it reaches the queue error screen you were too slow.

I'm about to cancel my Netflix account and I'd like to be able to keep the list of movies in my queue. Is there an easy way to export the list in plain text so I don't have to resort to printing it out?

After failing at the above suggestions several times, I just opened my Netflix RSS, went to File/Save As. it defaulted to .xml and saved it to the desktop. Then I opened it up in Excel. Got a lovely little spreadsheet that was ready to edit, sort, and delete unneeded columns. The entire 375 item queue was available in seconds.

I opened Internet Explorer 9, logged into Netflix, went to my queue, and right clicked the page. There was an option to "Export to Microsoft Excel" which when clicked opens Excel. Copy the address of the Netflix queue you would like to save, paste in Excel "address" bar, and click "Go."Go ahead and click "Import" and wait for Excel to import the data. Once finished, click "OK" on the Import Data window to import into the existing sheet, then delete any unnecessary rows and columns.

Because of the missing rss feature I also had to make use of jQuery. The following snippets run on the "My Activity" page return 3 newline separated lists which can be easily copied into a spreadsheet document. Nothing else than a browser with a javascript console should be needed.

I am playing with the Netflix API. I am confused on what they want as the 'UserID'. (for a protected query)I am sending in this string (after authentication) to get the User's queue: -public.netflix.com/users/'userID'/queues

Maybe I am not even approaching it from the right angle. Any documentation or code I have found glosses over that part (My netflix ID doesn't work and I assume it should be part of the oauth token I get back, not my normal netflix ID)

Can't find what you're looking for in your Netflix queue? Use this Chrome extension to easily update the sort order of your Netflix queue by title, genre, rating and combinations of the three. Makes it easier to find what you're looking for when using devices like ROKU, Wii and smart phones or tablets. Additional features coming soon. Please send us feedback about bugs or feature requests using the "send feedback' link on the details tab of the Chrome Web Store.Or contact us directly at Netflix...@gmail.comHow it works:After you have installed the QHelper Chrome extension do the following:1. Navigate to your Netflix DVD Queue or My List2. The QHelper menu bar will appear above the netflix menu bar.3. Click the Sort link. The QHelper sort window will appear.4. Select the desired sort order and click the Sort button.Version Updates:0.2.8 1. Resolved issue that caused menu bar to not display on the My List queue.

I am curious what the general take of the recent Netflix announcement is with regard to our ability to control the traffic. Announcement here . I have an opinion as to whether it is necessary, but that is another subject altogether.

It is clear there is only so much application ID that can occur if the data is encrypted, but can it be combined with URL rules to control the traffic? We would, for example, have to know all of the URLs/subnets that youtube (or Netflix) videos are streamed from. Is that even possible?

We have yet to implement any decryption, but I see that it is going to have to happen at some point. But if we implemented it, is anything going to be able to keep up with the decryption process when it is decrypting a bunch streaming data?

With decryption, you have access to the sub-functions within that application (ie: being able to tell the difference between netflix queue management and streaming, or the difference between facebook-posting and facebook-file-transfer).

Jared is absolutely correct regarding using decryption to enable inspection. To address the performance concern: whether or not decryption has an appreciable effect on your performance depends entirely on *how much* you are decrypting. Each platform has specific upper limits in terms of the maximum number of concurrent decrypted sessions. The additional overhead caused by decryption will depend on this volume. You can limit the scope of decrypted traffic using different criteria (e.g. URL category). This approach will let you inspect things that need to be, like youtube, facebook, etc. while not wasting resources on sessions that probably don't need to be (e.g. online banking, healthcare, etc).

It depends if Netflix just wants to protect the name of the content being watched, or if they really want hide the fact you watch Netflix at all. Like Jared said, you can see the domain name during the SSL/TLS handshake, so you will still be able to block Netflix if you want. After all, if a user cannot login to the main page, he won't be able to watch anything.

The problem I see is more with the QoS. The content is currently streamed from a lot of servers using domain names ending in *.nflxvideo.net. If Netflix encrypts those streams and change the domain name to something less obvious, the firewall will see it as generic encrypted traffic and the QoS rule for Netflix won't match, unless you have a decryption rule in place. I guess there will also be the option of simply giving a low priority to generic encrypted trafic.

Maybe a better example is: Someone logs onto Google.com, then goes to youtube via the google apps link. Unless I am missing something, the PA is seeing this traffic as SSL as the call to google is already encrypted.

90f70e40cf