Re: Twitter Now Lets You Pin And Swipe Between Up To Five Lists
Corette Guitreau
Overcoming these limitations required major changes to our ingestion pipeline and our indexing system, but we believe the results were worth the effort. Tweets are now available for searching within one second of creation, which allows us to power product features with strict real-time requirements, such as real-time conversations or the profile pages. Let's take a closer look at how we've achieved this.
Twitter Now Lets You Pin and Swipe Between Up to Five Lists
Download File --->>> https://lpoms.com/2yLKrV
The core of almost all search systems is a data structure called an inverted index. An inverted index is designed to quickly answer questions like "Which documents have the word cat in them?". It does this by keeping a map from terms to posting lists. A term is typically a word, but is sometimes a conjunction, phrase, or number. A posting list is a list of document identifiers (or document IDs) containing the term. The posting list often includes extra information, like the position in the document where the term appears, or payloads to improve the relevance of our ranking algorithms.1
We use Lucene as our core indexing technology. In standard Lucene, an index is subdivided into chunks called segments, and document IDs are Java integers. The first document indexed in a particular segment is assigned an ID of 0, and new document IDs are assigned sequentially. When searching through a segment, the search starts at the lowest document IDs in the segment and proceeds to the highest IDs in the segment.
To support our requirement of searching for the newest Tweets first, we diverge from standard Lucene and assign document IDs from high to low: the first document in a segment is assigned a maximum ID (determined by how large we want our Lucene segment to be), and each new document gets a smaller document ID. This lets us traverse documents so that we retrieve the newest Tweets first, and terminate queries after we examine a client-specified number of hits. This decision is critical in reducing the amount of time it takes to evaluate a search query and therefore in letting us scale to extremely high request rates.
In the new document ID assignment scheme, each Tweet is assigned a document ID based on the time that it was created. We needed to fit our document IDs into a 31-bit space, because Lucene uses positive Java integers as document IDs. Each document ID is unique within a segment, and our segments are usually writable for about 12 hours. We decided to allocate 27 bits to store timestamps with millisecond granularity, which is enough for a segment to last for a bit more than 37 hours. We use the last four bits as a counter for the number of Tweets with the same timestamp. This means that if we get more than 24 (16) Tweets with the same millisecond timestamp, then some of them will be assigned a document ID that is in a slightly incorrect order. In practice, this is extremely rare, and we decided that this downside was acceptable because we often ran into a similar situation in the old system when a Tweet was delayed for more than 15 seconds, which also resulted in the assignment of an unordered document ID.
Searcher threads would start at the most recently added item of the linked list and follow pointers until reaching the end of the list. Writers would only add new items to the start of the list, either by adding a new posting in the existing array or creating a new block and adding the new posting to the new block. After adding the item and setting up the links correctly, the writer would atomically update its pointer to the new head of the linked list. Searchers would either see the new pointer or the old pointer, but never an invalid pointer.
These linked lists supported a single writer and many searchers without using any locks, which was crucial for our systems: We had a searcher for every CPU core, with tens of cores per server, and locking out all of the searchers every time we needed to add a new document would have been prohibitively expensive. Prepending to a linked list was also very fast (O(1) in the size of the posting list) as it just required following a single pointer, allocating a new element, and updating the pointer.
To support this new requirement, we used a new data structure: skip lists. Skip lists support O(log n) lookups and insertions into a sorted set or map, and are relatively easy to adapt to support concurrency.
When we add an element, we always add it to the bottom level. If we add the element at level N, we randomly decide to add the element at level N + 1 with 20% probability and continue recursively until we don't make the random decision to add an element at the next higher level. This gives us 1/5th as many elements at each higher level. We have found a 1:5 ratio to be a good tradeoff between memory usage and speed.
Second, when we search for an element in a skip list, we track our descent down the levels of the skip list, and save that as a search finger. Then, if we later need to find the next posting with a document ID greater than or equal to a given value (assuming the value we are seeking is higher than the original value we found), we can start our search at the pointers given by the finger. This changes our lookup time from O(log n), where n is the number of elements in the posting list, to O(log d), where d is the number of elements in the posting list between the first value and the second value. This is critical because one of the most common and expensive queries in a search engine are conjunctions, which rely heavily on this operation.
Fourth, we only allocate one pointer for every level of the skip list. In a typical skip list, a node will have a value, a pointer to the next largest element in the list and a pointer to the lower level of the skip list. This means that a new value allocated into the second level will require the space for two values and four pointers. We avoid this by always allocating skip list towers contiguously. Each level K pointer will only point to other level K pointers, so to extract the value associated with a level K pointer P, you read the value at P - K. Once you reach a node with a value greater than the one you are searching for, you go back to the previous node, and descend a level by simply subtracting one from the pointer. This lets us allocate a value into the second level by just consuming the space for the value and two pointers. It also reduces the amount of time we need to spend chasing pointers, because a pointer into a tower is likely to be on the same cache line as the lower pointers in that tower.
In our new system, newer documents are no longer guaranteed to have smaller document IDs. To support atomic updates to the skip list posting lists, we take advantage of the fact that the skip lists allocate new values at the end of a given allocation pool. When a searcher is created, it atomically gets a copy of a data structure that tracks the current maximum pointer into each allocation pool, which we call the published pointer. Then, whenever the searcher traverses a posting list, if the address of a posting is greater than the published pointer, the searcher will skip over that posting, without returning the document as a potential match. This lets us update the underlying data structure while the searcher is running, but preserves our ability to atomically create and update documents, and achieves our goal of separating data structure level atomicity from the logical, application layer atomicity.
Because of the changes to our indexing system, we no longer needed our ingestion pipeline to have a buffer to sort all incoming Tweets. This was a big win for reducing our indexing latency, but we still had the delay caused by waiting for some fields to become available.
This approach allowed us to remove the final artificial delay in our ingestion pipeline, at the cost of very new Tweets not having complete data. Since most search use cases rely only on fields that are immediately available, we believe this was an acceptable price to pay.
We decided that the best way to test our new ingestion pipeline (in addition to writing many new tests) was to deploy it alongside the old ingestion pipeline. This meant that during the rollout period we had two full copies of indexing data. This strategy allowed us to gradually migrate to the new ingestion pipeline, and at the same time, we could easily switch back to the old stream of data if there was a problem.
We also set up a staging cluster of indexing servers that mirrored our production environment as closely as possible and started sending a fraction of our production requests to these servers in addition to production servers (a technique known as dark reads). This allowed us to stress test the changes to our core indexing data structures with real production traffic and load, with both the old and new streams of Tweet data. Once we were confident that the new data structures were working as expected, we reused this staging cluster to test the correctness of the data produced by the new ingestion pipeline. We set up this cluster to index the data produced by the new ingestion pipeline and compared the responses to production. Doing so revealed a few subtle bugs, which only affected dark traffic. Once we fixed them, we were ready to deploy the change to production.
Making changes to a data storage and retrieval system introduces unique challenges, especially when those systems serve hundreds of thousands of queries per second. To get low search indexing latency at Twitter, we needed to make significant changes to our ingestion pipeline and the core data structure used in our indexes, extensively test these changes, and carefully deploy them. We believe the effort was worth it: indexing a Tweet now takes one second, which allows us to satisfy the requirements of even the most real-time features in our product.
On an iPhone, you can check how much data each of your apps use by going to Settings > Cellular. For each the apps on the alphabetical list, you'll see a small number listed below its title that shows how much data it has used. Scroll to the bottom to see when it started counting this data usage, which is likely either when you first activated your iPhone or installed the app in question. At the bottom of the list, you can tap the Reset Statistics button to start a new count, which could be useful if you do this at the beginning of the month or your billing cycle and then set a reminder to check back 30 days later.
7fc3f7cf58