Some info on a real-life Akka project
Dag Liodden
I've been following Akka with interest for a long while. It's a great
technology, but have been relatively few real-life stories of what
people are actually using Akka for. Very recently, we decided to
rebuild some parts of the ad platform here at SmartAd and we quickly
realized that we were looking at a number of problem areas where the
actor model and Akka could be a very good fit. We decided to give it a
go and now, only a few weeks later, we're in production with a
substantial amount of Akka dependent code. We're by no means Akka
experts, but it turns out you don't need to be an expert in order to
reap some great benefits - given the right use case. In any case, I've
done a quick and dirty write-up of what we've been doing and we're
hoping that sharing this can be of interest to the Akka community.
SmartAd is building a real-time ad exchange platform for advertising
on mobile and connected devices. Real-time ad exchanges allows for
advertisers (among other things) to target audiences instead of buying
fixed set of ad slots that will be displayed “randomly” to users. To
developers without experience in the ad space, this might seem boring,
but real-time ad exchanges present some really interesting technical
challenges.
Take for instance the process backing a page view with ads served by a
real-time ad exchange auction (somewhat simplified):
1. A user opens a site (or app) which has ads in it.
2. As the page / app loads, the ad serving components fires off a
request to the ad exchange (this might just be due to an image tag on
the page).
3. The ad exchange enriches the request with any information about the
current user (tracking cookies are often employed for this) and and
display context information (“news article about parenting”, “blog
about food” etc).
4. The ad exchange forwards the enriched request to all bidders
registered with the ad exchange.
5. The bidders consider the provided user information and responds
with what price they are willing to pay for this particular ad slot.
6. The ad exchange picks the highest bidder and ensures that the
winning bidder’s ad is shown to to user.
Any latency in this process directly influences user experience
latency, so this has to happen really fast. All-in-all, the total time
should not exceed about 100ms and most ad exchanges allow bidders to
spend about 60ms (including network time) to return their bids. That
leaves the ad exchange with less than 40ms to facilitate the auction.
At SmartAd, this happens billions of times per month / tens of
thousands of times per second.
SmartAd is building bidders which will participate in auctions
facilitated by other ad exchanges, but we’re also building our own. We
are using Akka in several ways in several parts of the system. Here
are some examples:
Plain old parallelization
During an auction in the real-time exchange, it’s obvious that all
bidders must receive the bid requests in parallel. An auctioneer actor
sends the bid requests to bidder actors which in turn handles
throttling and eventually IO. We use futures in these requests and the
auctioneer discards any responses which arrive too late.
Inside our bidders, we also rely heavily on parallel execution. In
order to determine how much to pay for an ad slot, several data stores
are queried for information pertinent to the current user. In a
“traditional” system, we’d be doing this sequentially, but again, due
to the extreme latency constraints, we’re doing this concurrently.
Again, this is done with futures and data that is not available in
time, get cut from the decision making (and logged :)).
Maintaining state under concurrent load
This is probably the de facto standard use case for the actors model.
Bidders internal to our system are actors backed by a advertiser
campaign. A campaign includes, among other things, budget and “pacing”
information. The budget determines how much money to spend for the
duration of the campaign, whereas pacing information might set
constraints on how quickly or slowly the money should be spent. Ad
traffic changes from day to day and from hour to hour and our spending
algorithms considers past performance in order to spend the right
amount of money at the right time. Needless to say, these algorithms
use a lot of state and this state is in constant flux. A bidder with a
high budget may see tens of thousands of bid requests per second.
Luckily, due to round-robin load-balancing and the predictability of
randomness under heavy traffic, the bidder actors do not share state
across cluster nodes, they just share their instance count so they
know which fraction of the campaign budget to try to spend.
Pacing is also done for external bidders. Each 3rd party bidder end-
point has an actor coordinating requests and measuring latency and
throughput. The actor never blocks itself, but when an incoming bid
request is received, it considers the current performance of the 3rd
party system and decides whether to pass on the request and respond
negatively immediately, or forward the request to the 3rd party
request executor component (which handles the IO).
Batch processing
We store a lot of data about every single ad request we serve and this
is stored in a key-value data store. Due to the performance
characteristics of the data store, it is not feasible to store every
single data point one at at time - it must be batched up and performed
in parallel. We don’t need a durable messaging system for this (losing
a couple of hundred data points is no biggie). All our data logging
happens asynchronously and we have a basic load-balanced actors which
batches incoming messages and writes on regular intervals (using
Scheduler) or whenever the specified batch size has been reached.
Analytics
Needless to say, it’s not feasible / useful to store our traffic
information in a relational database. A lot of analytics and data
analysis is done “offline” with map / reduce on top the data store,
but this doesn’t work well for real-time analytics which our customers
love. We therefore have metrics actors that receives campaign bidding
and click / impression information in real-time, aggregates this
information over configurable periods of time and flushes it to the
database used for customer dashboards for “semi-real-time” display.
Five minute history is considered real-time in this business, but in
theory, we could have queried the actors directly for really real-time
data. :)
Our Akka journey started as a prototyping project, but Akka has now
become an integral part of our system. All of the above mentioned
components, except the 3rd party bidder integration, have been running
in production for a couple of weeks (on Akka 1.0RC3) and we have not
seen any issues at all so far.
Hope this has been an interesting read!
Cheers,
Dag
BTW: TapAd changed name to SmartAd last week, hence the tapad.com
email address.
Debasish Ghosh
--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
--
Debasish Ghosh
http://manning.com/ghosh
Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg
Jonas Bonér
--
Jonas Bonér
Specialist at Large
work: http://scalablesolutions.se
code: http://akka.io
twtr: @jboner
email address. -- You received this message because you are subscribed to the Google Groups "Akka User List" group. To post to this group, send email to akka...@googlegroups.com. To unsubscribe from this group, send email to akka-user+...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
Salanki
Martin Krasser
What akka-* components/modules have you actually used in your app? Only
akka-actor or others as well?
Am 15.02.11 03:24, schrieb Dag Liodden:
> Hi guys,
>
> I've been following Akka with interest for a long while. It's a great
> technology, but have been relatively few real-life stories of what
> people are actually using Akka for. Very recently, we decided to
> rebuild some parts of the ad platform here at SmartAd and we quickly
> realized that we were looking at a number of problem areas where the
> actor model and Akka could be a very good fit. We decided to give it a
> go and now, only a few weeks later, we're in production with a
> substantial amount of Akka dependent code. We're by no means Akka
> experts, but it turns out you don't need to be an expert in order to
> reap some great benefits - given the right use case. In any case, I've
> done a quick and dirty write-up of what we've been doing and we're
> hoping that sharing this can be of interest to the Akka community.
>
> SmartAd is building a real-time ad exchange platform for advertising
> on mobile and connected devices. Real-time ad exchanges allows for
> advertisers (among other things) to target audiences instead of buying
> fixed set of ad slots that will be displayed �randomly� to users. To
> developers without experience in the ad space, this might seem boring,
> but real-time ad exchanges present some really interesting technical
> challenges.
>
> Take for instance the process backing a page view with ads served by a
> real-time ad exchange auction (somewhat simplified):
>
> 1. A user opens a site (or app) which has ads in it.
> 2. As the page / app loads, the ad serving components fires off a
> request to the ad exchange (this might just be due to an image tag on
> the page).
> 3. The ad exchange enriches the request with any information about the
> current user (tracking cookies are often employed for this) and and
> display context information (�news article about parenting�, �blog
> about food� etc).
> 4. The ad exchange forwards the enriched request to all bidders
> registered with the ad exchange.
> 5. The bidders consider the provided user information and responds
> with what price they are willing to pay for this particular ad slot.
> 6. The ad exchange picks the highest bidder and ensures that the
> winning bidder�s ad is shown to to user.
>
> Any latency in this process directly influences user experience
> latency, so this has to happen really fast. All-in-all, the total time
> should not exceed about 100ms and most ad exchanges allow bidders to
> spend about 60ms (including network time) to return their bids. That
> leaves the ad exchange with less than 40ms to facilitate the auction.
> At SmartAd, this happens billions of times per month / tens of
> thousands of times per second.
>
> SmartAd is building bidders which will participate in auctions
> facilitated by other ad exchanges, but we�re also building our own. We
> are using Akka in several ways in several parts of the system. Here
> are some examples:
>
> Plain old parallelization
> During an auction in the real-time exchange, it�s obvious that all
> bidders must receive the bid requests in parallel. An auctioneer actor
> sends the bid requests to bidder actors which in turn handles
> throttling and eventually IO. We use futures in these requests and the
> auctioneer discards any responses which arrive too late.
>
> Inside our bidders, we also rely heavily on parallel execution. In
> order to determine how much to pay for an ad slot, several data stores
> are queried for information pertinent to the current user. In a
> �traditional� system, we�d be doing this sequentially, but again, due
> to the extreme latency constraints, we�re doing this concurrently.
> Again, this is done with futures and data that is not available in
> time, get cut from the decision making (and logged :)).
>
> Maintaining state under concurrent load
> This is probably the de facto standard use case for the actors model.
> Bidders internal to our system are actors backed by a advertiser
> campaign. A campaign includes, among other things, budget and �pacing�
> information. The budget determines how much money to spend for the
> duration of the campaign, whereas pacing information might set
> constraints on how quickly or slowly the money should be spent. Ad
> traffic changes from day to day and from hour to hour and our spending
> algorithms considers past performance in order to spend the right
> amount of money at the right time. Needless to say, these algorithms
> use a lot of state and this state is in constant flux. A bidder with a
> high budget may see tens of thousands of bid requests per second.
> Luckily, due to round-robin load-balancing and the predictability of
> randomness under heavy traffic, the bidder actors do not share state
> across cluster nodes, they just share their instance count so they
> know which fraction of the campaign budget to try to spend.
>
> Pacing is also done for external bidders. Each 3rd party bidder end-
> point has an actor coordinating requests and measuring latency and
> throughput. The actor never blocks itself, but when an incoming bid
> request is received, it considers the current performance of the 3rd
> party system and decides whether to pass on the request and respond
> negatively immediately, or forward the request to the 3rd party
> request executor component (which handles the IO).
>
> Batch processing
> We store a lot of data about every single ad request we serve and this
> is stored in a key-value data store. Due to the performance
> characteristics of the data store, it is not feasible to store every
> single data point one at at time - it must be batched up and performed
> in parallel. We don�t need a durable messaging system for this (losing
> a couple of hundred data points is no biggie). All our data logging
> happens asynchronously and we have a basic load-balanced actors which
> batches incoming messages and writes on regular intervals (using
> Scheduler) or whenever the specified batch size has been reached.
>
> Analytics
> Needless to say, it�s not feasible / useful to store our traffic
> information in a relational database. A lot of analytics and data
> analysis is done �offline� with map / reduce on top the data store,
> but this doesn�t work well for real-time analytics which our customers
> love. We therefore have metrics actors that receives campaign bidding
> and click / impression information in real-time, aggregates this
> information over configurable periods of time and flushes it to the
> database used for customer dashboards for �semi-real-time� display.
> Five minute history is considered real-time in this business, but in
> theory, we could have queried the actors directly for really real-time
> data. :)
>
> Our Akka journey started as a prototyping project, but Akka has now
> become an integral part of our system. All of the above mentioned
> components, except the 3rd party bidder integration, have been running
> in production for a couple of weeks (on Akka 1.0RC3) and we have not
> seen any issues at all so far.
>
> Hope this has been an interesting read!
>
> Cheers,
>
> Dag
>
> BTW: TapAd changed name to SmartAd last week, hence the tapad.com
> email address.
>
--
Martin Krasser
blog: http://krasserm.blogspot.com
code: http://github.com/krasserm
twitter: http://twitter.com/mrt1nz
√iktor Klang
--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
--
Viktor Klang,
Code Connoisseur
Work: Scalable Solutions
Code: github.com/viktorklang
Follow: twitter.com/viktorklang
Read: klangism.tumblr.com
Dag Liodden
there are no obvious reasons for us to adopt more modules at this
time.
Orjan
Dag Liodden
1. Our company name is now Tapad (again)
2. We're out of stealth mode (http://www.tapad.com)
3. We're hiring Akka developers in New York :) (http://www.tapad.com/
careers-back_end_developer_engineering-nyc.html)