Embedded BitTorrent: 4 of 5
Guy Macon
Embedded BitTorrent: 4 of 5
Inside the Attack that Crippled Revision3
29 May 2008 by Jim Louderback, CEO of Revision3
http://revision3.com/blog/2008/05/29/inside-the-attack-that-crippled-revision3
As many of you know, Revision3's servers were brought down over the
Memorial Day weekend by a denial of service attack. It's an all too common
occurrence these days. But this one wasn't your normal cybercrime - there's
a chilling twist at the end. Here's what happened, and why we're even more
concerned today, after it's over, than we were on Saturday when it started.
It all started with just a simple "hi". Now "hi" can be the sweetest word
in the world, breathlessly whispered into your ear by a long-lost lover, or
squealed out by your bouncy toddler at the end of the day. But taken to
excess - like by a cranky 3-year old-it gets downright annoying. Now
imagine a room full of hyperactive toddlers, hot off of a three hour
Juicy-Juice bender, incessantly shrieking "hi" over and over again, and you
begin to understand what our poor servers went through this past weekend.
On the Internet, computers say hi with a special type of packet, called
"SYN". A conversation between devices typically requires just one short SYN
packet exchange, before moving on to larger messages containing real data.
And most of the traffic cops on the Internet - routers, firewalls and load
balancers - are designed to mostly handle those larger messages. So a flood
of SYN packets, just like a room full of hyperactive screaming toddlers,
can cause all sorts of problems.
For adults, it's typically an inability to cope, followed either by quickly
fleeing the room, or orchestrating a massive Teletubbies intervention.
Since they lack both legs and a ready supply of plushies, Internet devices
usually just shut down.
That's what happened to us. Another device on the Internet flooded one of
our servers with an overdose of SYN packets, and it shut down - bringing
the rest of Revision3 with it. In webspeak it's called a Denial of Service
attack - aka DoS - and it happens when one machine overwhelms another with
too many packets, or messages, too quickly. The receiving machine attempts
to deal with all that traffic, but in the end just gives up.
In its coverage Tuesday CNet asked the question, "Now who would want to
attack Revision3?" Who indeed? So we set out to find out.
Internet attacks leave lots of evidence. In this case it was pretty easy to
see exactly what our shadowy attacker was so upset about. It turns out that
those zillions of SYN packets were addressed to one particular port, or
doorway, on one of our web servers: 20000. Interestingly enough, that's the
port we use for our BitTorrent tracking server. It seems that someone was
trying to destroy our BitTorrent distribution network.
Let me take a step back and describe how Revision3 uses BitTorrent, aka BT.
The BT protocol is a peer to peer scheme for sharing large files like
music, programs and video. By harnessing the peer power of many computers,
we can easily and cheaply distribute our huge HD-quality video shows for a
lot less money. To get started, the person sharing that large file first
creates a small file called a "torrent", which contains metadata, along
with which server will act as the conductor, coordinating the sharing. That
server is called the tracking server, or "tracker". You can read much more
about BitTorrent at Wikipedia, if you really want to understand how it
works.
Revision3 runs a tracker expressly designed to coordinate the sharing and
downloading of our shows. It's a completely legitimate business practice,
similar to how ESPN puts out a guide that tells viewers how to tune into
its network on DirecTV, Dish, Comcast and Time Warner, or a mall might
publish a map of its stores.
But someone, or some company, apparently took offense to Revision3 using
BitTorrent to distribute its own slate of shows. Who could that be?
Along with where it's bound, every Internet packet has a return address.
Often, particularly in cases like this, it's forged - or spoofed. But
interestingly enough, whoever was sending these SYN packets wasn't shy. Far
from it: it's as if they wanted us to know who they were.
A bit of address translation, and we'd discovered our nemesis. But instead
of some shadowy underground criminal syndicate, the packets were coming
from right in our home state of California. In fact, we traced the vast
majority of those packets to a public company called ArtistDirect
(ARTD.OB). Once we were able to get their Internet provider on the line,
they verified that yes, indeed, that Internet address belonged to a
subsidiary of ArtistDirect, called MediaDefender.
Now why would MediaDefender be trying to put Revision3 out of business?
Heck, we're one of the biggest defenders of media around. So I stopped by
their website and found that MediaDefender provides "anti-piracy solutions
in the emerging Internet-Piracy-Prevention industry." The company aims to
"stop the spread of illegally traded copyrighted material over the Internet
and peer-to-peer networks." Hmm. We use the Internet and peer-to-peer
networks to accelerate the spread of legally traded materials that we own.
That's sort of directly opposite to what Media Defender is supposed to be
doing.
Who pays MediaDefender to disrupt peer to peer networks? I don't know who's
ponying up today, but in the past their clients have included Sony,
Universal Music, and the central industry groups for both music and movies
-- the RIAA and MPAA. According to an article by Ars Technica, the company
uses "its array of 2,000 servers and a 9GBps dedicated connection to
propagate fake files and launch denial of service attacks against
distributors." Another Ars Technica story claims that MediaDefender used a
similar denial of service attack to bring down a group critical of its
actions.
Hmm. Now this could have been just a huge misunderstanding. Someone could
have incorrectly configured a server on Friday, and left it to flood us
mercilessly with SYN packets over the long Memorial Day weekend. If so,
luckily it was pointed at us, and not, say, at the intensive care unit at
Northwest Hospital and Medical Center But Occam's razor leads to an
entirely different conclusion.
So I picked up the phone and tried to get in touch with ArtistDirect
interim CEO Dimitri Villard. I eventually had a fascinating phone call with
both Dimitri Villard and Ben Grodsky, Vice President of Operations at Media
Defender.
First, they willingly admitted to abusing Revision3's network, over a
period of months, by injecting a broad array of torrents into our tracking
server. They were able to do this because we configured the server to track
hashes only - to improve performance and stability. That, in turn, opened
up a back door which allowed their networking experts to exploit its
capabilities for their own personal profit.
Second, and here's where the chain of events come into focus, although not
the motive. We'd noticed some unauthorized use of our tracking server, and
took steps to de-authorize torrents pointing to non-Revision3 files. That,
as it turns out, was exactly the wrong thing to do. MediaDefender's
servers, at that point, initiated a flood of SYN packets attempting to
reconnect to the files stored on our server. And that torrential cascade of
"Hi"s brought down our network.
Grodsky admits that his computers sent those SYN packets to Revision3, but
claims that their servers were each only trying to contact us every three
hours. Our own logs show upwards of 8,000 packets a second.
"Media Defender did not do anything specific, targeted at Revision3?,
claims Grodsky. "We didn't do anything to increase the traffic" - beyond
what they'd normally be sending us due to the fact that Revision3 was
hosting thousands of MediaDefender torrents improperly injected into our
corporate server. His claim: that once we turned off MediaDefender's
back-door access to the server, "traffic piled up (to Revision3 from
MediaDefender servers because) it didn't get any acknowledgment back."
Putting aside the company's outrageous use of our servers for their own
profit, and the large difference between one connection every three hours
and 8,000 packets a second, I'm still left to wonder why they didn't just
tell us our basement window was unlocked. A quick call or email and we'd
have locked it up tighter than a drum.
It's as if McGruff the Crime Dog snuck into our basement, enlisted an army
of cellar rats to eat up all of our cheese, and then burned the house down
when we finally locked him out - instead of just knocking on the front door
to tell us the window was open.
In the end, here's what I know:
* A torrential flood of SYN packets rained down on Revision3's network over
Memorial Day weekend.
* Those packets - up to 8,000 a second - came primarily from computers
controlled by MediaDefender, who is in the business of shutting down
illegal torrent sites.
* Revision3 suffered measurable harm to its business due to that flood of
packets, as the attacks on our legitimate and legal Torrent Tracking server
spilled over into our entire Internet infrastructure. Thus we were unable
to serve videos and advertising through much of the weekend, and into
Tuesday - and even our internal email servers were brought down.
* Denial of service attacks are illegal in the US under 12 different
statutes, including the Economic Espionage Act and the Computer Fraud and
Abuse Act.
Although I can only guess, here's what I think really happened. Media
Defender was abusing one of Revision3's servers for their own purposes -
quite without our approval. When we closed off their backdoor access,
MediaDefender's servers freaked out, and went into attack mode - much like
how a petulant toddler will throw an epic tantrum if you take away an
ill-gotten Oreo.
That tantrum threw upwards of 8,000 SYN packets a second at our servers.
And that was enough to bring down both our public facing site, our RSS
server, and even our internal corporate email - basically the entire
Revision3 business. Smashing the cookie jar, as it were, so that no one
else could have any Oreos either.
Was it malicious? Intentional? Negligent? Spoofed? I can't say. But what I
do know is that the FBI is looking into the matter - and it's far more
serious than toddlers squabbling over broken toys and lost cookies.
MediaDefender claims that they have taken steps to ensure this won't happen
again. "We've added a policy that will investigate open public trackers to
see if they are associated with other companies", promised Grodsky, "and
first will make a communication that says, hey are you aware of this."
In the end, I don't think Media Defender deliberately targeted Revision3
specifically. However, the company has a history of using their servers to,
as Ars Technica said, "launch denial of service attacks against
distributors." They saw us as a "distributor" - even though we were using
BitTorrent for legitimate reasons. Once we shut them out, their vast
network of servers were automatically programmed to implement a scorched
earth policy, and shut us down in turn. The long Memorial Day weekend
holiday made it impossible for us to contact either Media Defender or their
ISP, which only exacerbated the problem.
All I want, for Revision3, is to get our weekend back - both the countless
hours spent by our heroic tech staff attempting to unravel the mess, and
the revenue, traffic and entertainment that we didn't deliver.
If it can happen to Revision3, it could happen to your business too. We're
simply in the business of delivering entertainment and information - that's
not life or death stuff. But what if MediaDefender discovers a tracker
inside a hospital, fire department or 911 center? If it happened to us, it
could happen to them too. In my opinion, Media Defender practices risky
business, and needs to overhaul how it operates. Because in this country,
as far as I know, we're still innocent until proven guilty - not drawn,
quartered and executed simply because someone thinks you're an outlaw.
-- Jim Louderback, CEO - Revision3
-------------------------------------------------------------------------
Guy Macon <http://www.GuyMacon.com/> Guy Macon <http://www.GuyMacon.com/>
Guy Macon <http://www.GuyMacon.com/> Guy Macon <http://www.GuyMacon.com/>
Guy Macon <http://www.GuyMacon.com/> Guy Macon <http://www.GuyMacon.com/>
Guy Macon <http://www.GuyMacon.com/> Guy Macon <http://www.GuyMacon.com/>
Guy Macon
Embedded BitTorrent: 5 of 5
Peer-to-peer poisoners: A tour of MediaDefender
18 March 2007 By Nate Anderson
http://arstechnica.com/articles/culture/mediadefender.ars
A war of attrition
When your company poisons peer-to-peer networks for a living, public
relations usually takes a back seat to discretion; quiet is the rule in the
P2P content-protection industry. That's why Jonathan Lee, the company's VP
of business development, isn't worried that the corporate web site is down
when I reach him in his Santa Monica office. "It's kind of ugly anyway," he
says.
For a company like MediaDefender, the largest such firm in existence,
privacy comes naturally, but a 2005 acquisition by ArtistDirect has
encouraged the firm to take its services public as it starts to look beyond
its original client base-music labels and movie studios-and dives headfirst
into the brave new world of providing legitimate P2P content for
advertisers.
Such advertising deals may be the future, but the company's bread and
butter continues to be P2P disruption of movies and music downloads.
MediaDefender is quite good at this, as it should be after five years of
anti-piracy work. Unlike DRM providers that focus on protecting the
product, MediaDefender tries to protect the distribution channel-and only
for a limited time. Recognizing that it is impossible to shut down the
sharing of copyrighted works, the company focuses instead on mitigation.
Record labels and movie companies can pay between $5,000 and $15,000 per
title for differing levels of protection that extend over different time
periods.
For most content owners, MediaDefender's services are needed at the
beginning of a product's life cycle. Lee points out that most movies and
albums makes the majority of their money in the first few months after
release. MediaDefender's value proposition is not that it can stop such
files from being shared, but that it can make sharing difficult for a month
or two in order to give the legitimate product more traction.
Is it live or is it MediaDefender?
How it works
To work its magic on the various P2P networks, Lee describes four
strategies that MediaDefender uses. All four are powered by a back end of
2,000 servers co-located around the world, and the company has contracts
for 9GBps of Internet bandwidth. For a 60-person operation, these numbers
are (to put it mildly) a bit high, but the scale of its system usually
ensures that the company gets prompt attention and good deals when it goes
shopping. It also means that employees who stay late after work to game on
the corporate LAN always have a good connection.
Those 2,000 servers do four things that MediaDefender refers to as
decoying, spoofing, interdiction, and swarming. Here's how they work...
Four main methods
Decoying. This, in a nutshell, is the serving of fake files that are
generally empty or contain a trailer. The goal is to make legitimate
content a needle in a haystack, so MediaDefender works hard to ensure that
its copies of files show up in the top ten spots when certain keywords are
searched for. Everything about the file is tailored to look like the work
of pirates, from the file size (movies are often compressed enough to fit
on a CD) to the naming conventions to the pirate scene tag. With massive
bandwidth and plenty of servers, the company has little trouble in getting
these decoy files to appear at the top of search results, but decoying has
a down side: the bandwidth. Because MediaDefender actually serves these
large but bogus files, it incurs a significant bandwidth bill by using this
technique.
Spoofing. Spoofing sends searchers down dead ends. MediaDefender coders
have written their own software that interacts with the various P2P
protocols and sends bogus returns to search requests, usually directing
people to nonexistent locations. Because most people only look at the top
five search results, MediaDefender tries to frustrate their first attempts
to download a file in hopes that they will just give up.
Interdiction. While the first two techniques try to prevent searchers from
locating files, interdiction prevents distributors from serving them. The
tool is generally used when media is leaked or newly released; the goal is
to slow its spread in those crucial first days. MediaDefender servers
attempt to create constant connections to the files in question, saturating
the provider's upstream bandwidth and preventing anyone else from grabbing
the data.
Swarming. Though he acknowledges the BitTorrent networks can be hard to
disrupt, Lee points out that MediaDefender can use "swarming" to make life
more difficult for users trying to download copyrighted content. BitTorrent
works by using a hash file to reassemble a file from many pieces, each of
which may have been downloaded from a different user. MediaDefender simply
serves up its chunks of these files, but instead of providing the proper
data, its chunks contain static or nothing at all. BitTorrent will discard
such junk data, but a flood of it can slow a user's download to a crawl.
Does all of this really curtail P2P usage? Lee admits that the company will
never stop file-swapping, but says that isn't the point of what it does.
Instead, the goal is to make files hard to find for a short period of time
so that studios, music labels, and artists can make money from selling the
legitimate product. Companies that use MediaDefender's services will often
run their own download tests (or contract with one of the firms that does
this) to make sure that they are receiving a return on their investment.
Apparently, they are. MediaDefender counts every major music label and most
studios among its clients, with the notable exception of Disney. Lee says
that initially, his company expected to work largely with trade
organizations like the RIAA and the MPAA. When it actually approached them,
however, the trade groups were more focused on court cases and
Congressional lobbying. While they approve of MediaDefenders's work, the
actual contracts are signed directly with labels and studios, many of whom
pay millions for the company's services.
A brave new world: advertising
In recent months, MediaDefender has shifted some of its efforts in a new
direction: using its P2P technology and massive bandwidth to serve files,
rather than stop them. Last year, the company partnered with Jay-Z and Coke
in a widely-covered promotion that saw MediaDefender pushing a legitimate
piece of Jay-Z concert footage to fans who searched for videos by the
artist. In essence, these are "decoys" that contain real content.
The company has also helped promote Vitamin Water commercials that were
deemed too "edgy" for network television, along with video game trailers
and exclusive P2P remixes. The goal is to diversify-a necessary safeguard
in an industry that has few clients. There simply aren't that many major
movie studios and music labels, but there are millions of potential clients
with fat ad budgets who wouldn't mind reaching the millions of young,
tech-savvy people who make use of P2P networks.
Lee says that even music and movie companies have changed their stance in
the last few years, and while none condone illegal downloads, they have
realized that this is a huge potential market. This is especially true for
smaller indie labels, for whom exposure is sometimes more important than
legitimate sales. Some of these small firms have actually paid
MediaDefender to serve content by their acts, often in response to users
searching for a related (but better known) artist.
This mingling of licit and illicit content on P2P networks raises some
questions, of course. How are users to know in advance if content is legal
or not? Are some labels actually encouraging the use of such networks, even
as their trade groups prosecute those who use them? Does serving legitimate
content show confusion about what can and cannot be shared and downloaded?
This was, in fact, a major concern that the industry had. For years,
content owners refused to place any legal material on P2P networks for fear
of legitimizing them. That fear largely vanished in the wake of the Supreme
Court's Grokster decision. Once it was well established that such networks
could be held liable for copyright infringement, content owners actually
felt more free to make use of the networks for legitimate uses of their
own.
But anti-piracy work still accounts for 99 percent of MediaDefender's
work-work that Lee knows is not popular in all circles. Last year, for
instance, the company began recruiting on college campuses for the first
time. Students would approach company reps and tell them that they hated
what they did. "But five minutes later," says Lee, "they came back and
asked us for a job." Hackers, he says, "love screwing with each other," and
MediaDefender gives them an impressive platform and some serious bandwidth
to hack on. Besides, "you can't get that mad" about what the company does,
Lee says with a laugh. "I mean, you're looking for pirated stuff!"
Update:
Various forum posters and bloggers have commented on MediaDefender's
"swarming" claim in particular, arguing that BitTorrent's hash-based
technology prevents file disruption and that MediaDefender could simply not
disrupt the network. We contacted the company for clarification and were
told that the details of their BitTorrent work remain secret, but that the
company does indeed employ swarming on BitTorrent networks.
Because of the anti-corruption technology on such networks, MediaDefender
tries to stall downloads and make files frustrating to grab by serving bad
data. The file corruption discussed in the article should not have
referenced BitTorrent; such swarming causes corruption only on networks
without similar error-checking (the article text has been corrected).
MediaDefender's goal with BitTorrent is to slow down transfers. Making them
slow enough counts as a "win" for the company, though this does seem like a
hollow victory, as the consumer still has the correct file in the end.
Paul E. Bennett
>
>
>
> Embedded BitTorrent: 4 of 5
>
> Inside the Attack that Crippled Revision3
> 29 May 2008 by Jim Louderback, CEO of Revision3
>
http://revision3.com/blog/2008/05/29/inside-the-attack-that-crippled-revision3
>
> As many of you know, Revision3's servers were brought down over the
> Memorial Day weekend by a denial of service attack. It's an all too
> common occurrence these days. But this one wasn't your normal
> cybercrime - there's a chilling twist at the end. Here's what happened,
> and why we're even more concerned today, after it's over, than we were
> on Saturday when it started.
[%X]
> If it can happen to Revision3, it could happen to your business too.
> We're simply in the business of delivering entertainment and
> information - that's not life or death stuff. But what if MediaDefender
> discovers a tracker inside a hospital, fire department or 911 center?
> If it happened to us, it could happen to them too. In my opinion, Media
> Defender practices risky business, and needs to overhaul how it
> operates. Because in this country, as far as I know, we're still
> innocent until proven guilty - not drawn, quartered and executed simply
> because someone thinks you're an outlaw.
>
> -- Jim Louderback, CEO - Revision3
Is a reference to this going to appear in RISKS as it would be worthy of
a mention there too, especially for those more critical legal
applications that may be affected by such practices.
--
********************************************************************
Paul E. Bennett...............<email://Paul_E....@topmail.co.uk>
Forth based HIDECS Consultancy
Mob: +44 (0)7811-639972
Tel: +44 (0)1235-811095
Going Forth Safely ..... EBA. www.electric-boat-association.org.uk..
********************************************************************