Denial of Service, Deconstructed
Denial-of-service attacks are an old and crass way to disrupt a network, and yet still are immensely effective. DoS attacks overload the pipes that connect computers to the Internet with massive amounts of legitimate but useless data. DoS attacks create epic traffic jams. The cars in this analogy would be requests for service that hackers send to the target website. Each time the target site gets a request, it must deny it. But because the hacker sends massive numbers of requests from thousands of computers, the target must use nearly all of its time and resources just to deny these requests for service, effectively blocking access to anyone with a legitimate request.
Before that, though, the hacker must create a network of computers big enough to overwhelm the target. They don't buy these computers, they commandeer them. They plant software scripts on systems distributed throughout the world (hence, distributed denial of service, or DDoS). These compromised computers are called zombies, or bots, because they generate attack traffic automatically, without the owners' knowledge.
Hackers create zombies by scanning for exposed systems that they can manipulate remotely. Often these are home and office broadband users. (Lately, existing bot networks have been found scanning for more computers to turn into bots when they're not launching attacks of their own—akin to an army recruiting its soldiers in peacetime. One security consultant said he connected an unsecured computer to the Internet to see what would happen, and it was recruited within three minutes.) Hackers can also insert their attack code through phishing, spyware, viruses and social engineering. Universities have long been popular spots for creating zombies because of the number of easily accessible, unsecured public computers.
With a zombie network in place, the only issue left is scale. The more zombies on a network, and the more aggregate upstream bandwidth they have, the swifter and more severe havoc they can wreak. Several hundred computers could generate 100MB of traffic, enough to knock a small network offline. A 10,000-computer bot network could deliver a 1Gb attack, enough to knock anyone offline who hasn't installed some rudimentary anti-DDoS infrastructure.
Some experts believe that right now different sets of hackers are engaged in an arms race to see who can build the biggest zombie network. Not for bragging rights, but for renting out the networks to anyone who wants to launch an attack, the raw capitalist idea being that the biggest network will generate the best rental business.
Tuesday, Nov. 25, 2003: Running Out of Time
The extortionists' e-mail that arrived on this morning demonstrated that they were losing whatever patience they had: [all typos sic] "I told you that if you try and f*** with us that your site will be down forever.... The excuse that you were in the hospital does not matter to me. So here are your choices: 1) You have until 4pm est today to send us our $40K. 2) You have until 4pm est Wednesday to send us $50K if you can not send the $40K today. 3) You do not pay and your site will be down for 4 days starting Thursday and it will cost you $75K to come back up Monday. 4) You do nothing and do not respond to this email within an hour and we will make sure you are down forever...."
Richardson was panicked. He can't remember precisely when—the entire week has blurred in his memory—but by this time, he had reported the crime to the National Hi-Tech Crime Unit (NHTCU) in Scotland Yard. According to an NHTCU spokeswoman, the unit had already opened a similar investigation with a British gaming site called CanBet.
According to Richardson and Lyon, the NHTCU encouraged Richardson to wire two extortion payments of a few thousand dollars each to separate Western Union offices in Eastern Europe. The NHTCU wanted to nab anyone who showed up to take the cash. (NHTCU won't confirm this; the spokeswoman said the unit does not discuss investigative tactics.) Richardson agreed, but for a different reason: He wanted his site back up. "I knew another person [in the industry] who was successful getting back online by sending three or four small payments like this," Richardson says, "and those guys didn't even have a solution to the problem when they paid. I knew Barrett was getting closer and closer to a solution. So I sent the payments, thinking maybe I can get a good week out of this."
But no one took the bait. After about two weeks, Richardson pulled the money back.
Wednesday, Nov. 26, 2003: Barrett's Big Bet
From Sacramento, Lyon instructed the PureGig engineers who would turn on his system 630 miles southeast, in Phoenix. Another 2,400 miles southeast from Phoenix, everyone at BetCris waited impatiently.
Lyon's system intercepted traffic headed for BetCris's servers in Costa Rica, diverted it to his creation in Phoenix, scrubbed off the attack traffic and delivered legitimate traffic back to Costa Rica. It was designed to bar DDoS traffic from touching BetCris. If the system failed, it couldn't defend BetCris, and it wouldn't be able to send legitimate traffic to Costa Rica. But BetCris itself wasn't getting attacked. The system did a lot of other stuff too: monitoring, capacity planning, logging and analysis.
It wasn't perfect. After it was installed, Lyon had to tweak routers on the network, install new versions of software and add capacity to his system. The extortionists kept changing attack vectors, and Lyon and his team kept tweaking. It was a constant battle, but Lyon was confident that the system would enable BetCris.com to stay online. Wilson at PureGig called Lyon's system "ingenious" not because it was unique—it was monitoring and filtering at a proxy location—but because Lyon's monitoring and filtering seemed to stop attacks better than any other effort he'd seen.
But when it was first turned on, the extortionists stuffed too much traffic down its throat. Wilson recalls the math: "We had 100MB links to the DNS servers. We went from handling under 2MB per link to, all of a sudden, 600MB." That's six times a full load. Imagine Fenway Park, which holds about 35,000 people. Now imagine 200,000 people trying to get inside Fenway Park at one time.
The DNS servers were overloaded, and Phoenix got tense.
Costa Rica had been tense for nearly a week (as much as half a million dollars in lost revenue), but now BetCris was bordering on despair. Mickey Richardson lacked sleep, and he struggled to make decisions and lead. His IT staff was fracturing, feeling impotent as they watched the attacks and waited for Lyon. BetCris's small call center staff was getting abused around the clock by customers calling in to vent frustration and demand to know what the heck was going on. The simple task of creating a smart message about what was happening eluded Richardson. "You can't just have your call center staff tell people you were hacked," Richardson says, because it creates more questions than answers.
At the same time, his decision not to pay the extortionists was affecting other wagering sites that shared the same ISP and were experiencing network problems. "I'm getting calls from friendly competitors saying, 'Look, Mickey, we paid. Just pay. We're going down because of you.'"
He was running out of time and energy. Richardson remembers around this time having to update his staff—275 or so people who weren't entirely sure they'd have a job soon—and he couldn't even find words. He thought, "I wish they could read my mind because I'm too exhausted to explain it anymore. I don't have any answers."
In hindsight, Richardson says, he would have spent more time preparing for these human issues attached to the crisis—decision making under pressure, keeping the staff together—and less time worrying about technical defenses. Yes, create those technical defenses and make sure you have a crisis response plan. But also focus more on issues like exhaustion and emotional distress, and how they can be handled.
It was in this context that Richardson received an e-mail, at 11:12 a.m. It caused him to feel, for the first time, "blind fear."
"I would like to thank you for not keeping your end of the deal and making this upcoming weekend an enjoyable one for me." The extortionists demanded $75,000, but then seemed to disregard the money. "I do not care how long I have to destroy your business and I will. You will learn the hard way that you do not make a deal and then f*** around with us.... Let the games begin."
Richardson would soon learn they were not bluffing. They could destroy his business, and they were going to try. For BetCris to survive, Lyon's slapdash system in Phoenix, which was just starting to find its purchase, would have to stand up to the biggest DDoS attack any of them had ever seen.
The DNS servers that had overloaded in Phoenix were brought back online in a couple of hours, after Lyon and Wilson adapted some filtering scripts and increased the size of their network pipes.
Lyon then spent Thanksgiving and Friday eating leftover turkey his girlfriend delivered and tweaking his system to absorb bigger DDoS attacks. On Friday, he believed it could handle a 1Gb attack, and he felt good about that. He assured a frayed Richardson that he'd never see an attack that big. It would take tens of thousands of zombie computers.
Which is exactly what happened. It turns out the extortionists had more than 20,000 zombies. PureGig's data center suffered badly, which affected several of its ISP customers. PureGig decided to take Lyon's system offline to fix it.
"The attack went to 1.5Gb, with bursts up to 3Gb. It wasn't targeted at one thing. It was going to routers, DNS servers, mail servers, websites. It was like a battlefield, where there's an explosion over here, then over there, then it's quiet, then another explosion somewhere else," says Lyon. "They threw everything they had at us. I was just in shock."
Richardson recalls the attack: "So I have Barrett on the line, who I think is the second coming, and he says, 'Let me think about this. Give me some time.' And I say, 'OK, I don't want to pressure you. I have faith. But if you don't fix it, I'm out of business.'"
Why Online Extortion Works
It was never supposed to have gotten to this point; Richardson was supposed to have paid long ago. The extortionists expertly optimized the chances of it.
To ensure a quick, quiet transaction, the extortionists did what all extortionists (in the physical or online world) do: They exploited the problem of the commons. An ecological principle, the problem of the commons states that people will act in self-interest if it profits them in the short term, even if that act will hurt everyone, including themselves, in the long term. Every act, every threat, every negotiation tactic, every single move extortionists make is designed to make paying the protection fee not only appealing, but in fact, the smartest business decision you can make in the short term, even if you know in the long run that you haven't stopped the problem at all.
Thus, extortionists attack when it hurts the target the most; they ask for $10,000 to $100,000 (generally considered the sweet spot of extortionist profitability versus victim willingness to pay, depending on the size of the victim company).
In BetCris's case, the extortionists revealed they were Eastern European, which would make them hard to find, never mind prosecute. Online crime laws are weaker in Eastern Europe than in the United States and the desire to enforce them weaker still (and the FBI wouldn't get involved with offshore gaming sites being extorted from overseas).
The online version of extortion provides unique advantages (relative anonymity, low probability of prosecution, lots of easy targets, diminished chance of physical violence) that have made it a highly lucrative business alternative for bad guys.
BetCris was just another easy target. What the extortionists didn't count on was the unlikely confluence of Richardson's resolve, Lyon's ingenuity and an ISP that would provide them a place to fight back.
Friday, Dec. 12, 2003: BetCris Wins the War of Attrition
The extortionists must have screamed "Hooy na ny!" or some other Russian expletive after their blitzkrieg, when Lyon "got the chemistry down" and managed to absorb the massive amounts of attack traffic and get PureGig and BetCris back up and running. Lyon assumed the bad guys would come back with something bigger, as hard as that was to imagine, so he set out to scale up his system "for whatever was next, a 6Gb attack or something."
But for the next week, the attack stayed steady at around 1Gb. BetCris, Lyon and PureGig had entered a war of attrition. The extortionists would find a way to kick Lyon's system, Lyon and Lebumfacil would tweak it and get back up. Cat and mouse. "Attack, counterattack, back and forth," Lebumfacil says. "It was 24-by-7 monitoring for two weeks." Wilson and PureGig stopped noticing any of this because the attacks had been segregated from PureGig's other traffic.
And then, suddenly, the attacks stopped.
At 8:46 a.m. on Friday, Dec. 12, two weeks after the assault that nearly put him out of business and three weeks after he first read the words "Your site is under attack," Richardson received an e-mail: "Dear Mickey, I tried getting to your site today and I could not. I thought with all the money you spent you would not have these problems anymore. I guess you wasted your money instead of keeping your word. Good luck. P.S. I bet you feel real stupid that you did not keep your word. I figure by now you have lost 5 times what we asked and by the end of the year your decision will cost you more than 20 times what we asked."
Richardson knew this was an admission of defeat, even if it was disguised as braggadocio. His site was up. The extortionists couldn't get to it because they were blocked. He hadn't paid them a dime. They made no more threats. They couldn't because they couldn't back them up with action. The extortionists had lost.
And yet, the e-mail was not far off. Richardson figures it cost him a million dollars in lost revenue and IT investments to win this war. "It was worth it," he says. "I just didn't know it would take a couple years off my life."
"It was amazing we made that system work against that attack," Lyon says. "It was a wake-up call on how good the bad guys had gotten."
And Lyon knows the bad guys have gotten even better since. They've built zombie networks of 35,000 machines, capable of delivering a steady stream of 3Gb traffic. Peter Rendell, CEO of Top Layer Networks, which makes intrusion prevention and anti-DDoS hardware, says he expects botnets to pass 50,000 machines (and 4Gb to 5Gb) by the end of this year. It's an arms race, as defenses scale, then offenses scale, though Lyon is convinced the defenses have far outpaced what extortionists can throw at them.
But the bad guys have a response. Extortionists have encrypted DoS attack scripts and have put them on peer-to-peer networks, making criminals who use them nearly impossible to track or contain. They're registering domains and then attacking those domains, only those domains are redirected to other targets. "The only way to stop that is to delete the domain," Lyon says, "and that's not something you can just do." Lyon stopped an attack but certainly didn't stop the problem.
Still, he wouldn't learn of all this until later, after he decided to start a business and, as he did with Don Best, track down the BetCris extortionists. At that moment, though, after the extortionists admitted defeat, he was ready to relax. He booked a vacation in San Jose, Costa Rica, for New Year's. Finally, he'd meet the people he saved and celebrate with them.
New Year's, 2004: Visit to an Online Gaming Hotbed
Costa Rica is about the size of West Virginia, bookended by Nicaragua to the northwest and Panama to the southeast on the Central American isthmus. With coastlines on both the Pacific Ocean and Caribbean Sea, and mountainous terrain inland, Costa Rica sits along the Ring of Fire, so volcanoes and earthquakes are native. Political strife is not. The CIA calls Costa Rica a "Central American success story."
Lured by its stability, BetCris located there in 1993. Richardson joined as a "utility man" in 1996. Back then, the business wasn't online, it was a call center. BetCris's call center once employed more than 500 operators at peak hours, but the number dwindled as the business moved online. Today, maybe 30 operators will man a call center at peak hours, or during an extortion crisis.
As the Internet took off, so did San Jose as an offshore gaming mecca, for several reasons. The government encouraged the industry to expand its economy. (BetCris supports an industry group to lobby local politicians.) Also, the people are educated, with an excellent work ethic, Richardson says. Costa Rica has a 96 percent literacy rate. More high-level employees at gaming companies are Costa Ricans, including all of BetCris's accounting staff and 90 percent of its managers.
The other reason gaming companies swarmed here is, of course, because it's not the United States, where gambling laws are difficult to negotiate. Today, hundreds of offshore gaming companies, most of them online ventures, operate from San Jose. In BetCris's seven-story headquarters alone, Richardson says, there are 10 such enterprises, two software companies and a telecom company—pretty much offering everything you need to get started in the online gambling business in one building. The competition is mostly friendly. Richardson says it's not unusual to bump into competitors at a restaurant and join them for dinner.
The valley that makes up the San Jose metropolitan area holds almost half the country's 4 million people. Richardson says the valley gets blistering hot, and downtown San Jose is "undesirable." But BetCris, and most of the gaming and tourism industries, are above all that, nestled in the higher elevations of the valley's surrounding mountains, where Richardson compares the weather—and the lifestyle—favorably to San Diego.
When Lyon arrived here, he felt a sense of pride for helping. He saw "this beautiful building with this top-notch data center," he recalls. "And I met all the people who work there, and I kept thinking, I protected all of this. Me and my keyboard helped all these people keep their jobs. It was so neat to see how good a thing it was that we did."
Richardson and Lyon bonded immediately. There was a party with professional-grade fireworks launched from Richardson's front lawn. They went to dinner, talked about life and the attacks. Lyon had developed antipathy to the extortionists; he wanted to nail them. He told Richardson and Lebumfacil he was going to start a business, a service whereby people could subscribe to his anti-DDoS attack infrastructure. Lyon recruited Lebumfacil to help him start DigiDefense. BetCris was his first customer. Richardson gave them office space to start.
That business talk, though, was in the background. Lyon relaxed, went deep-sea fishing and zip-lining through the rain forest.
