The real-world implications of the rise of internet computing
EVEN when the sky is blue over Quincy, clouds hang in the air. The small town in the centre of the state of Washington is home to half a dozen huge warehouses that power the global “computing clouds” run by internet companies such as Yahoo! and Microsoft. The size of several football pitches, these data centres are filled with thousands of powerful computers and storage devices and are hooked up to the internet via fast fibre-optic links.
Yet even more intriguing than the buildings' size is their location. Quincy is literally in the middle of nowhere, three hours' drive from the nearest big city, Seattle. But it turns out to be a perfect location for data centres. As computing becomes a utility, with services that can be consumed from everywhere and on any device, ever more thought is being put into where to put the infrastructure it needs.
Where the cloud touches down is not just the business of the geeks. Data centres are essential to nearly every industry and have become as vital to the functioning of society as power stations are. Lately, centres have been springing up in unexpected places: in old missile bunkers, in former shopping malls—even in Iceland. America alone has more than 7,000 data centres, according to IDC, a market-research firm. And each is housing ever more servers, the powerful computers that crunch and dish up data. In America the number of servers is expected to grow to 15.8m by 2010—three times as many as a decade earlier.
Until a few years ago, the location of servers was an afterthought, says Jonathan Koomey, a consulting professor of environmental engineering at Stanford University. Most sat in cupboards or under desks. The computers in corporate data centres were often housed in the firm's basement. And dedicated “server farms”, which came of age during the dotcom bubble and often housed the machines of internet start-ups, were mostly built in Silicon Valley and other high-tech hubs.
The geography of the cloud
Now this haphazard landscape is becoming more centralised. Companies have been packing ever more machines into data centres, both to increase their computing capacity and to comply with new data-retention rules. As space gets tight and energy costs climb, many firms have begun consolidating and simplifying their computing infrastructure. Hewlett-Packard, the world's biggest computer-maker, for instance, is replacing its 85 data centres across the world with just six in America.
Internet firms, meanwhile, need ever larger amounts of computing power. Google is said to operate a global network of about three dozen data centres with, according to some estimates, more than 1m servers. To catch up, Microsoft is investing billions of dollars and adding up to 20,000 servers a month.
As servers become more numerous, powerful and densely packed, more energy is needed to keep the data centres at room temperature. Often just as much power is needed for cooling as for computing. The largest data centres now rival aluminium smelters in the energy they consume. Microsoft's $500m new facility near Chicago, for instance, will need three electrical substations with a total capacity of 198 megawatts. As a result, finding a site for a large data centre is now, above all, about securing a cheap and reliable source of power, says Rich Miller of Data Center Knowledge, a website that chronicles the boom in data-centre construction.
The availability of cheap power is mainly why there are so many data centres in Quincy. It is close to the Columbia River, with dams that produce plenty of cheap hydroelectric power. There is water for cooling, fast fibre-optic links, and the remoteness provides security. For similar reasons, Google chose to build a new data centre at The Dalles, a hamlet across the Columbia River in the state of Oregon.
Such sites are in short supply in America, however. And with demand for computing picking up in other parts of the world, the boom in data-centre construction is spreading to unexpected places. Microsoft is looking for a site in Siberia where its data can chill. Iceland has begun to market itself as a prime location for data centres, again for the cool climate, but also because of its abundant geothermal energy. Hitachi Data Systems and Data Islandia, a local company, are planning to build a huge data-storage facility. It will be underground, for security and to protect the natural landscape.
So will all data centres end up in remote places like Quincy or Iceland? Not necessarily. For many applications, speed is of the essence. To make sure that its web-search results show up almost instantly, Google has to distribute its data centres around the world. Financial-services firms want to have access to trading data in real time, which explains the high density of data centres near New York and London. And fast-moving online games must be hosted near their players.
Even when speed does not matter, many firms want their servers close by, says Mike Foust, the boss of Digital Realty Trust, which builds and rents out data centres. Sometimes this is for maintenance; sometimes it is because “server huggers” do not want to let go. Security also counts. The Boyd Company, which advises companies about where to put their data centres, thinks more should be built in the provinces. Demand for secure locations for back-up centres, which many firms now have to maintain, will give rise to huge regional data centres, such as the one being built in Newport in Wales.
The criteria that companies use to pick a site keep evolving, says Mike Manos, Microsoft's director of data centres. His team feeds 35 sets of data into an electronic “heat” map of the world. With colours that range from green to red, it shows where conditions are favourable and which places should be avoided. And Microsoft needs a lot of choice: if a new service suddenly becomes popular, it needs to be able to increase computing capacity quickly. That is also why it uses shipping containers pre-loaded with up to 2,000 servers, which can be up and running within hours. In the firm's Chicago data centre, over 200 such containers will populate an entire floor.
Yet it will not just be market economics that determines the shape of the clouds. Local governments give tax breaks in the hope that the presence of big data centres will attract other businesses (the computing plants themselves usually employ only a few dozen people). Regulation is a factor, too. SWIFT, a bank-transfer consortium, has announced plans to build a data centre in neutral Switzerland, so that data collected in Europe will not be stored in an American facility, where it could be subpoenaed by the United States government.
In future the geography of the cloud is likely to get even more complex. “Virtualisation” technology already allows the software running on individual servers to be moved from one data centre to another, mainly for back-up reasons. One day soon, these “virtual machines” may migrate to wherever computing power is cheapest, or energy is greenest. Then computing will have become a true utility—and it will no longer be apt to talk of computing clouds, so much as of a computing atmosphere.
http://www.economist.com/business/PrinterFriendly.cfm?story_id=11413148
Inside the Attack that Crippled Revision3
Quoted from http://revision3.com/[..]
on May 29th, 2008 at 07:49 am by Jim Louderback
As many of you know, Revision3’s servers were brought down over the Memorial Day weekend by a denial of service attack. It’s an all too common occurrence these days. But this one wasn’t your normal cybercrime – there’s a chilling twist at the end. Here’s what happened, and why we’re even more concerned today, after it’s over, than we were on Saturday when it started.
It all started with just a simple “hi”. Now “hi” can be the sweetest word in the world, breathlessly whispered into your ear by a long-lost lover, or squealed out by your bouncy toddler at the end of the day. But taken to excess – like by a cranky 3-year old–it gets downright annoying. Now imagine a room full of hyperactive toddlers, hot off of a three hour Juicy-Juice bender, incessantly shrieking “hi” over and over again, and you begin to understand what our poor servers went through this past weekend.
On the internet, computers say hi with a special type of packet, called “SYN”. A conversation between devices typically requires just one short SYN packet exchange, before moving on to larger messages containing real data. And most of the traffic cops on the internet – routers, firewalls and load balancers – are designed to mostly handle those larger messages. So a flood of SYN packets, just like a room full of hyperactive screaming toddlers, can cause all sorts of problems.
For adults, it’s typically an inability to cope, followed either by quickly fleeing the room, or orchestrating a massive Teletubbies intervention. Since they lack both legs and a ready supply of plushies, internet devices usually just shut down.
That’s what happened to us. Another device on the internet flooded one of our servers with an overdose of SYN packets, and it shut down – bringing the rest of Revision3 with it. In webspeak it’s called a Denial of Service attack – aka DoS – and it happens when one machine overwhelms another with too many packets, or messages, too quickly. The receiving machine attempts to deal with all that traffic, but in the end just gives up.
(Note the photo of our server equipment responding to the DoS Attack)In its coverage Tuesday CNet asked the question, “Now who would want to attack Revision3?” Who indeed? So we set out to find out.
Internet attacks leave lots of evidence. In this case it was pretty easy to see exactly what our shadowy attacker was so upset about. It turns out that those zillions of SYN packets were addressed to one particular port, or doorway, on one of our web servers: 20000. Interestingly enough, that’s the port we use for our Bittorrent tracking server. It seems that someone was trying to destroy our bittorrent distribution network.
Let me take a step back and describe how Revision3 uses Bittorrent, aka BT. The BT protocol is a peer to peer scheme for sharing large files like music, programs and video. By harnessing the peer power of many computers, we can easily and cheaply distribute our huge HD-quality video shows for a lot less money. To get started, the person sharing that large file first creates a small file called a “torrent”, which contains metadata, along with which server will act as the conductor, coordinating the sharing. That server is called the tracking server, or “tracker”. You can read much more about Bittorrent at Wikipedia, if you really want to understand how it works.
Revision3 runs a tracker expressly designed to coordinate the sharing and downloading of our shows. It’s a completely legitimate business practice, similar to how ESPN puts out a guide that tells viewers how to tune into its network on DirecTV, Dish, Comcast and Time Warner, or a mall might publish a map of its stores.
But someone, or some company, apparently took offense to Revision3 using Bittorrent to distribute its own slate of shows. Who could that be?
Along with where it’s bound, every internet packet has a return address. Often, particularly in cases like this, it’s forged – or spoofed. But interestingly enough, whoever was sending these SYN packets wasn’t shy. Far from it: it’s as if they wanted us to know who they were.
A bit of address translation, and we’d discovered our nemesis. But instead of some shadowy underground criminal syndicate, the packets were coming from right in our home state of California. In fact, we traced the vast majority of those packets to a public company called Artistdirect (ARTD.OB). Once we were able to get their internet provider on the line, they verified that yes, indeed, that internet address belonged to a subsidiary of Artist Direct, called MediaDefender.
Now why would MediaDefender be trying to put Revision3 out of business? Heck, we’re one of the biggest defenders of media around. So I stopped by their website and found that MediaDefender provides “anti-piracy solutions in the emerging Internet-Piracy-Prevention industry.” The company aims to “stop the spread of illegally traded copyrighted material over the internet and peer-to-peer networks.” Hmm. We use the internet and peer-to-peer networks to accelerate the spread of legally traded materials that we own. That’s sort of directly opposite to what Media Defender is supposed to be doing.
Who pays MediaDefender to disrupt peer to peer networks? I don’t know who’s ponying up today, but in the past their clients have included Sony, Universal Music, and the central industry groups for both music and movies – the RIAA and MPAA. According to an article by Ars Technica, the company uses “its array of 2,000 servers and a 9GBps dedicated connection to propagate fake files and launch denial of service attacks against distributors.” Another Ars Technica story claims that MediaDefender used a similar denial of service attack to bring down a group critical of its actions.
Hmm. Now this could have been just a huge misunderstanding. Someone could have incorrectly configured a server on Friday, and left it to flood us mercilessly with SYN packets over the long Memorial Day weekend. If so, luckily it was pointed at us, and not, say, at the intensive care unit at Northwest Hospital and Medical Center But Occam’s razor leads to an entirely different conclusion.
So I picked up the phone and tried to get in touch with ArtistDirect interim CEO Dimitri Villard. I eventually had a fascinating phone call with both Dimitri Villard and Ben Grodsky, Vice President of Operations at Media Defender.
First, they willingly admitted to abusing Revision3’s network, over a period of months, by injecting a broad array of torrents into our tracking server. They were able to do this because we configured the server to track hashes only – to improve performance and stability. That, in turn, opened up a back door which allowed their networking experts to exploit its capabilities for their own personal profit.
Second, and here’s where the chain of events come into focus, although not the motive. We’d noticed some unauthorized use of our tracking server, and took steps to de-authorize torrents pointing to non-Revision3 files. That, as it turns out, was exactly the wrong thing to do. MediaDefender’s servers, at that point, initiated a flood of SYN packets attempting to reconnect to the files stored on our server. And that torrential cascade of “Hi”s brought down our network.
Grodsky admits that his computers sent those SYN packets to Revision3, but claims that their servers were each only trying to contact us every three hours. Our own logs show upwards of 8,000 packets a second.
“Media Defender did not do anything specific, targeted at Revision3″, claims Grodsky. “We didn’t do anything to increase the traffic” – beyond what they’d normally be sending us due to the fact that Revision3 was hosting thousands of MediaDefender torrents improperly injected into our corporate server. His claim: that once we turned off MediaDefender’s back-door access to the server, “traffic piled up (to Revision3 from MediaDefender servers because) it didn’t get any acknowledgment back.”
Putting aside the company’s outrageous use of our servers for their own profit, and the large difference between one connection every three hours and 8,000 packets a second, I’m still left to wonder why they didn’t just tell us our basement window was unlocked. A quick call or email and we’d have locked it up tighter than a drum.
It’s as if McGruff the Crime Dog snuck into our basement, enlisted an army of cellar rats to eat up all of our cheese, and then burned the house down when we finally locked him out – instead of just knocking on the front door to tell us the window was open.
In the end, here’s what I know:
- A torrential flood of SYN packets rained down on Revision3’s network over Memorial Day weekend.
- Those packets – up to 8,000 a second – came primarily from computers controlled by MediaDefender, who is in the business of shutting down illegal torrent sites.
- Revision3 suffered measurable harm to its business due to that flood of packets, as the attacks on our legitimate and legal Torrent Tracking server spilled over into our entire internet infrastructure. Thus we were unable to serve videos and advertising through much of the weekend, and into Tuesday – and even our internal email servers were brought down.
- Denial of service attacks are illegal in the US under 12 different statutes, including the Economic Espionage Act and the Computer Fraud and Abuse Act.
Although I can only guess, here’s what I think really happened. Media Defender was abusing one of Revision3’s servers for their own purposes – quite without our approval. When we closed off their backdoor access, MediaDefender’s servers freaked out, and went into attack mode – much like how a petulant toddler will throw an epic tantrum if you take away an ill-gotten Oreo.
That tantrum threw upwards of 8,000 SYN packets a second at our servers. And that was enough to bring down both our public facing site, our RSS server, and even our internal corporate email – basically the entire Revision3 business. Smashing the cookie jar, as it were, so that no one else could have any Oreos either.
Was it malicious? Intentional? Negligent? Spoofed? I can’t say. But what I do know is that the FBI is looking into the matter – and it’s far more serious than toddlers squabbling over broken toys and lost cookies.
MediaDefender claims that they have taken steps to ensure this won’t happen again. “We’ve added a policy that will investigate open public trackers to see if they are associated with other companies”, promised Grodsky, “and first will make a communication that says, hey are you aware of this.”
In the end, I don’t think Media Defender deliberately targeted Revision3 specifically. However, the company has a history of using their servers to, as Ars Technica said, “launch denial of service attacks against distributors.” They saw us as a “distributor” – even though we were using Bittorrent for legitimate reasons. Once we shut them out, their vast network of servers were automatically programmed to implement a scorched earth policy, and shut us down in turn. The long Memorial Day weekend holiday made it impossible for us to contact either Media Defender or their ISP, which only exacerbated the problem.
All I want, for Revision3, is to get our weekend back – both the countless hours spent by our heroic tech staff attempting to unravel the mess, and the revenue, traffic and entertainment that we didn’t deliver.
If it can happen to Revision3, it could happen to your business too. We’re simply in the business of delivering entertainment and information – that’s not life or death stuff. But what if MediaDefender discovers a tracker inside a hospital, fire department or 911 center? If it happened to us, it could happen to them too. In my opinion, Media Defender practices risky business, and needs to overhaul how it operates. Because in this country, as far as I know, we’re still innocent until proven guilty – not drawn, quartered and executed simply because someone thinks you’re an outlaw.
- Jim Louderback
CEO - Revision3UPDATE
We’ve received several requests for some technical data to illustrate the specifics of the attack. So we’ve provided a text file with some more “under the hood” data.This file represents every packet we identified as being part of the DoS for a period of time less than .02 *seconds* on Monday morning. If you count, there’s a total of 96 packets. (We removed 12 legitimate packets from the trace). We used a combination of tcpdump and wireshark to gather this information. (this particular trace is from tcpdump)
View the text file: rev3packettrace.txt
Google Donates and Helps Collect Aid
After Cyclone Nargis struck Burma, Google put a donation link just below the search box on its site to Web users could actively participate in relief efforts.
The ruling military junta in Burma banned Google and Gmail from the region in the summer of 2006.
Google added the donation link, under it's search box. When you click this donation link it provides two donation options; one to Unicef and the other to Direct Relief International. Google will match these donations and provide up to $1 million to victims.
Google's donation page does not include any fees or charges, so 100 per cent of a donation will be transferred directly to relief efforts.
Microsoft is also helping out despite being banned in Burma.
DOT Org Domain Face 10% Increase in Registration Fee
Companies that own .org domains will soon see rates increases to renew, transfer, and register new domains. The company that operates the .org domain, Public Interest Registry (PIR), has decided to raise wholesale rates from $6.15 to $6.75 per domain this year—a 10 percent fee increase, following last year's 2.5 percent increase.
PIR informed the Internet Corporation for Assigned Names and Numbers (ICANN) of its plans in a letter (PDF) sent earlier this month. The increase will go into effect on November 9, giving those who own or want .org domains another six months to take advantage of the current price.
PIR doesn't require ICANN's approval in order to raise rates, but it is limited by ICANN on how much it can raise them per year. According to ICANN's registry agreement for .org domains, service fees cannot be increased by more than 1.1 times the previous year's max service fees. The original agreement started out in 2006 at $6 per domain, meaning that 2007's maximum service fees would have been $6.60, and 2008's maximum is $7.26.
VeriSign, has raised the prices for .com and .net the maximum allowable each year under its agreement with ICANN.
Host Color has added osTicket support
Host Color has added osTicket, an open source ticket support system to hosting plans.
Many web entrepreneurs and website owners, who have established successful online businesses often ask how to expand to the next level, improve their services and have better communication with their customers and partners. Host Color offers them osTicket, an organized solution which will help them handle customer requests and manage their priority.
osTicket is easy-to-use support ticket system that integrates inquires made via mail or a web-based form into simple multi-user web interface. The software archives customer requests and responses, so you can organize them and provide your customers with quick and accurate help. osTicket is a professional website tool that allows you trouble free migration from your current support system if you need more advanced solution.
Host Color knows that customer support is the most essential part of any successful online business. You can attract a lot of customers but if you have lax support you can't keep them long. It is crucial to have a reliable Help Desk and provide customers with prompt responses. osTicket is the right solution for that.
Review your Host
Unlike other web hosting review sites Hostjury doesn't want to give you our opinionated view of web hosting providers (often geared towards the host that pays the most for advertising) - we want to give you, the user, the ability to recommend and review your web hosting provider to other users and to share your real hosting reviews.
Think of it as a "one-stop-shop" for web hosting reviews and information.
Help us to help you and other users by reviewing your webhost! And maybe win one of those iPods!