sorted by: recent | see : popular

Content Tagged with infrastructure + Google

Update: Google says sorry about the GMail outage. That’s good enough for me. Here is what Todd Jackson, GMail product manager had to say on the company blog.

Many of you had trouble accessing Gmail for a couple of hours this afternoon, and we’re really sorry. The issue was caused by a temporary outage in our contacts system that was preventing Gmail from loading properly.

We’ve identified the source of this issue and fixed it. In addition, as with all issues that affect Gmail and our other services, we’re conducting a full review of what went wrong and moving quickly to update our internal systems and procedures accordingly. We don’t usually post about problems like this on our blog, but we wanted to make an exception in this case since so many people were impacted.

Original post below the fold.

It’s almost becoming routine, these outages at Google’s Gmail service. After we reported last week that there were some problems with Google Apps, today a much bigger outage hit Google’s email service, taking down the entire system, it seems.

Given that our company relies on Google’s Gmail and GTalk service, our operations came to a standstill this afternoon. We aren’t a large company but the losses are very real, especially in productivity. I wonder how the big customers of Google — folks like Sanmina-SC — are dealing with this e-blackout.

If an outage of this magnitude can strike Google, the company with a fearsome infrastructure, I wonder who — if any — can plan for the worst. I guess it’s time to stop picking on Twitter, which was fast becoming synonymous with the word “outage.” In fact, in recent weeks, not only Google’s services but those of Amazon’s S3 and Apple’s MobileMe have gone on the blink, leading us to rethink our assumptions about the reliability of the web as a platform. Clearly a lot of work still needs to be done.

How is your workday being impacted by Gmail outage? Share your stories with us.

Photo courtesy of Kyle May via Flickr.

Many entrepreneurs today have their heads in the clouds. They’re either outsourcing most of their network infrastructure to a provider such as Amazon Web Services or are building out such infrastructures to capitalize on the incredible momentum around cloud computing. I have no doubt that this is The Next Big Thing in computing, but sometimes I get a little tired of the noise. Cloud computing could become as ubiquitous as personal computing, networked campuses or other big innovations in the way we work, but it’s not there yet.

Because as important as cloud computing is for startups and random one-off projects at big companies, it still has a long way to go before it can prove its chops. So let’s turn down the noise level and add a dose of reality. Here are 10 reasons enterprises aren’t ready to trust the cloud. Startups and SMBs should pay attention to this as well.

1. It’s not secure. We live in an age in which 41 percent of companies employ someone to read their workers’ email. Certain companies and industries have to maintain strict watch on their data at all times, either because they’re regulated by laws such as HIPAA, Gramm-Leach Bliley Act or because they’re super paranoid, which means sending that data outside company firewalls isn’t going to happen.
2. It can’t be logged. Tied closely to fears of security are fears that putting certain data in the cloud makes it hard to log for compliance purposes. While there are currently some technical ways around this, and undoubtedly startups out there waiting to launch their own products that make it possible to log “conversations” between virtualized servers sitting in the cloud, it’s still early days.
3. It’s not platform agnostic. Most clouds force participants to rely on a single platform or host only one type of product. Amazon Web Services is built on the LAMP stack, Google Apps Engine locks users into proprietary formats, and Windows lovers out there have GoGrid for supporting computing offered by the ServePath guys. If you need to support multiple platforms, as most enterprises do, then you’re looking at multiple clouds. That can be a nightmare to manage.
4. Reliability is still an issue. Earlier this year Amazon’s S3 service went down, and while the entire system may not crash, Mosso experiences “rolling brownouts” of some services that can effect users. Even inside an enterprise, data centers or servers go down, but generally the communication around such outages is better and in many cases, fail-over options exist. Amazon is taking steps toward providing (pricey) information and support, but it’s far more comforting to have a company-paid IT guy on which to rely.
5. Portability isn’t seamless. As all-encompassing as it may seem, the so-called “cloud” is in fact made of up several clouds, and getting your data from one to another isn’t as easy as IT managers would like. This ties to platform issues, which can leave data in a format that few or no other cloud accepts, and also reflects the bandwidth costs associated with moving data from one cloud to another.
6. It’s not environmentally sustainable. As a recent article in The Economist pointed out, the emergence of cloud computing isn’t as ethereal as is might seem. The computers are still sucking down megawatts of power at an ever-increasing rate, and not all clouds are built to the best energy-efficiency standards. Moving data center operations to the cloud and off corporate balance sheets is kind of like chucking your garbage into a landfill rather than your yard. The problem is still there but you no longer have to look at it. A company still pay for the poor energy efficiency, but if we assume that corporations are going to try to be more accountable with regard to their environmental impact, controlling IT’s energy efficiency is important.
7. Cloud computing still has to exist on physical servers. As nebulous as cloud computing seems, the data still resides on servers around the world, and the physical location of those servers is important under many nation’s laws. For example, Canada is concerned about its public sector projects being hosted on U.S.-based servers because under the U.S. Patriot Act, it could be accessed by the U.S. government.
8. The need for speed still reigns at some firms. Putting data in the cloud means accepting the latency inherent in transmitting data across the country and the wait as corporate users ping the cloud and wait for a response. Ways around this problem exist with offline syncing, such as what Microsoft Live Mesh offers, but it’s still a roadblock to wider adoption.
9. Large companies already have an internal cloud. Many big firms have internal IT shops that act as a cloud to the multiple divisions under the corporate umbrella. Not only do these internal shops have the benefit of being within company firewalls, but they generally work hard — from a cost perspective — to stay competitive with outside cloud resources, making the case for sending computing to the cloud weak.
10. Bureaucracy will cause the transition to take longer than building replacement housing in New Orleans. Big companies are conservative, and transitions in computing can take years to implement. A good example is the challenge HP faced when trying to consolidate its data center operations. Employees were using over 6,000 applications and many resisted streamlining of any sort. Plus, internal IT managers may fight the outsourcing of their livelihoods to the cloud, using the reasons listed above.

Cloud computing will be big, both in and outside of the enterprise, but being aware of the challenges will help technology providers think of ways around the problems, and let cloud providers know what they’re up against.

Data center and hosting provider RackSpace Inc., has filed to raise up to $400 million in an initial public offering. Its financials seem generally sound (unlike many tech companies it’s actually profitable), although profits did drop by 10 percent in the last year.

However, rapid expansion (including its investment in the cloud) are to blame for the decline in profits. The company made $362 million last year and more details can be gleaned from its S-1 filing with the Securities and Exchange Commission. Some tidbits of interest include RackSpace spending $7.3 million in power used to operate 36,692 servers in 2007.

In the coming year, the company anticipates expanding its data center facilities by a least 72,000 square feet, and may also opt to find a new data center location outside of its existing facilities. It also plans to launch a platform product for customers who want hosted infrastructure but also have the need and skills to customize the hosted infrastructure to a high degree.

The San Antonio company follows in the footsteps of Google and NetSuite with its auction-style offering. Should the offering go well investors Sequoia Capital and Norwest Venture Partners stand to gain. The two firms hold 11.6 percent and 16.2 percent of RackSpace stock respectively.

Our friends at Royal Pingdom have put together a nifty little map that shows Google’s various data centers around the world. They say that…

Google has been looking at sites in Asia, such as Taiwan and Malaysia. There are also reports of a possible data center in Lithuania (Eastern Europe). Google is even more secretive about their US locations, but they have bought 466 acres of land in Blythewood, South Carolina.

Data Center Knowledge has this comprehensive FAQ on Google’s data centers.

This Friday marks the beginning of South by Southwest in Austin, starting with the Interactive Festival. Every year, geeks galore descend on my hometown, only to be replaced by filmmakers and then musicians. The geeks are my favorites, but you knew I’d say that.

As a reporter I look forward to the event, and as a resident I bemoan the lack of parking downtown, the full restaurants and the deluge of hipsters alternately making plans to move here or dissing the place for its provincialism. This year’s interactive lineup has an impressive array of companies who have built their business on the web, from Yahoo and Google to startups such as Facebook and MOG.

And 1,700 miles away in the colder climate of Philadelphia, about 1,200 network engineers will gather to perform the less-celebrated task of making sure the Internet keeps humming along. The Internet Engineering Task Force is holding one of its thrice-annual meetings to talk about the transition to IPv6, the problem of building faster routers when there’s ever more routing information to take into consideration, and a host of other issues relating to the core of the Internet.

Listening to Jari Arkko, an area director for the IETF, talk about the goals at this IETF meeting, it struck me how much the Internet has changed technology. I’m very much a hardware geek, in love with data center infrastructure, networking and chips, so I am now amazed at what a technology company can do without this level of engineering.

In the early days of the Internet, many of these technology firms had to at least figure out their data center architectures and how they would deliver and support their online shopping sites or web auction houses. But thanks to hosted services, that’s less important today. You no longer have to be a techie to start a technology company.

This is great for the billions of people using the Internet to access services and content, and speaks to the maturity of the web. However, it’s important to give credit where credit is due. So while the technology companies attending SXSW are slamming down the drinks and hobnobbing with the digerati, let’s take a moment to toast the engineers who make it all possible. And for those network engineers in Philly, it’s Beer Week up there next week, so sneak out of those plenary sessions and toss one back. Hack into my online bank account, and it’s on me.

Aquantia has raised $26 million in second-round funding to deliver 10 Gigabit Ethernet over copper, giving equipment vendors and data center operators a chance to deploy 10Gig-E without using fiber … eventually. The chips aren’t available yet, and Aquantia isn’t giving out time lines.

While cheaper than fiber, 10 Gig-E over copper consumes so much power it isn’t any more cost efficient for folks operating a data center. That has limited its appeal to core networking equipment, but the data centers are feeling the pinch of Ethernet pipes. As virtualization leads to higher server utilization, the existing Ethernet pipes represent a bottleneck. It’s kind of like fully utilizing a movie theater; when it’s time to leave, there’s a wait to get out the doors. So far folks such as Google are making do with custom-built switches.

Eric O’Brien, a general partner with Aquantia investor Lightspeed Venture Partners, says Aquantia’s chip, which has moved past first silicon but isn’t generally available, operates at a substantial power reduction compared to others attempting to play in this space. He declined to get more specific, and the company isn’t talking. If Aquantia, or its competitors such as SolarFlare Communications or Teranetics Force10 Networks, can reduce the power consumption problem, 10 Gig-E could make it out of the high-end equipment at the core of the network, and into servers. A market that large would certainly justify the more than $150 million VCs have put into this space so far.

As everyone knows, you get what you pay for. That maxim certainly holds true for Internet infrastructure, especially when it comes to servers. Over the past few years there has been an explosion of low-cost appliance servers – also known as pizza box servers — and they now account for a formidable portion of the Internet infrastructure. And though cheap in price, they are turning out to be power hogs.

“These servers are cheap to buy but consume a lot of energy and their utilization is pretty low,” said Jonathan Koomey, project scientist at Lawrence Berkeley National Laboratory and consulting professor at Stanford University, who recently conducted a study on the power requirements for servers. “The utilization is below 20 percent and we really need to focus on virtualization to get more from these boxes.”

According to his estimates, volume servers, or the low-end devices that include the pizza box servers, consumed over 50.5 billion kilowatt  hours in 2005, up from 19.7 billion kWh in 2000. That number surely must have increased by now. From 1996 through 2006, the sales of volume servers jumped from 1.41 million units to 7.282 million units, according to information collected by International Data Corp., a market research firm.

“Some of these (pizza box servers) throw up a lot of heat and are power hogs,” said Tim Sullivan, Chief Technology Officer of Internap, a data center and CDN provider, during a call with company’s management earlier today. He said that they are asking clients to better utilize the data center space and putting fewer of these pizza boxes in a more efficient manner.

One of the reasons these pizza box servers are inefficient is because they’re being made to do tasks for which they were not built. Back in the late 1990s, the form factor became popular with the large corporations, and there was plenty of space (and power) in the data centers. However, as web infrastructure needs have increased, so has the number of servers. Even tiny startups are beginning to buy 1,000 of these boxes to just stay in business.

As their numbers increase, and the problems mount, it makes me wonder if we’ll soon see the pendulum swing to the “big iron.” Thoughts, anyone?

Back in the day, when PC stocks were kings on Wall Street, a pesky college kid named Michael Dell figured out that he could do an end run around the then-established PC makers by developing a smarter way of making and selling boxes. His strategy was simple: get components and PCs from the factories in Asia to the U.S. as fast as possible, but only after he had charged for the machine.

By squeezing the supply chain as hard as he could, he turned Dell into a fearsome (and loathsome) competitor. With his help, the supply chain for the PC era came to consist of foundries, ships, U.S. assembly plants and UPS trucks. Google (GOOG), with over $200 billion in market capitalization, is following a similar strategy, fine tuning and adapting it for the Web & broadband.

Instead of trucks and assembly plants, however, Google’s supply chain is made up of fiber networks, data centers, switches, servers and storage devices. From that perspective, its business model is no different than that of Dell’s (DELL): Google has to deliver search results (information, if you want to be generous about their other projects) as fast as possible at as low a cost as possible.

To better understand Google and its business model, one needs to break it down into three data inputs.

• Relevancy of results.
• Speed of search.
• Cost of executing a search query.

While their results aren’t optimal, they are good enough. Just like Microsoft Windows was good enough to dominate the market. Google, according to Hitwise, now has 64 percent of the total search market. And although a typical Google query can often be an act of futility, we put up with it because the results are fast. If they’re wrong, we can just start all over again.

The faster the results show up on our browsers, the less inclined we’ll be to switch to a rival search engine, no matter how great the rival’s search methodology may be. The faster (and more efficient) its infrastructure, the more easily Google can keep serving the ad-based money machine.

In other words, the company has to make sure that the speed of its search is really, really fast. Any random search on Google these days takes between 0.12 to 0.06 seconds. Now that is really, really fast. Google does this by indexing the Internet quite well. The magic is in delivering the search results from this index at lightening speed, and that requires an infrastructure — oodles of bandwidth and specialized hardware — that is finely tuned, much like a Formula One Car.

Against this backdrop, it makes perfect sense for Google to build their own servers, storage systems, Internet switches and perhaps, sometime in the future, even optical transport systems. Let me rephrase that: Imagine connecting thousands of hosts (storage and server systems) at speeds of, say, 10 gigabits per second, in a manner that allows any-to-any connections.

The number of racks, fiber, routers and everything in between is mind-boggling. If this system were built using gear from established hardware makers, it would take a superhuman effort to make it all work together. In other words, the sheer cost to keep such a beast going would suck up a major component of the infrastructure.

A better option is to have gear that is customized for your processes, ones in which you have a major operational expenditure advantage. In the telecom bubble, large service providers were brought to their knees by operational expenditures.

With the exception of optical systems, Google has built or is building the gear. It has been rumored to be a big buyer of dark fiber to connect its data centers, which helps explain why the company spent nearly $3.8 billion over the past seven quarters on capital expenditures.

You can argue that building customized gear is an expensive strategy, but when you are the scale of Google, it starts to become less of an issue. Why? Because process-optimized infrastructure ensures that Google’s cost of executing a query keep going down.

To sum it up, Google’s gigantic infrastructure is the big barrier to entry for its rivals, and will remain so, as long as the company keeps spending billions on it. That said, there’s another thing Google could learn from Dell: Maintain the quality of your search results — customers will only put up with shoddiness for so long.

Note #1: Ethan, you are absolutely right about the software aspect of Google architecture, and I was going to do a separate post. This one is already 750 words.

Note #2: Earth2Tech has a post about Google’s vertically integrated green energy strategy.