» tagged pages
» logout

sorted by: recent | see : popular
Content Tagged with amazon + Web

Tech Stocks Take It On The Chin As Market Continues To Freefall

It’s a blood bath out there this morning. The S&P 500 is at a four-year low as the credit crisis keeps getting worse, despite the passage of the government’s $700 billion bailout plan. The market is taking tech stocks down with it. Google is down 4 percent to $368, its lowest point since 2006. Apple is down 6 percent to $91. Microsoft is down nearly 5 percent to $25. Amazon, Yahoo, eBay—all down.

Fears of a credit freeze are growing as the contagion spread to banks in Europe. The Fed is already flooding the market with more cash through new powers it was granted in the bailout package. All of this makes you wonder if A) the U.S. government acted fast enough and B) whether the bailout package is going to end up doing any good.

As far as tech stocks are concerned, already as I write this, there seems to be somewhat of a rally going on in some of these stocks (particularly Google) from the lows where they opened. But if the economy falters, tech stocks won’t be a safe haven for investors, even if they are cash-rich and not as exposed to the credit debacle as companies in other sectors. The markets always tend to overreact to systemic risk because nobody knows how far the problems are going to spread. What we are seeing is panic in the face of the unknown. It reminds me of the market panic after 9/11. Investors whop loaded up on tech stocks then ended up making a lot of money.

Does this signal a buying opportunity, or are investors better off running for the hills? Who is buying (or selling) what out there? Tell us in comments.

Crunch Network: CrunchBase the free database of technology companies, people, and investors

Web2.0: TechCrunch

Expanding the Cloud: Microsoft Windows Server on Amazon EC2

cloud-reflection-office-small.jpg

The backend servers that power the world of Internet Services have become increasingly diverse. With today's announcement that Microsoft Windows Server is available on Amazon EC2 we can now run the majority of popular software systems in the cloud. Windows Server ranked very high on the list of requests by customers so we are happy that we will be able to provide this.

One particular area that customers have been asking for Amazon EC2 with Windows Server was for Windows Media transcoding and streaming. There is a range of excellent codecs available for Windows Media and there is a large amount of legacy content in those formats. In past weeks I met with a number of folks from the entertainment industry and often their first question was: when can we run on windows?

There are many different reasons why customers have requested Windows Server; for example many customers want to run ASP.NET websites using Internet Information Server and use Microsoft SQL Server as their database. Amazon EC2 running Windows Server enables this scenario for building scalable websites. In addition, several customers would like to maintain a global single Windows-based desktop environment using Microsoft Remote Desktop, and Amazon EC2 is a scalable and dependable platform on which to do so.

Amazon EC2 with Windows Server is still currently in private beta testing, but will be available for general use before the end of the year. Keep an eye on the AWS Weblog for information about Amazon Web Services at the Microsoft Professional Developer Conference.

User:dolander: All Things Distributed

AWS Startup Challenge 2008

The last week for submitting the applications for the AWS Startup Challenge has started. Looking at the proposals that are being submitted it looks like this will be another very inspiring challenge. These proposals are reviewed by a panel and five finalists will be selected. The finalists will come to Seattle to compete for $50K in cash, $50K in AWS credits, 2 years of Premium Support and more. All finalists will receive Rightscale Premium for 6 months and there will be a number of promotional events that includes all the finalists.

Last year there were 900 applications which made for very intense proposal reading sessions. Eventual Ooyala won the challenge and got to smash the server to bits. The videos of last year's finalists are still online.

If you have a brilliant idea/business that we should be evaluating you have until October 10 to let us know.

User:dolander: All Things Distributed

Cheap, Easy Audio Transcription with Mechanical Turk - Waxy.org

The result: my 36-minute recording was transcribed while I slept, in less than three hours, for a grand total of $15.40.

podcasting: del.icio.us tag/podcasting

Amazon Launches Content Delivery Network. Rivals, Watch Out!

It was a matter of when, not if, that Amazon would launch a content delivery business in addition to its current suite of web services that include S3 storage service and EC2 on-demand computing. The Seattle-based company has announced its intention to offer a content delivery service that could shake the very nature of the industry and pose a serious challenge to not only dozens of CDN upstarts but also become a thorn in the side of existing giants such as Akamai Technologies and Limelight Networks.

In an email to its customers today, Amazon said that the service will be available later this year and will utilize the company’s points of presence in North America, Europe and Asia.

This new service will provide you a high-performance method of distributing content to end users, giving your customers low latency and high data transfer rates when they access your objects. The initial release will help developers and businesses who need to deliver popular, publicly readable content over HTTP connections.

Ironically, Amazon was beaten to the CDN punch by New York-based Voxel, which started offering CDN services based on Amazon’s S3 service. ”We are announcing this right now because we want to give a heads up to our customers,” said Adam Selipsky, VPr of product management and developer relations for AWS. It’s more like putting their competition on notice, but Adam was too polite to say that. “It is a more horizontal and broad offering.” In other words, while it is not going to replace Akamai tomorrow, it is going to make CDNs affordable even for the tiniest startup, without major cash outlays.

Why is this service disruptive? Amazon is going to bring a level of transparency to a business that has a sales model much like a brokerage firm in the 1980s. Amazon wants to make buying CDN services as simple as buying a book. Amazon executives told me that company is going to be charging its customers on usage instead of the long-term contracts current players foist on their clients.

In addition, the company will publish its prices on the web — most importantly it is going to be inexpensive. And that will make the service even more attractive to hundreds of small companies who are already using Amazon Web Services for their web operations, who don’t want to sign long contracts with CDN operators. When I asked Tal Saraf, general manager of the AWS Content Delivery Service, if the company expected the video-delivery to be one of the most used service, he said the company expected to delivery all sorts of content, including web objects (images, JavaScripts etc.)

You’ll start by storing the original version of your objects in Amazon S3, making sure they are publicly readable. Then, you’ll make a simple API call to register your bucket with the new content delivery service. This API call will return a new domain name for you to include in your web pages or application. When clients request an object using this domain name, they will be automatically routed to the nearest edge location for high performance delivery of your content. It’s that simple.

Amazon executives declined to talk about the pricing. “We will talk about the pricing when we launch the service,” Selipsky said. He declined to comment on the impact the pricing will have on their competitors -– nearly two dozen content delivery networks –- and how much their business is going to suffer. Dow Jones Venture Source estimates that from 2005 through the second quarter of 2008, almost $980 million was invested in content delivery companies.

If Amazon delivers what it is promising -– a simple, API-based CDN – then it would put then not only ahead of all CDN players, but also force rivals to meet the rules (and pricing) set by Amazon. There is a good chance that it’s going to drive weaker players right out of the game.

My final take on this news: Akamai is less likely to be impacted in the near term, but it further commoditizes the CDN business and forces a big shakeout in the industry, taking down the small and the weak. Akamai has been focusing on value-add services, as a way to stay ahead of the commoditization of the basic CDN services.

Read Amazon CTO Werner Vogels take on the news on his blog.


900 million PCs or 300 billion mobile handsets. Which is the bigger opportunity?
Mobilize 08: GigaOM’s Next-Generation Mobile Conference

Technology-News: GigaOm

Amazon Gets In On the Content Delivery Business

Amazon Logo

It was only a matter of time before Amazon would decide to release a Content Delivery Network, and according to the company in an official statement released today, it has done just that.

Designed to complement its S3 storage service and EC2 web services, the CDN will be available later this year and will provide users with a high performance method of distributing content to end users. Amazon claims it will have low latency and high data transfer rates when users access the content and it will be specifically designed (in the beginning at least) for “developers and businesses who need to deliver popular, publicly readable content over HTTP connections.”

Although the CDN space is crowded with similar services from Akamai Technologies and Limelight Networks, Amazon thinks it knows how to be successful in the space. And one of the key components of its plan is to undercut others on price and make it much easier to buy CDN services.

According to Amazon, it will charge customers based on usage instead of the common practice of charging through long-term contracts, but it would not discuss pricing at this time.

Amazon getting into the CDN business seems like the ideal move for a company that’s trying to provide storage and on-demand computing services already. And considering its size makes it easier for it to adapt its business model to satisfy smaller businesses and those that are less likely to want to enter into long-term agreements, Amazon could quite easily push its competitors aside and cement itself as the leader in the market.

And with a video streaming and distribution service already in place, Amazon is quickly becoming a CDN for itself, so it may know a thing or two about providing a robust service to its customers when its CDN becomes available later this year.

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

Web2.0: TechCrunch

Expanding the Cloud

For many the "Cloud" in Cloud Computing signifies the notion of location independence; that somewhere in the internet services are provided and that to access them you do not need any specific knowledge of where they are located. Many applications have already been built using cloud services and they indeed achieve this location transparency; their customers do not have to worry about where and how the application is being served.

However for developers to do their job properly the cloud cannot be fully transparent. As much as we would like to make it easy and simple for everyone, building high-performance and highly reliable applications in the cloud requires that the developers have more control. For example a reality is that failures can happen; servers can crash and networks can become disconnected. Even if these are only temporary glitches and are transient errors, the developer of applications in the cloud really wants to make sure his or her application can continue to serve customers even in the face of these rare glitches. A similar issue is that of network latency; as much as we would like to see the cloud to be transparent, the transport of network packets is still limited to the speed of light (at best) and customers of cloud applications may experience a different performance depending on where they are located in relation to where the applications are running. We have seen that for many applications that works just fine, but there are developers who would like more control over how their customers are being served and for example would like to give all their customers low latency access, regardless of their location.

At Amazon we have been building applications on these cloud principles for several years now and we are very much aware of the tools that developers need to build applications that are required to meet very high standards with respect to scalability, reliability, performance and cost-effectiveness. We are also listening very closely to the feedback AWS customers are giving us to make sure we expose the right tools for them to do their job. We launched Amazon S3 in Europe to ensure that developers could build applications that could serve data out of a European storage cloud. We launched Regions and Availability Zones (combined with Elastic IPs) for Amazon EC2 such that developers would have better control over where their applications would be running to ensure high-availability. We are now ready to expand the cloud even further and bring the cloud storage to its customers' doorstep.

Today we are announcing that we are expanding the cloud by adding a new service that will give developers and businesses the ability to serve data to their customers world-wide, using low-latency and high data transfer rates. Using a global network of edge locations this new service can deliver popular data stored in Amazon S3 to customers around the globe through local access.

We have developed this content delivery service using the robust AWS principles we know work well for our customers:

  • Cost-effective: no commitments and no minimum usage requirements. You only pay for what you use in a manner similar to the other Amazon Web Services.
  • Simple to use: one API call gets you going. You store the data you want to distribute in an Amazon S3 bucket and you use this API call to register this bucket with the content distribution service. The registration will provide you with a new domain name that you can use in url's to access the data through this service with HTTP. When your customer accesses your content through your new url the data it refers to will be delivered through a network of edge servers.
  • Works well with other services: The service integrates seamlessly with Amazon S3 and the data/content served through the service can be accessed using the standard HTTP access techniques.
  • Reliable: Amazon S3 will give you durable storage of your data, and the network of edge locations on three continents used by the new service will deliver your content to your customers from the most appropriate location.

This is an important first step in expanding the cloud to give developers even more control over how their applications and their data are served by the cloud. The service is currently in private beta but we expect to have the service widely available before the end of the year. You can get a few more details and sign up to get notified when the service is becoming on this AWS page Also check Jeff Bar's posting on the AWS weblog.

User:dolander: All Things Distributed

Amazon Is Getting Into the Alcohol Business

Amazon Wine Sales

Amazon has again announced that it will start selling US-produced wine to its American customers by the beginning of October.

But before you start getting excited about buying wine online, there’s one catch: it will only be available to people in about 26 states due to interstate regulations, but the company is working with New Vine Logistics, a firm that specializes in interstate transactions, to avoid any legal issues that may arise in the process and attempt to increase the distribution of wine as time goes on.

Amazon is jumping in on the wine craze at the right time. Annual wine consumption is on the rise and the Stonebridge Research Group said American wine sales last year were between $30 billion and $32 billion. But e-commerce sales accounted for just 7 percent of those sales and few services online can connect wine lovers with US-produced wine.

Age restrictions will obviously play a major role in Amazon’s plan to sell wine and there’s currently no indication of how it will stop the sale of the alcohol to minors. Regardless, Amazon thinks it’s on to something with selling wine on its site and wants to capitalize on the growing market.

Amazon invested $30 million into Wineshopper.com in 1999, but that site lasted one year. Now we’ll see if this venture lasts any longer.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

Web2.0: TechCrunch

Hulu vs. Amazon: Amazon Never Stood A Chance

Amazon VOD Front Page

Since Amazon released its video on demand service earlier today, I’ve pored over each and every section of the site to determine if it has what it takes to supplant Hulu as the best online video service providing professional network content. And after doing just that, I’ve quickly realized that it doesn’t.

Amazon has done all it can to solidify its stance in the online video market. First, it launched its Unbox service to compete with film streaming and now it has tried to compete in on-demand streaming of TV shows and movies. And by making TV shows and movies available online to be streamed directly to your computer, it’s quickly becoming apparent that Amazon is not necessarily focusing all its attention on iTunes, but on what it perceives to be the next frontier in video: online streaming.

Selection

Amazon is quick to point out that the videos it offers on its new service don’t have ads breaking into the flow of the shows. But that is because you have to pay $1.99 for each TV episode.  Granted, Amazon’s VOD service does have some free shows — I watched 30 Rock a few times — but the list of free shows is laughable, compared to the sheer number of paid shows.

So is it even fair to compare a paid service to a free one?  Sure it is.  Right now, for instance, Hulu is offering five episodes of The Office for free on its site with just a handful of commercials in each. I tried playing those same shows on Amazon to see if I could stream them for free on its service and was presented with a banner telling me I needed to buy the show after just 2 minutes of viewing.

If you’re fine with paying for TV shows, you’ll be happy to know that Amazon’s offering has quite a few — it claims it has over 40,000 movies and TV shows available. In fact, practically every episode from each season of every show in Amazon’s VOD service is available for purchase.  I was pleased to see that I could choose from a slew of networks like TV Land and NBC.  So Amazon lives up to its name in terms of breadth of selection.  But there was something frustrating about the fact that so few shows had a free stream available, given all the free TV to be found on the Web.

Experience

Overall, the Amazon VOD experience is quite similar to Hulu: you pick a network from the links on the left column, choose your show, and find your episode. The picture quality of the videos I watched were highly-detailed and I was pleasantly surprised by Amazon’s decision to offer such a large display instead of the somewhat smaller screens found on Hulu.

That said, the actual video playback was suspect at times and the video streams would skip too often. That may be the result of all the attention Amazon’s VOD is receiving today, but so far, I haven’t seen any improvements. I also didn’t like that the video playback screens were too cluttered with extras. The pages feature all the episodes below the video.  And Amazon’s ubiquitous recommendation engine picks underneath only muddies an otherwise clean screen.

Are no commercials worth it?

The main selling points of Amazon’s Video On Demand service are that it doesn’t have commercials, it has a slew of shows available and the picture quality is superb.But the streaming is choppy, the site layout is suspect, and some of the same content is available elsewhere for free.

Even though Hulu commercials can annoy me when I’m excited to see what happens next on The Office, I’ve never realized until now just how much I prefer them to shelling outcash for what is often a hit-or-miss experience.  I don’t mind paying for a show I enjoy, but $1.99 seems a bit steep for one episode and if it’s being played for free on Hulu, I’m not sure why I would even consider using Amazon’s service.

Amazon Free Video

Amazon Free Videos

Amazon Paid Videos

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

Web2.0: TechCrunch

Amazon Associates Perl Script

This script was created to utilize Amazon.com's Web Services to provide a real-time listing of books or other items on your site that provide links for your visitors to purchase these items from Amazon US, UK, DE, JP, FR or CA, using your affiliate code. This script parses Amazon.com's XML feed to provide search results and bestseller listings in HTML for your visitors.

XML: del.icio.us/tag/xml

Amazon EBS - Elastic Block Store has launched

Today marks the launch of Amazon EBS (Elastic Block Store), the long awaited persistent storage service for EC2. Details can be found on the EC2 detail page, the press release and Jeff Barr's posting over on the AWS evangelists blog. Also the folks at Rightscale have two detailed postings: why Amazon EBS matters and Amazon EBS explained.

With the launch of the Elastic Block Store we complete an important milestone in offering a complete suite of storage solutions as part of the Amazon Infrastructure Services. Back in the days when we made the architectural decision to virtualize the internal Amazon infrastructure one of the first steps we took was a deep analysis of the way that storage was used by the internal Amazon services. We had to make sure that the infrastructure storage solutions we were going to develop would be highly effective for developers by addressing the most common patterns first. That analysis led us to three top patterns:

  1. Key-Value storage. The majority of the Amazon storage patterns were based on primary key access leading to single value or object. This pattern led to the development of Amazon S3.
  2. Simple Structured Data storage. A second large category of storage patterns were satisfied by access to simple query interface into structured datasets. Fast indexing allows high-speed lookups over large dataset. This pattern led to the development of Amazon SimpleDB. A common pattern we see is that secondary keys to objects stored in Amazon S3 are stored in SimpleDB, where lookups result in sets of S3 (primary) keys.
  3. Block storage. The remaining bucket holds a variety of storage patterns ranging special file systems such as ZFS to applications managing their own block storage (e.g. cache servers) to relational databases. This category is served by Amazon EBS which provides the fundamental building block for implementing a variety of storage patterns.

I have written before about the basic features of Amazon EBS:

  • Amazon EBS will be offered in the form of storage volumes which you can mount into your EC2 instance as a raw block storage device. It basically looks like an unformatted hard disk. Once you have the volume mounted for the first time you can format it with any file system you want or if you have advanced applications such as high-end database engines, you could use it directly.
  • Developers can create multiple volumes, in size ranging from 1 GB to 1TB. This volume will be created within a specified Availability Zone and will be accessible by your EC2 instances running in that Availability Zone. As to be expected with a volume abstraction only one instance can have the volume mounted at any given time. Volumes can migrate and be reattached to other instances if necessary for failure handling or application migration reasons.
  • The consistency of data written to this device is similar to that of other local and network-attached devices; it is under control of the developer when and how to force flush data to disk if you want to bypass the traditional lazy-writer functionality in the operating systems file-cache. Because of the session oriented model for access to the volume you do not need to worry about eventual consistency issues.

However Amazon EBS isn't just a massive volume storage array within an Availability Zone, it provides a unique feature that allows for the creation of novel storage management scenarios: the ability to create snapshots and store those snapshots into Amazon S3. These snapshots can then be used as the starting point for creating new volumes within any availability zone.

We see developers use this feature for long term backup purposes, for use in rollback strategies, for (world-wide) volume re-creation purposes. Snapshots also play an important role in building fault-tolerance scenarios when combined with managing applications using Elastic IP addresses and Availability Zones.

Congratulations to the EBS team for delivering a great service that will help a lot of EC2 customers managing their storage efficiently.

User:dolander: All Things Distributed

Root Cause

For those of you interested in the details of last Sunday's Amazon S3 Availability issue you should read the detailed explanation posted at the AWS Status Dashboard. Root cause was single bit corruption of internal state messages that are distributed via Gossip techniques.

User:dolander: All Things Distributed

10 Reasons Enterprises Aren’t Ready to Trust the Cloud

Many entrepreneurs today have their heads in the clouds. They’re either outsourcing most of their network infrastructure to a provider such as Amazon Web Services or are building out such infrastructures to capitalize on the incredible momentum around cloud computing. I have no doubt that this is The Next Big Thing in computing, but sometimes I get a little tired of the noise. Cloud computing could become as ubiquitous as personal computing, networked campuses or other big innovations in the way we work, but it’s not there yet.

Because as important as cloud computing is for startups and random one-off projects at big companies, it still has a long way to go before it can prove its chops. So let’s turn down the noise level and add a dose of reality. Here are 10 reasons enterprises aren’t ready to trust the cloud. Startups and SMBs should pay attention to this as well.

  1. It’s not secure. We live in an age in which 41 percent of companies employ someone to read their workers’ email. Certain companies and industries have to maintain strict watch on their data at all times, either because they’re regulated by laws such as HIPAA, Gramm-Leach Bliley Act or because they’re super paranoid, which means sending that data outside company firewalls isn’t going to happen.
  2. It can’t be logged. Tied closely to fears of security are fears that putting certain data in the cloud makes it hard to log for compliance purposes. While there are currently some technical ways around this, and undoubtedly startups out there waiting to launch their own products that make it possible to log “conversations” between virtualized servers sitting in the cloud, it’s still early days.
  3. It’s not platform agnostic. Most clouds force participants to rely on a single platform or host only one type of product. Amazon Web Services is built on the LAMP stack, Google Apps Engine locks users into proprietary formats, and Windows lovers out there have GoGrid for supporting computing offered by the ServePath guys. If you need to support multiple platforms, as most enterprises do, then you’re looking at multiple clouds. That can be a nightmare to manage.
  4. Reliability is still an issue. Earlier this year Amazon’s S3 service went down, and while the entire system may not crash, Mosso experiences “rolling brownouts” of some services that can effect users. Even inside an enterprise, data centers or servers go down, but generally the communication around such outages is better and in many cases, fail-over options exist. Amazon is taking steps toward providing (pricey) information and support, but it’s far more comforting to have a company-paid IT guy on which to rely.
  5. Portability isn’t seamless. As all-encompassing as it may seem, the so-called “cloud” is in fact made of up several clouds, and getting your data from one to another isn’t as easy as IT managers would like. This ties to platform issues, which can leave data in a format that few or no other cloud accepts, and also reflects the bandwidth costs associated with moving data from one cloud to another.
  6. It’s not environmentally sustainable. As a recent article in The Economist pointed out, the emergence of cloud computing isn’t as ethereal as is might seem. The computers are still sucking down megawatts of power at an ever-increasing rate, and not all clouds are built to the best energy-efficiency standards. Moving data center operations to the cloud and off corporate balance sheets is kind of like chucking your garbage into a landfill rather than your yard. The problem is still there but you no longer have to look at it. A company still pay for the poor energy efficiency, but if we assume that corporations are going to try to be more accountable with regard to their environmental impact, controlling IT’s energy efficiency is important.
  7. Cloud computing still has to exist on physical servers. As nebulous as cloud computing seems, the data still resides on servers around the world, and the physical location of those servers is important under many nation’s laws. For example, Canada is concerned about its public sector projects being hosted on U.S.-based servers because under the U.S. Patriot Act, it could be accessed by the U.S. government.
  8. The need for speed still reigns at some firms. Putting data in the cloud means accepting the latency inherent in transmitting data across the country and the wait as corporate users ping the cloud and wait for a response. Ways around this problem exist with offline syncing, such as what Microsoft Live Mesh offers, but it’s still a roadblock to wider adoption.
  9. Large companies already have an internal cloud. Many big firms have internal IT shops that act as a cloud to the multiple divisions under the corporate umbrella. Not only do these internal shops have the benefit of being within company firewalls, but they generally work hard — from a cost perspective — to stay competitive with outside cloud resources, making the case for sending computing to the cloud weak.
  10. Bureaucracy will cause the transition to take longer than building replacement housing in New Orleans. Big companies are conservative, and transitions in computing can take years to implement. A good example is the challenge HP faced when trying to consolidate its data center operations. Employees were using over 6,000 applications and many resisted streamlining of any sort. Plus, internal IT managers may fight the outsourcing of their livelihoods to the cloud, using the reasons listed above.

Cloud computing will be big, both in and outside of the enterprise, but being aware of the challenges will help technology providers think of ways around the problems, and let cloud providers know what they’re up against.

Technology-News: GigaOm

Of Parascale & Other Cloud Computing News

Parascale, a Cupertino, Calif-based start-up that has developed a storage file system for a cloud of computers announced that it had attracted $11.37 million in Series A funding from Charles River Ventures and Menlo Ventures. The company recently changed its chief executive and brought in Sajai Krishnan, a former NetApp executive to run the company.

I spoke to Sajai briefly this morning, though I have not had a chance to dig deeply into the company’s technology just yet. The company is going to officially release its software, currently in trials in Fall 2008. It is targeting the streaming media/video industry and others who want to get storage for less than 50 cents a gigabyte. Others like ISPs can use it to set up their own grids and offer competitors to AWS.

What essentially they have developed is software that gets commodity storage drives attached to plain vanilla low cost servers to behave like a giant cloud of storage space, which can be used (and managed) using protocols such as HTTP, FTP and NFS. The company describes its approach as virtual storage grid.

The general idea is not novel, though the company’s commercial rivals (such as Isilon & polyServe (part of HP)) can handles limited number of nodes, an industry euphemism for storage-attached servers. Parascale claims it can handle hundreds of nodes making it easier for the company to handle terabytes of data.

How it works is that a control server – lets call it the brain of the storage cloud – communicates over Gigabit Ethernet connections with storage nodes and makes them all behave like one giant storage cloud. Similar systems from more traditional storage companies would use custom high-speed connection technology like Fiber Channel.
Storage nodes are x86-based Linux servers that support cheap SATA drives. The brain essentially stores the metadata of the files on storage nodes and at all times knows where data has been placed, file versions, and other such information. A software management console helps manage the flow of data in-and-out of the system.

This approach to build high volume storage systems has received a lot of attention, thanks to the success of Google File System and of late, the open source Hadoop platform, championed by Yahoo and Apache Foundation. Our sister blog, OStatic had noted that Hadoop was putting companies like EMC at risk.

Add newcomers likes Parascale to that list, for we have seen many a few open source projects upend proprietary efforts. I am quite bullish on the prospects of open source cloud projects. Still, the company has garnered some positive reviews from early users of its trial software.

In Other Cloud Computing News:

  • The Much talked about start-up, SmugMug launched an online back-up service, SmugVault, based on Amazon Web Services’ platform. It costs 22 cents per gigabyte per month, but to upload it will cost 30 cents per gigabyte, while downloads cost 51 cents a gigabyte. Thomas Hawk points out that at those prices, a two terabyte archive would cost $440 a month, not to mention a $600 one time fee and extra to get the stuff off the drive. Now that’s expensive. I am sticking to Bingo from Joyent.
  • ADC, a telecom equipment maker has teamed up with APC, a power management company to develop fiber optic switches and storage area network devices for the data centers. They want to tap into the recent trend to use fiber in the data centers. ADC recently showed off a ton of new gear for the data center market.
  • If this story interests you then you should definitely check out our upcoming conference, Structure 08.

    Technology-News: GigaOm

    A Window on the Cloud

    Outsourcing compute power is wonderful — until something goes wrong. Unfortunately, when an Amazon Web Service goes down it’s hard to know why, and it’s even harder to know how well a particular cloud is performing in the first place. To make the cloud more transparent, open source cloud management software vendor Hyperic has launched www.CloudStatus.com, a web site that lets a user peek in on the various compute clouds to see how things are running.

    CloudStatus measures service availability, latency and throughput for cloud-based infrastructure and application services. The initial release provides metrics for Amazon’s Elastic Compute Cloud, Simple Storage Service, SimpleDB, Simple Queue Service and Flexible Payment Service.

    Hyperic sends a software agent to make requests against various cloud services, and according to CEO Javier Soltero, it racks up quite a large bill doing do. The web site views are free, but Soltero says Hyperic also plans to launch a line of services for paying customers. It’s a decent idea, but my worry is that Amazon or another cloud provider could shut the service down, either by offering their own status service or by stopping the Hyperic agent. Given the rush to provide dashboards, application-testing products and other services on top of established computing services, I’m eager to see how startups keep their footing in the clouds.

    If this story interests you then you should definitely check out our upcoming conference, Structure 08.

    Technology-News: GigaOm

    Setting up on EC2

    Thanks to my friend Dustin, and his EC2 demo using Elasticfox Firefox Extension for Amazon EC2 I got an EC2 image setup. With other references Link 1,Link 2,Link 3 I was also able to create my own AMI.

    Some notes specific for my configuration.

    Pre-config ElasticFox key for launching directly from ElasticFox SSH connections.

    mkdir ~/ec2-keys
    mv ~/Downloads/elasticfox.pem ~/ec2-keys/id_elasticfox
    chmod 600 ~/ec2-keys/id_elasticfox
    chmod 700 ~/ec2-keys/
    ssh -i /Users/rbradfor/ec2-keys/id_elasticfox root@ec2-99-999-999-999.compute-1.amazonaws.com
    

    Installed Software.

    apt-get update
    apt-get -y autoremove
    apt-get -y install apache2
    apt-get -y install mysql-server
    # Prompts for password (very annoying)
    apt-get -y install php5
    apache2ctl graceful
    echo "Hello World" > /var/www/index.html
    echo "” > /var/www/phpinfo.php
    

    Configuration to save AMI.

    scp -i ~/ec2-keys/id_elasticfox ~/ec2-keys/id_elasticfox pk-CHK7DP4475BWUKIUF4WFDIW3VMYDYOHQ.pem cert-CHK7DP4475BWUKIUF4WFDIW3VMYDYOHQ.pem root@ec2-99-999-999-999.compute-1.amazonaws.com:/mnt
    
    ec2-bundle-vol -d /mnt -c cert-CHK7DP4475BWUKIUF4WFDIW3VMYDYOHQ.pem -k pk-CHK7DP4475BWUKIUF4WFDIW3VMYDYOHQ.pem -u AccountNumber -r i386 -p ubuntu804_lamp
    ec2-upload-bundle -b rbradford_804_lamp_ami -m /mnt/ubuntu804_lamp.manifest.xml -a AccessID -s SecretKey
    

    MySQL: Planet MySQL

    Why Cloud Computing Needs Security

    Bribery, extortion and other con games have found new life online. Today, botnets threaten to take vendors down; scammers seduce the unsuspecting on dating sites; and new viruses encrypt your hard drive’s contents, then demand money in return for the keys.

    Startups, unable to bear the brunt of criminal activity, might look to the clouds for salvation: After all, big cloud computing providers have the capacity and infrastructure to survive an attack. But the clouds need to step it up; otherwise, their single points of failure simply provide more appealing targets for the bad guys, letting them take out hundreds of sites at once.

    Last Friday, Amazon’s U.S. site went off the air, and later some of its other properties were unavailable. Lots of folks who wouldn’t let me quote them, but should know, said that this was a denial-of-service attack aimed at the company’s load-balancing infrastructure. Amazon is designed to weather huge amounts of traffic, but it was no match for the onslaught.

    When it comes to online crime, the hackers have the advantage. A simple Flash vulnerability nets them thousands of additional zombies, meaning attacks can come from anywhere. During Amazon’s attack, legitimate visitors were greeted with a message saying they were abusing Amazon’s terms of service, which could mean that those visitors were either using PCs that were part of the attack, or were on the same networks as infected attackers. The botnets are widespread, and you can’t block them without blocking your customers as well.

    Other rackets give the attacker an unfair edge, too: It takes an army of machines to crack the 1024-bit encryption on a ransom virus, but only one developer to write it.

    A brand like Amazon can weather a storm, because people will return once the storm has passed. But just look at the Twitter exodus to see how downtime from high traffic loads can tarnish a fledgling brand. Slideshare survived such an attack in April, and while many other sites admit to being threatened, they won’t go on the record as saying so.

    Up-and-coming web sites are often great targets, as they often lack the firewalls, load-balancers and other infrastructure needed to fight back. And it’s not just criminals: In some cases, the attacker is a competitor; in others, it’s someone who just doesn’t like what you’re doing.

    Fighting off hackers is expensive. Auren Hoffman calls this the Black Hat Tax, and points out that many top-tier Internet companies spend a quarter of their resources on security. No brick-and-mortar company devotes this much attention to battling fraud.

    Wanting to survive an attack is yet another reason for startups to deploy atop cloud computing offerings from the likes of Amazon, Google, Joyent, XCalibre, Bungee, Enki and Heroku. But consolidation of the entire Internet onto only a few clouds may be its Achilles’ heel: Take down the cloud, and you take down all its sites. That’s one reason carriers like AT&T and CDNs like Akamai are betting that a distributed cloud will win out in the end.

    Cloud operators need to find economies of scale in their security models that rival the efficiencies of hackers. Call it building a moat for the villagers to protect them from the barbarians at the gate. Otherwise, this will remain a one-sided battle that just gives hackers more appealing targets.

    If this story interests you then you should definitely check out our upcoming conference, Structure 08.

    Technology-News: GigaOm

    Why Amazon Went Down, and Why It Matters

    Amazon.com’s U.S. retail site became unavailable around 10:25 AM PST, and now appears to be back up. Amazon’s not naming names — all that director of strategic communications Craig Berman would say was that: “Amazon’s systems are very complex and on rare occasions, despite our best efforts, they may experience problems.”

    Berman did confirm, however, that neither Amazon Web Services nor international sites were affected.

    So what happened? Let’s look at the facts.

    • Traffic to https://www.amazon.com was getting there. So DNS was configured properly to send traffic to Amazon’s data centers. Global server load balancing (GSLB) is the first line of defense when a data center goes off the air. Either GSLB didn’t detect that the main data center was down, or there was no spare to which it could send visitors.
    • When traffic hit the data center, the load balancer wasn’t redirecting it. This is the second line of defense, designed to catch visitors who weren’t sent elsewhere by GSLB.
    • If some of the servers died, the load balancer should have taken them out of rotation. Either it didn’t detect the error, or all the servers were out. This is the third line of defense.
    • Most companies have an “apology page” that the load balancer serves when all servers are down. This is the fourth line of defense, and it didn’t work either.
    • The HTTP 1.1 message users saw shows something that “speaks” HTTP was on the other end. So this probably wasn’t a router or firewall.

    This sort of thing is usually caused by a misconfigured HTTP service on the load balancer. But that would happen late at night, be detected, and rolled back. It could also happen from a content delivery network (CDN) not retrieving the home page properly.

    So my money’s on an AFE or CDN problem. But as Berman notes, Amazon’s store is a complex application and much of their infrastructure doesn’t follow “normal” data center design. So only time (and hopefully Amazon) will tell.

    Site operators can learn from this: Look into GSLB, and make sure you have geographically distributed data centers (possibly through AWS Availability Zones.) It’s another sign we can’t take operations for granted, even in the cloud.

    Technology-News: GigaOm

    GigaOM Interview: Amazon CEO Jeff Bezos

    Amazon.com founder and CEO Jeff Bezos was one of the attendees at this week’s D6 Conference in Carlsbad, Calif., to be interviewed on stage, where he talked about Kindle at length. But right after his chat with Walt Mossberg and Kara Swisher, I caught up with him to discuss Amazon Web Services and his company’s efforts in cloud computing. Here is a short excerpt from that conversation, captured on my Sanyo Xacti. In particular, he talks about…

    • How and when Amazon began its cloud computing effort.
    • Why Amazon has become an innovator with Amazon Web Services and how it relates to their core business of being an online retailer.
    • Whether or not Wall Street recognizes Amazon’s cloud efforts.
    • What’s next for Amazon Web Services.
    • Whether or not Amazon has plans for a VC fund or for cloud computing startups.

    For even more info about Amazon’s cloud computing efforts, join us at our upcoming conference, Structure ‘08, where CTO Werner Vogels will be delivering a keynote address.

    If this story interests you then you should definitely check out our upcoming conference, Structure 08.

    Technology-News: GigaOm

    Amazon EC2 image of BlueDragon/CentOS5/Jetty

    Free way of hosting a ColdFusion application using the newly open sourced BlueDragon engine

    jetty: del.icio.us/tag/jetty

    Amazon Now Serving OpenSolaris on EC2

    During our on-stage chat at Startup Camp, Sun Microsystems CEO Jonathan Schwartz