'Net Features : amazon web services

Janitor Monkey Will Clean Up Your Old AWS Instances

Michael Garrity — Tue, 08 Jan 2013 22:12:00 GMT

If you’re like me, you probably sat down with your family over the holiday break to relax and watch something on Netflix, only to find that its service was uncharacteristically down. Well, it seems that the company has corrected this problem, and now wants to share what it learned with the rest of us.

Netflix officially released a new open source tools for killing old Amazon Web Services (AWS) instances this week known as Janitor Monkey. This solution, which is part of the company’s “Simian Army” of internal management tools, is meant for enterprise organizations that employ a public cloud through AWS, particularly when they are unknowingly spending unnecessary money because they have forgotten to shut off an old instance.

“It is pretty easy to lose track of the cloud resources that are no longer needed or used,” say Netflix’s Michael Fu and Corey Bennett in a recent blog post. “Perhaps you forgot to delete the cluster with the previous version of your application, or forgot to destroy the volume when you no longer needed the extra disk.”

Janitor Monkey works largely with Netflix’s open source Asgard tool, which lets administrators delete their unused resources; this process is streamlined thanks to Janitor Monkey, which does the work of automatically tracking down these useless instances and allowing Asgard to clean them up.

According to Netflix, Janitor Monkey had deleted over 5000 resources from the company’s production and test environments when it was being used as an in-house product.

In addition to AWS instances, Janitor Monkey also detects EBS volumes, EBS volume snapshots and auto-scaling groups, and all of them come with their own unique rules for how they are marked. (For example, EBS volumes are marked as “cleanup candidates” if they haven’t been attached an instance for 30 days.) Once Janitor Monkey inspects these resources and applies the appropriate rules to them, it will determine if they are, in fact, cleanup candidates and, if so, it marks them for clean up and schedules a time for Asgard to do it. These events will be logged in an Amazon SimpleDB table by default, and it should be small enough to fit inside Amazon’s free-pricing tier.

However, just to double check itself and make sure it isn’t acting too hastily, Janitor Monkey will then inform the administrations attached the specific resources with an alert sent two days before it is scheduled to be cleaned up. Out of the box, Janitor Monkey is automatically configured to run on non-holiday weekends at 11 a.m., but it can be modified to run at other customized times.

High Storage Instances Come to Amazon EC2

Michael Garrity — Fri, 28 Dec 2012 21:09:00 GMT

Amazon Web Services (AWS) wanted to give its Elastic Compute Cloud (EC2) users a present, and what better way to celebrate the holidays than with family – a new instance family, that is.

The cloud computing services provider from Amazon recently announced High Storage instances for EC2. This additional instance family is optimized for applications that required rapid access to large amounts of data, while also providing AWS customers with 35 EC2 Compute Units of computing capacity, 117 GiB of RAM and 48 TB of storage across 24 hard disk drives. Moreover, these instances are capable of delivering more than 2.4 GB of sequential I/O performance per second.

In other words, because High Storage instances provide such massive amounts of direct attached storage per instance, they are ideal for data-intensive applications like Hadoop workloads, log processing, data warehousing and parallel file systems for processing and analyzing large datasets in the AWS cloud.

High Storage instances follow eight other EC2 instance families, including Cluster Compute and High I/O instances, to help meet the evolving application requirements of Amazon EC2 customers. Like the others, High Storage instances were designed to enhance the performance and efficiency of even the most demanding applications. They also power the new petabyte-scale data warehousing service Amazon Redshift and can help Amazon Elastic MapReduce customers process larger quantities of data more resourcefully, who helps significantly lower costs.

AWS customers can immediately launch High Storage instances through the AWS Management Console, Amazon EC2 and Elastic MapReduce Command Line Interfaces, AWS SDKs and various other third-party libraries. However, at the moment this new instance family is only available in the US East Region; they will be made available in other AWS Regions “in the coming months.”

Riak CS Lets Users Replicate Data Globally

Michael Garrity — Sat, 08 Dec 2012 14:46:00 GMT

Customers can now replicate Riak Cloud Storage (Riak CS) data across multiple data centers to assure its users that they can avoid disruptions from outages and serve content faster to multiple geographic locations, all while building upon its existing compatibility with Amazon S3.

The cloud storage service from Basho Technologies will now allow customers to spread their stored data over a series of data centers located around the world, and the integration of the Amazon Web Services (AWS) infrastructure guarantees high availability. Basically, Riak CS customers will now be able to replicate their stored data on any data center they want on their own terms, in addition all of the benefits of using AWS, just to sweeten the deal. This solution also reduces some of the risks commonly associated with AWS.

Moreoever, these new replication capabilities will break large data objects into smaller blocks that will be streamed to the underlying Riak cluster, where they will then be replicated for high availability. A manifest for each object will be maintained, allowing the service to retrieve each block from the cluster and present the full object to the end-user. Global information, bucket information and manifests will all be streamed in real-time from a primary implementation to a secondary site for multi-site replication, and objects can be replicated in either “full” or “real-time” sync modes.

Riak CS, built upon Riak, the open source distributed NoSQL database, was just released by Basho last year, and the Amazon S3 integration, which provides Riak CS customers with access to S3 tools and frameworks, was only announced early in 2012.

NetApp Brings AWS to the Enterprise

Michael Garrity — Mon, 03 Dec 2012 14:41:00 GMT

NetApp, the cloud storage provider, recently brought the power of Amazon Web Services (AWS) to its customers.

The company’s new solution, NetApp Private Storage for AWS, will use the AWS Direct Connect service to allow enterprise users to establish a dedicated network connection that links their existing infrastructure to AWS. This means they can now replicate data from their on-premises NetApp storage environments in a Direct Connect facility.

Enterprise companies will now be able to use NetApp with a variety of AWS services, including Amazon EMR for more cost-effective big data analysis, disaster recovery deployments with Amazon EC2 and disk backup through Amazon S3, in addition to being able to seamlessly move enterprise applications to the cloud and between AWS regions.

Right now, there is a fear on the part of some enterprise organizations when it comes to
moving workloads (especially larger ones) to AWS, but by going through NetApp, which has been serving enterprises for 20 years, these companies may feel more comfortable using Amazon’s service.

“Our work with NetApp provides customers with the performance, security, compliance, and availability benefits of AWS with the high value enterprise offerings customers have come to expect from NetApp,” says Terry Wise, the head of Worldwide Partner Ecosystems at AWS.

Currently, NetApp Private Storage for AWS is available to select resellers in North America, but it should be available in Europe and Asia sooner than later.

AWS Comes to CoreSite's New York Datacenter

Michael Garrity — Tue, 21 Aug 2012 09:00:00 GMT

Amazon Web Services (AWS) is going metropolitan. For the first time, AWS Direct Connect will have a presence in New York thanks to a new partnership with datacenter provider CoreSite.

CoreSite provides datacenter products and interconnection services to over 750 customers, including enterprise-level organizations, communications providers, cloud and content companies, financial firms, media and entertainment companies, government agencies and more. And now, the company boasts AWS Direct Connect availability in its New York datacenter.

This move helps CoreSite meet the increasing customer demand for access to the AWS cloud, and will help improve its ability to service digital content communities, financial organizations and managed service providers, as Direct Connect allows customers to directly access cloud services using a secure private network connection. Direct access provides increased scalability for throughput and a more consistent network performance for reduced network costs. It also helps reduce bandwidth costs while increasing capacity.

More than anything, this decision makes it easy for CoreSite customers to establish a dedicated network connection from their location to the AWS cloud without accessing the public cloud over the Internet. This leads to improved application performance and enhanced security – two of the biggest concerns for companies using the cloud.

Customers can access Direct Connect using a standard, single-mode fiber 1 Gbps or 10 Gbps Ethernet cross connection or an Any2 Internet exchange connection, which can be ordered through the MyCoreSite online customer portal.

CoreSite has been offering AWS Direct Connect from its Los Angeles datacenter since late last year.

Can Google Compute Compete with AWS?

Pete Prestipino — Tue, 03 Jul 2012 10:50:00 GMT

Google is rolling out a suite of infrastructure-as-a-service products that many expect might just be the strongest competitor yet to market leader Amazon Web Services.

Supporting its own services as well as over one million active applications, Google currently has one of the largest data centers in the world and more importantly the network to connect them.

The Google Compute Engine will provide Linux virtual machines and provide storage and connectivity. Initially, developers needing an infrastructure for large-scale computing will be able to execute batch processing jobs like video transcoding and rendering, analyze big data in the cloud using frameworks like Hadoop, and run high-performance and grid computing workloads using the Google Compute Engine.

In relation to pricing, Compute Engine is a bit cheaper (approximately 10-20%) and Compute Engine also offers slightly larger default instance types, but doesn't offer the high memory or high cpu instances that AWS does.

Based on what you know now, is it time for a switch from AWS to Google Compute?

Deliver Dynamic Content with Amazon CloudFront

Michael Garrity — Tue, 15 May 2012 11:00:00 GMT

Amazon Web Services (AWS) is helping businesses get more interactive by launching dynamic content support for Amazon CloudFront, giving customers a simple and cost-effective way to improve the performance, reliability and global reach of their sites and delivery of their content, including dynamic content that changes for every end-user.

AWS customers can accelerate all of the content on their websites, both dynamic and static, for a single price and no up-front fees.

The ability to deliver personalized, dynamic content through CloudFront saves businesses a lot of time and effort that used to go into improving the performance and reliability of the more dynamic aspects of their websites. Typically, this used to require custom codes that had to be written, and even then the solutions offered would be hard to configure and manage.

Now users can run all kinds of Web applications and accelerate their entire sites quickly through the AWS Management Console with no additional cost or architecture complexity. The experience even improves when dynamic content is delivered with origin servers running in EC2, as Amazon will monitor and streamline the network paths from each CloudFront edge location to the various AWS Regions, improving latency and reliability in the process.

AWS Adds SSH Client, Drops Prices (Again)

Michael Garrity — Mon, 12 Mar 2012 20:30:00 GMT

At this point, it's pretty much common knowledge that Amazon Web Services (AWS), the e-commerce giant's cloud computing services provider, is going to make waves in the industry, and this last week really helped to reinforce that idea.

In terms of functionality, the company recently added an SSH client to the AWS console using MindTerm's Java-based SSH offering. The client will be available for Amazon EC2 users, and the announcement was packaged with news about a new instance type, the Medium (m1.medium), as well as the ability to launch 64-bit operating systems on the m1.small and c1.medium instances.

The new Medium instance type will come with 3.75 GB of RAM, a virtual core running at two ECU, 410 GB of instance storage, the ability to launch 32- and 64-bit operating systems and moderation I/O performance. In addition, the capability to run a 64-bit OS means users can create a single Amazon Machine Image (AMI) and run it on a wide range of instance types.

Including an SSH client makes it far easier for users (especially Windows users) to connect to an EC2 instance in just a couple of clicks.

Plus, less than a week ago the company dramatically slashed prices (for the 19th time since its inception) for EC2 instances, ElastiCache, Amazon Relational Dadabase Service and Amazon Elastic Map Reduce.

EC2 prices will drop by as much as 10 percent for on-demand services, while prices for Reserved Instances will by cut by 37 percent. Moreover, AWS is offering volume discounts of 10 percent for additional Reserved Instances for customers that already own over $250,000 worth. At $2 million it becomes a 20-percent discount.

Amazon's big announcements are already shaking up the market, as just a few days later Microsoft announced that it will be cutting pricing for its Azure Storage and Compute service.

Amazon Web Services' Simple Workflow Service (SWF)

Pete Prestipino — Thu, 23 Feb 2012 04:00:00 GMT

Amazon has just introduced Amazon Simple Workflow Service (Amazon SWF), enabling developers to coordinate processes when building distributed applications.

Developers using the service will be able to structure various processing steps in an application as tasks. Amazon SWF can then coordinate those tasks and manage their execution dependencies, scheduling and concurrency based on a developer’s application logic. The service stores tasks, dispatches them to application components, tracks their progress, and keeps their latest state. Pretty awesome.

NASA's Jet Propulsion Lab (JPL) is already using Amazon SWF as part of several space and earth science missions including the Mars Exploration Rover. The Amazon SWF Service gives JPL the ability to leverage internal and external resources, enabling their applications to scale dynamically and run in a distributed manner. NASA JPL has integrated Amazon SWF into their Cloud Oriented Architecture and its reference implementation, Polyphony.

“With Amazon SWF, developers can now easily coordinate distributed application components across on-premises and cloud environments using their choice of programming languages,” says Reto Kramer, general manager of AWS' Application Connection Services. “By relying on Amazon SWF to handle the coordination of distributed task execution, developers can now focus on building the differentiating aspects of their applications and leave the undifferentiated heavy lifting of building and managing workflow engines to AWS.”

Best of 2011 - Cloud Services

Michael Garrity — Thu, 29 Dec 2011 16:00:00 GMT

It’s been written countless times that the cloud is the future of Web tech and will possibly be the most important investment most businesses make in the near future – especially enterprise companies.

Of course, there are those with strong (and valid) opinions on the subject, both in favor of cloud adoption and stringently against it. However, as the debate ensues, there are many companies in the industry providing great services to customers of various sizes and requirements.

2011 saw Internet empires and hungry young companies alike playing a crucial role in the development of cloud technologies and, more importantly, services. Here are five of the best cloud service providers from the past 12 months.

Amazon Web Services
Most people just know Amazon as the online retail site, but Web professionals are aware of its other prominent role on the ‘Net as one of the largest cloud service providers available.

This may be a somewhat controversial choice seeing as the service has some highly-noted mishaps this past year, but nonetheless it remains a leading innovator in the industry, including adding a “white glove” service for premium customers that will route your calls (and concerns) to the nearest engineering specialist.

AWS will scale to meet the application demands of its clients, “whether one server or a large cluster,” offers a pay-as-you-go, low cost service, instantly deploys applications and allows for full open and flexible control of all of your data. Despite the blunders of 2011 (of which customers have understandable concerns), Amazon Web Services continues to be (one of?) the best cloud provider out there.

Box
Box is probably the busiest provider on this list. It has been constantly innovating its product over the last year, especially in mobile applications, and has been able to carve out a significant segment of the market as a result.

While it may be most popular as a personal consumer service, Box has accrued and impressive list of businesses and enterprise-level companies as customers, including Proctor & Gamble, Six Flags, Taylor Made, Pandora, Clear Channel, the San Francisco Giants and more.

Businesses are helped to a host of features in areas like content management with desktop syncing and link-based file sharing, mobile access on Android and iOS devices, online workspaces, a host of administrative controls and easy integration into existing business software like Google Apps, Salesforce and NetSuite, as well as offering APIs for custom integrations.

Rackspace
In terms of revenue generated, Rackspace trails only Amazon Web Services. While that’s not necessarily an indicator of competence, it does speak highly that so many users are willing to trust it over major brands like Microsoft and Google.

This year saw less in the way of innovation than most companies, the soft launch of OpenStack was notable and they released their impressive Hybrid Hosting option that provides both cloud and managed hosting in one service. Still, Rackspace has continued to impress with solid service and security; after all, there are currently over 100,000 companies currently working with Rackspace.

Rackspace’s on-demand scalable servers come with up to 30 GB of storage, only require businesses to pay for what they use, are fully-customizable with root access and are completed by the company’s “Fanatical Support.”

Dropbox
Many have at least heard of Dropbox, and it has long been one of the more heavily-utilized cloud service providers available. (For instance, my own list of Dropbox Alternatives suggests that it has become the standard for business and personal cloud computing.)

Like Box, Dropbox offers a really impressive mobile service that is compatible with Windows, Mac, Linux, iPad, iPhone, Android and Blackberry devices and still works even when the user is offline. The service also provides businesses with the ability to utilize the same tools they work with daily, get started in minutes, put all of their services on a single bill and manage and migrate accounts. Heck, you even get unlimited version history for all of your files hosted with Dropbox.

Evernote
If the opinions of tech industry insiders around the globe are to be trusted, it looks like Evernote may be the next big thing in cloud computing. What sets this product apart is that it is more of a note-taking and idea-saving service than a traditional cloud storage platform. The idea with Evernote is that users can save important ideas in “notebooks” and then retrieve these ideas later, which is made simple thanks to a search function and the ability to tag things you save.

It also has mobile compatibility, so you can access your ideas from almost any mobile device, as “Evernote works with nearly every computer, phone and mobile device out there.” What makes Evernote an especially convenient tool is that you can take notes everywhere you go and keep all of them in sync and readily available on any device you may use anywhere.

What’s New with Amazon’s Cloud

Allison Howen — Tue, 06 Sep 2011 23:00:00 GMT

Amazon is constantly releasing new features and upgrades to its cloud— Amazon Web Services. If you don't constantly pay attention to developments on the most popular cloud environment, you risk getting left behind.

To catch everyone up, here are some of the newest features worthy of your attention:

Easier Access: Adobe Flash and outbound access to port 843 are no longer needed, instead the S3 tab can be used from behind a regular or transparent proxy.

Folder Upload: A new and advanced uploader allows entire folders as well as objects larger than 5 GB to be uploaded at once. To enable the uploader, click on the upload button and then enable the enhanced uploader.

Jump: The jump feature allows the console to locate files and objects easier, by bringing up matching files as you start to type the prefix of the object or folder you are searching for.

ElastiCache: This feature adds caching logic to your application. When Cache Clusters are created, they can make your application run smoother, by caching key pieces of date into memory.

Elastic MapReduce on EC2 Spot Instances: This feature takes two of Amazon’s Elastic Compute Cloud’s (EC2) features, Spot Instances and Elastic MapReduce and allows users to launch managed Hadoop clusters with unused EC2 capacity. This allows for the running of long jobs, cost-driven workloads, data-critical workloads and application testing at a discount between 50 to 66 percent.

Amazon Announces Notification Service for Developers

Linc Wonham — Thu, 08 Apr 2010 13:36:00 GMT

Amazon has announced the launch of its cloud-based simple notification service that will allow Web developers to create a push notification system for their applications. The new service, Amazon SNS, will deliver messages to users across a number of different formats, including E-mail and HTTP.

Amazon SNS provides a simple Web services interface that can be used to create topics that developers want to notify applications or people about (such as smartphone applications), subscribe clients to those topics, publish messages, and have the messages delivered over the clients’ protocol of choice. Developers who use SNS will be charged only for the services they use (all true cloud services are scalable and flexible).

Customers will pay no charges for the first 100,000 requests, no charges for the first 100,000 HTTP notifications and no charges for the first 1,000 E-mail notifications, according to Amazon Web Services. After those limits are reached, charges will be $.06 per 100,000 API calls.

Amazon’s Virtual Private Cloud

Pete Prestipino — Wed, 26 Aug 2009 14:03:00 GMT

Amazon Web Services launched Amazon VPC, a bridge between a company’s existing IT infrastructures and the AWS cloud.

Amazon VPC (which integrates today with Amazon EC2) enables companies to connect their infrastructure to a set of isolated AWS compute resources via a virtual private network (VPN) connection, extending management capabilities such as security services, firewalls, and intrusion detection systems to include their AWS resouces.

“For the last three years, AWS has provided companies of all sizes with on-demand, highly elastic and highly reliable technology resources in the cloud. As more and more enterprises leverage the cloud, they want a simple, seamless way to migrate their large and complex IT infrastructures to AWS, and to use the security and management controls that their IT teams already know,” said Andy Jassy, Senior Vice President, Amazon Web Services. “We built Amazon VPC for this purpose—to allow any company to seamlessly connect their existing resources to the AWS cloud as if it were a part of their own datacenter.”

Stay up to date on the latest Internet trends:
Request a professional subscription to Website Magazine,
the most popular print publication on Web success.

Move to the Front of the Cloud

Pete Prestipino — Thu, 15 Jan 2009 15:14:00 GMT

Amazon has unveiled a new web content delivery service named Amazon CloudFront.

The service could prove useful in improving the performance of a website by giving developers and businesses an easy way to distribute content to end users with low latency, high data transfer speeds, and no commitments - the same as the benefits of using a content delivery network (CDN).

Amazon CloudFront delivers content using a global network of edge locations. Requests for objects (files) are automatically routed to the nearest edge location, so content is delivered with the best possible performance. Using a content delivery network results in low latency and higher data transfer speeds overall, but only until recently has such an opportunity been available on such a scale.

CloudFront works with Amazon Simple Storage Service (Amazon S3) to store the original, definitive versions of your files. Like other Amazon Web Services, there are no contracts or monthly commitments for using Amazon CloudFront. Users pay for as much or as little content as is actually delivered through the service.

Amazon has even made an Amazon CloudFront manager available. The application will come in handy for S3 users as they upload files to to the AS3 network and sitribute them through CloudFront. Pricing (seen below) seems reasonable.

Data Transfer
$0.170 per GB – first 10 TB / month data transfer out
$0.120 per GB – next 40 TB / month data transfer out
$0.100 per GB – next 100 TB / month data transfer out
$0.090 per GB – data transfer out / month over 150 TB

Requests
$0.010 per 10,000 GET requests

Amazon Slashes S3 Prices

Pete Prestipino — Thu, 09 Oct 2008 20:28:00 GMT

Just a quick note: Amazon Web Services announced that effective November 1st, 2008, a new tiered pricing model for Amazon S3 will go into effect. The new model features four price tiers, with prices decreasing based on the amount of storage used by each customer. If you are familiar with pricing at S3 currently, you will be able to tell that Amazon is offering volume discounts for those that use more storage. While the price-cutting does not apply to data transfers, it's still going to help those using the service some some money.

Some additional information about S3:

- Currently there are over 29 billion objects stored in Amazon S3 vs. 22 billion at the end of Q2 2008.
- On Oct. 1, the service peaked at over 70,000 requests per second to store, retrieve, or delete an object.
- Over 400,000 developers have registered to use Amazon Web Service.