'Net Features : aws

Janitor Monkey Will Clean Up Your Old AWS Instances

Michael Garrity — Tue, 08 Jan 2013 22:12:00 GMT

If you’re like me, you probably sat down with your family over the holiday break to relax and watch something on Netflix, only to find that its service was uncharacteristically down. Well, it seems that the company has corrected this problem, and now wants to share what it learned with the rest of us.

Netflix officially released a new open source tools for killing old Amazon Web Services (AWS) instances this week known as Janitor Monkey. This solution, which is part of the company’s “Simian Army” of internal management tools, is meant for enterprise organizations that employ a public cloud through AWS, particularly when they are unknowingly spending unnecessary money because they have forgotten to shut off an old instance.

“It is pretty easy to lose track of the cloud resources that are no longer needed or used,” say Netflix’s Michael Fu and Corey Bennett in a recent blog post. “Perhaps you forgot to delete the cluster with the previous version of your application, or forgot to destroy the volume when you no longer needed the extra disk.”

Janitor Monkey works largely with Netflix’s open source Asgard tool, which lets administrators delete their unused resources; this process is streamlined thanks to Janitor Monkey, which does the work of automatically tracking down these useless instances and allowing Asgard to clean them up.

According to Netflix, Janitor Monkey had deleted over 5000 resources from the company’s production and test environments when it was being used as an in-house product.

In addition to AWS instances, Janitor Monkey also detects EBS volumes, EBS volume snapshots and auto-scaling groups, and all of them come with their own unique rules for how they are marked. (For example, EBS volumes are marked as “cleanup candidates” if they haven’t been attached an instance for 30 days.) Once Janitor Monkey inspects these resources and applies the appropriate rules to them, it will determine if they are, in fact, cleanup candidates and, if so, it marks them for clean up and schedules a time for Asgard to do it. These events will be logged in an Amazon SimpleDB table by default, and it should be small enough to fit inside Amazon’s free-pricing tier.

However, just to double check itself and make sure it isn’t acting too hastily, Janitor Monkey will then inform the administrations attached the specific resources with an alert sent two days before it is scheduled to be cleaned up. Out of the box, Janitor Monkey is automatically configured to run on non-holiday weekends at 11 a.m., but it can be modified to run at other customized times.

High Storage Instances Come to Amazon EC2

Michael Garrity — Fri, 28 Dec 2012 21:09:00 GMT

Amazon Web Services (AWS) wanted to give its Elastic Compute Cloud (EC2) users a present, and what better way to celebrate the holidays than with family – a new instance family, that is.

The cloud computing services provider from Amazon recently announced High Storage instances for EC2. This additional instance family is optimized for applications that required rapid access to large amounts of data, while also providing AWS customers with 35 EC2 Compute Units of computing capacity, 117 GiB of RAM and 48 TB of storage across 24 hard disk drives. Moreover, these instances are capable of delivering more than 2.4 GB of sequential I/O performance per second.

In other words, because High Storage instances provide such massive amounts of direct attached storage per instance, they are ideal for data-intensive applications like Hadoop workloads, log processing, data warehousing and parallel file systems for processing and analyzing large datasets in the AWS cloud.

High Storage instances follow eight other EC2 instance families, including Cluster Compute and High I/O instances, to help meet the evolving application requirements of Amazon EC2 customers. Like the others, High Storage instances were designed to enhance the performance and efficiency of even the most demanding applications. They also power the new petabyte-scale data warehousing service Amazon Redshift and can help Amazon Elastic MapReduce customers process larger quantities of data more resourcefully, who helps significantly lower costs.

AWS customers can immediately launch High Storage instances through the AWS Management Console, Amazon EC2 and Elastic MapReduce Command Line Interfaces, AWS SDKs and various other third-party libraries. However, at the moment this new instance family is only available in the US East Region; they will be made available in other AWS Regions “in the coming months.”

Riak CS Lets Users Replicate Data Globally

Michael Garrity — Sat, 08 Dec 2012 14:46:00 GMT

Customers can now replicate Riak Cloud Storage (Riak CS) data across multiple data centers to assure its users that they can avoid disruptions from outages and serve content faster to multiple geographic locations, all while building upon its existing compatibility with Amazon S3.

The cloud storage service from Basho Technologies will now allow customers to spread their stored data over a series of data centers located around the world, and the integration of the Amazon Web Services (AWS) infrastructure guarantees high availability. Basically, Riak CS customers will now be able to replicate their stored data on any data center they want on their own terms, in addition all of the benefits of using AWS, just to sweeten the deal. This solution also reduces some of the risks commonly associated with AWS.

Moreoever, these new replication capabilities will break large data objects into smaller blocks that will be streamed to the underlying Riak cluster, where they will then be replicated for high availability. A manifest for each object will be maintained, allowing the service to retrieve each block from the cluster and present the full object to the end-user. Global information, bucket information and manifests will all be streamed in real-time from a primary implementation to a secondary site for multi-site replication, and objects can be replicated in either “full” or “real-time” sync modes.

Riak CS, built upon Riak, the open source distributed NoSQL database, was just released by Basho last year, and the Amazon S3 integration, which provides Riak CS customers with access to S3 tools and frameworks, was only announced early in 2012.

NetApp Brings AWS to the Enterprise

Michael Garrity — Mon, 03 Dec 2012 14:41:00 GMT

NetApp, the cloud storage provider, recently brought the power of Amazon Web Services (AWS) to its customers.

The company’s new solution, NetApp Private Storage for AWS, will use the AWS Direct Connect service to allow enterprise users to establish a dedicated network connection that links their existing infrastructure to AWS. This means they can now replicate data from their on-premises NetApp storage environments in a Direct Connect facility.

Enterprise companies will now be able to use NetApp with a variety of AWS services, including Amazon EMR for more cost-effective big data analysis, disaster recovery deployments with Amazon EC2 and disk backup through Amazon S3, in addition to being able to seamlessly move enterprise applications to the cloud and between AWS regions.

Right now, there is a fear on the part of some enterprise organizations when it comes to
moving workloads (especially larger ones) to AWS, but by going through NetApp, which has been serving enterprises for 20 years, these companies may feel more comfortable using Amazon’s service.

“Our work with NetApp provides customers with the performance, security, compliance, and availability benefits of AWS with the high value enterprise offerings customers have come to expect from NetApp,” says Terry Wise, the head of Worldwide Partner Ecosystems at AWS.

Currently, NetApp Private Storage for AWS is available to select resellers in North America, but it should be available in Europe and Asia sooner than later.

Amazon Web Services Down Under

Pete Prestipino — Mon, 12 Nov 2012 18:15:00 GMT

Amazon Web Services has announced the launch of a new Asia Pacific (Sydney) Region, the ninth in its global cloud computing platform.

“Over 10,000 customers in Australia and New Zealand are already using AWS, and this is before opening our new AWS Region in Australia today,” said Andy Jassy, Senior Vice President, Amazon Web Services. “With the ability to achieve single-digit millisecond latency to end users in Sydney, store data locally in Australia, and get to market more quickly and inexpensively by using AWS’s unmatched infrastructure technology platform, we expect the launch of AWS’s Sydney Region to further increase the amount of Australian and New Zealand customers leveraging AWS.”

At launch, the Sydney region consists of two seperate Availability Zones (datacenters in seperate distinct locations within a region which are designed to be operationally independent) and will support numerous AWS services including Elastic Compute Cloud, Map Reduce, DynamoDB, Simple DB and the Amazon Relational Database Service among others.

Among the more than 10,000 customers in Australia and New Zealand that are already using AWS, are the Commonwealth Bank of Australia (for running customer-facing Web apps), BrandScreen (which is using Elastic MapReduce to test machine learning algorithms and process data for its real-time ad trading platform), and Halfbrick Studioes - makers of FruitNinja - which is using DynamoDB and multiple Availability Zones to host millions of regular players.

AWS Comes to CoreSite's New York Datacenter

Michael Garrity — Tue, 21 Aug 2012 09:00:00 GMT

Amazon Web Services (AWS) is going metropolitan. For the first time, AWS Direct Connect will have a presence in New York thanks to a new partnership with datacenter provider CoreSite.

CoreSite provides datacenter products and interconnection services to over 750 customers, including enterprise-level organizations, communications providers, cloud and content companies, financial firms, media and entertainment companies, government agencies and more. And now, the company boasts AWS Direct Connect availability in its New York datacenter.

This move helps CoreSite meet the increasing customer demand for access to the AWS cloud, and will help improve its ability to service digital content communities, financial organizations and managed service providers, as Direct Connect allows customers to directly access cloud services using a secure private network connection. Direct access provides increased scalability for throughput and a more consistent network performance for reduced network costs. It also helps reduce bandwidth costs while increasing capacity.

More than anything, this decision makes it easy for CoreSite customers to establish a dedicated network connection from their location to the AWS cloud without accessing the public cloud over the Internet. This leads to improved application performance and enhanced security – two of the biggest concerns for companies using the cloud.

Customers can access Direct Connect using a standard, single-mode fiber 1 Gbps or 10 Gbps Ethernet cross connection or an Any2 Internet exchange connection, which can be ordered through the MyCoreSite online customer portal.

CoreSite has been offering AWS Direct Connect from its Los Angeles datacenter since late last year.

Deliver Dynamic Content with Amazon CloudFront

Michael Garrity — Tue, 15 May 2012 11:00:00 GMT

Amazon Web Services (AWS) is helping businesses get more interactive by launching dynamic content support for Amazon CloudFront, giving customers a simple and cost-effective way to improve the performance, reliability and global reach of their sites and delivery of their content, including dynamic content that changes for every end-user.

AWS customers can accelerate all of the content on their websites, both dynamic and static, for a single price and no up-front fees.

The ability to deliver personalized, dynamic content through CloudFront saves businesses a lot of time and effort that used to go into improving the performance and reliability of the more dynamic aspects of their websites. Typically, this used to require custom codes that had to be written, and even then the solutions offered would be hard to configure and manage.

Now users can run all kinds of Web applications and accelerate their entire sites quickly through the AWS Management Console with no additional cost or architecture complexity. The experience even improves when dynamic content is delivered with origin servers running in EC2, as Amazon will monitor and streamline the network paths from each CloudFront edge location to the various AWS Regions, improving latency and reliability in the process.

Just How Big is Amazon S3? Really Big (and Growing)

Pete Prestipino — Mon, 09 Apr 2012 16:30:00 GMT

The size of the cloud just keeps growing – at least it does for Amazon.

The company announced late last week that at the end of the first quarter of 2012, there were 905 billion objects in Amazon S3 and that it routinely handles 650,000 requests per second for those objects – with occasional peaks far above that number.

Amazon S3's cloud growth is by all accounts phenomenal. Back in the fourth quarter of 2006, there were just 2.9 billion objects stored. That number had increased to 40 billion in the fourth quarter of 2008 and 262 billion in the fourth quarter of 2010. Just one year later, the total number of objects stored in Amazon S3 had skyrocketed to 762 billion.

Amazon indicated that the S3 object count continues to grow at a rapid clip even after they added object expiration and multi-object deletion at the end of the year. Every day, well over a billion objects are added via the S3 APIs, AWS Import/Export, the AWS Storage Gateway, all sorts of backup tools, and through Direct Connect pipes.

AWS Adds SSH Client, Drops Prices (Again)

Michael Garrity — Mon, 12 Mar 2012 20:30:00 GMT

At this point, it's pretty much common knowledge that Amazon Web Services (AWS), the e-commerce giant's cloud computing services provider, is going to make waves in the industry, and this last week really helped to reinforce that idea.

In terms of functionality, the company recently added an SSH client to the AWS console using MindTerm's Java-based SSH offering. The client will be available for Amazon EC2 users, and the announcement was packaged with news about a new instance type, the Medium (m1.medium), as well as the ability to launch 64-bit operating systems on the m1.small and c1.medium instances.

The new Medium instance type will come with 3.75 GB of RAM, a virtual core running at two ECU, 410 GB of instance storage, the ability to launch 32- and 64-bit operating systems and moderation I/O performance. In addition, the capability to run a 64-bit OS means users can create a single Amazon Machine Image (AMI) and run it on a wide range of instance types.

Including an SSH client makes it far easier for users (especially Windows users) to connect to an EC2 instance in just a couple of clicks.

Plus, less than a week ago the company dramatically slashed prices (for the 19th time since its inception) for EC2 instances, ElastiCache, Amazon Relational Dadabase Service and Amazon Elastic Map Reduce.

EC2 prices will drop by as much as 10 percent for on-demand services, while prices for Reserved Instances will by cut by 37 percent. Moreover, AWS is offering volume discounts of 10 percent for additional Reserved Instances for customers that already own over $250,000 worth. At $2 million it becomes a 20-percent discount.

Amazon's big announcements are already shaking up the market, as just a few days later Microsoft announced that it will be cutting pricing for its Azure Storage and Compute service.

Amazon Web Services' Simple Workflow Service (SWF)

Pete Prestipino — Thu, 23 Feb 2012 04:00:00 GMT

Amazon has just introduced Amazon Simple Workflow Service (Amazon SWF), enabling developers to coordinate processes when building distributed applications.

Developers using the service will be able to structure various processing steps in an application as tasks. Amazon SWF can then coordinate those tasks and manage their execution dependencies, scheduling and concurrency based on a developer’s application logic. The service stores tasks, dispatches them to application components, tracks their progress, and keeps their latest state. Pretty awesome.

NASA's Jet Propulsion Lab (JPL) is already using Amazon SWF as part of several space and earth science missions including the Mars Exploration Rover. The Amazon SWF Service gives JPL the ability to leverage internal and external resources, enabling their applications to scale dynamically and run in a distributed manner. NASA JPL has integrated Amazon SWF into their Cloud Oriented Architecture and its reference implementation, Polyphony.

“With Amazon SWF, developers can now easily coordinate distributed application components across on-premises and cloud environments using their choice of programming languages,” says Reto Kramer, general manager of AWS' Application Connection Services. “By relying on Amazon SWF to handle the coordination of distributed task execution, developers can now focus on building the differentiating aspects of their applications and leave the undifferentiated heavy lifting of building and managing workflow engines to AWS.”

Big Changes Coming to Amazon Web Services

Michael Garrity — Fri, 05 Aug 2011 16:00:00 GMT

Amazon Web Services (AWS), the Web services division of the e-commerce juggernaut, has dropped some big news about some new capabilities for customers that are designed to help support the security, network management, dedicated connectivity and identity management requirements that enterprises need when they install mission-critical applications in the cloud.

One huge change taking place is the expansion of Amazon Virtual Private Cloud (Amazon VPC) to all AWS regions. This will allow enterprises to launch their Amazon VPC ecosystems around the world, including from the east and west coasts of the United States, Europe, Singapore and Tokyo, as well as the multiple AWS Availability Zones within each region.

With Amazon VPC, businesses can now procure a private section of AWS and then open their AWS resources in a virtual network that they define and control. This includes selecting the IP address range, creating subnets and configuring route tables and network gateways. Now, businesses have an easy method of leveraging AWS while keeping the same security and management controls already familiar to them.

AWS also announced AWS Direct Connect, which will allow enterprises to bypass the Internet and deliver data to and from AWS by using a private connection, which will help increase bandwidth throughout, reduce networking latency and costs, and provide a more consistent networking experience for when enterprises need to move data between AWS and their data centers.

AWS Identity and Access Management (AWS IAM) also got an overhaul, adding new functionality that enables “identity federation,” which is the ability for enterprises to use their existing corporate identities to allow users both secure and direct access to AWS resources without having to create a new AWS identity for them. Because of this, enterprises can programmatically request the necessary security credentials that grant their corporate identities access to AWS resources that are controlled by that enterprise.

Both Amazon VPC and IAM can be accessed through the AWS Management Console. Console support for AWS Direct Connect will come along later this year.

Amazon Introduces New Email Service for Businesses

Linc Wonham — Tue, 25 Jan 2011 14:42:00 GMT

Amazon has announced the launch of a bulk and transactional email-sending service for businesses and developers. As part of Amazon Web Services (AWS), the new Amazon Simple Email Service (Amazon SES) is designed to give customers a highly scalable and cost-effective alternative to building in-house email solutions or licensing, installing and operating a third-party email service.

The service integrates with other AWS services, allowing customers to seamlessly send emails from applications being hosted on services such as Amazon EC2. Amazon says there is no long-term commitment, minimum spend or negotiation required, so businesses can utilize a free usage tier and after that pay low fees for the number of emails sent plus data transfer.

Pricing is $0.10 per thousand email messages sent, and customers can send 2,000 email messages for free each day when these emails originate from Amazon EC2 or AWS Elastic Beanstalk (data transfer fees may still apply if a customer exceeds their AWS free monthly bandwidth allowance).

Amazon SES uses content filtering technologies to scan outgoing email messages to help ensure that the content meets ISP standards. The email message is then either queued for sending or routed back to the sender for corrective action.

Amazon SES also provides a built-in feedback loop, which includes notifications of bouncebacks, failed and successful delivery attempts, and spam complaints. For more information or to get started using Amazon SES, visit AWS.