Blog

Recap of AWS re:Invent 2015 in Las Vegas

Joseph Lawson
Category: Conference

Recap of AWS re:Invent 2015 in Las Vegas

Amazon Web Services’ (AWS) re:Invent conference is one of the biggest cloud conferences in the world. With over 18,000 people in attendance, there are sure to be a few good ideas lurking about. For this article I’m going to go over a few of my favorite observations and share what significance they hold for future computing. Going all in was a very big theme this year for the conference. AWS trotted out many different companies to deliver the message, “You can trust AWS”, but we already knew that. So what is happening behind the scenes? Ideas such as microservices and containerization are maturing. This maturation led to more breakout sessions talking best practices and patterns versus last year where mainly overviews were being given. Also AWS has been around long enough to have stores of companies that migrated to the cloud, out, and then back again. As always, some of the best minds in AWS were there to share their practices. Finally Amazon isn’t losing steam and continues expanding its numerous solutions for compute needs. It is clear that eventually no realm of technology is safe from their sights.

AWS is eating software

AWS’s secret sauce is the ability to deliver a well documented, API driven with UI service that delivers easy access to the technology stacks you want to run but don’t want to master. AWS announced some enhanced services as well as new ones. They have recently started into not only server side services like databases or on-demand servers but also business side services such as workstation proxies, email and now data analytic interfaces. Finally, AWS is better enabling audit and compliance as a service. Below is non-exhaustive list of some of the more interesting features they introduced.

Infrastructure Services

Infrastructure as a service is at the heart of all of Amazon Web Services offerings. As usual, AWS isn’t resting and is listening to customers to add the features desired.

  • Amazon Elasticsearch Service: Amazon is taking all the fun difficult parts out of Elasticsearch such as authentication, cluster discovery and ELK pipeline management. Along the way they are exposing many of their metric pipelines such as Amazon CloudWatch Logs, AWS CloudTrail and Amazon CloudWatch to easily pipe the data from the various silos into Elasticsearch.
  • Amazon Lambda Update: AWS Lambda is the purest form of cloud computing; on demand functional code execution without any infrastructure management. They introduced official support for Python, a CRON like invoke service for Lambda functions as well as the ability to run Lambda functions within a VPC. These are all really big deals as they clear out some of the remaining hurdles for Lambda adoption across all cloud designs.
  • EC2 Container Service Update: The biggest update here is that AWS finally has their own container register service. The registry is catch up on AWS’s part. Google had this feature for over a year now and I think not being able to easily tie in your private containers was hindering some adoption of the technology.
  • EC2 Instance Update: The message here was on two parts. First: HUGE instances the X1, 100+ cores with 2TB of memory. Now you can finally run your entire datacenter on one instance! Second: Nano instances, T2.nano with 512 MB of memory and 1 compute core. This instance is perfect for running that low traffic site that still needs a little backend processing.
  • Amazon RDS Update – MariaDB: This is a big deal for AWS as they continue to roll out “not owned by Oracle” RDB choices. Maria is to MySQL as Jenkins is to Hudson. The OSS community forked the Oracle aquired product and continue their march towards freedom. AWS is now letting you use it without hassle.
  • Amazon Kinesis Firehose: The Firehose provides a simple API endpoint to submit metrics to a variety of AWS services. It takes your payload and puts it into S3 or Redshift immediately. Amazon is very good about taking a common use case for several of their services where it may take a little code lifting to enable and then spin it into another service. This is basically a combination of Amazon API Gateway, AWS Lambda, Amazon S3, and Amazon Redshift. I can imagine that integration into Amazon Elasticsearch Service isn’t far behind.
  • EC2 Dedicated Hosts: If on demand dedicated virtual instances weren’t enough to satisfy your auditor AWS has now introduced bare metal dedicated hosts. This partially fits into the compliance and audit section but is still infrastructure.

Business Services

Sometimes the hardest part of using the cloud is the journey there. Amazon is working to make this easier.

  • AWS Import/Export Snowball: The Snowball is a physical device that AWS sends to you via FedEx to copy approximately 50TB of data per device and mail to back to AWS for them to load into S3. The saying, “Never underestimate the bandwidth of the delivery truck” still rings true.
  • Amazon QuickSight: This is a big shot over the bow of businesses such as Adobe SiteCatalyst, Oracle Business Analytics, Salesforce and other statistic warehousing firms. The secret sauce to QuickSight is that they immediately integrate into nearly all the data producing and consuming services that AWS offers.

Compliance and Audit Services

Amazon is trying their best to give business zero excuses to be on the cloud. With their latest suite of compliance and audit services, they are further positioning themselves as the best in the business to partner with, get work done, and do it in a secured fashion.

  • Amazon Inspector: This service allows you to perform security scans on your infrastructure and run an agent on your instances to scan network, filesystem and process activity to ensure that best practices are being met or bring notice of those needed mediation.
  • AWS Config Rules: The config rules are Lambda functions that check the state of your infrastructure against rules you define as Lambda functions. The config rules can be launched via a configuration change (tracked by AWS Config) or a periodic trigger (Lambda CRON anyone?).

Breakout Session Highlights

One of the primary reasons to attend re:Invent is the breakout sessions and being able to talk to the creators and movers of the services. Even as the sessions slides and videos are freely available the best way to make time to learn and explore these sessions is to be there in person. I’ll make separate posts for my favorite sessions but here is the bullet list of those which I found interesting or insightful:

  • BDT209 Amazon Elasticsearch Service: Being that it was announced a week prior, Amazon Elasticsearch Service was a little absent from the conference but they did have a quick overview session to present the latest service. The key factors to consider for Amazon Elasticsearch Service is that they have combined many of the features of the Elasticsearch, Logstash and Kibana (ELK) stack to make it easy to integrate into Amazon. Session video: [AWS re:Invent 2015 (BDT209) New! Amazon Elasticsearch Service for Real-time Analytics](https://www.youtube.com/watch?v=s7dJESec_dY) and slides: (BDT209) Launch: Amazon Elasticsearch For Real-Time Data Analytics.
  • SPOT304 Faster, Cheaper, Safer Products with AWS: Adrian Cockcroft Shares Experiences Helping Customers Move to the Cloud: The great cloud thinker Adrian Cockcroft of Battery Ventures dives into best practices and patterns that he evangelizes and sees emerging from his consulting experiences. Session video: [AWS re:Invent 2015 (SPOT304) How Adrian Cockcroft Helped Move Customers to AWS](https://www.youtube.com/watch?v=LMYYJuh9t70) and slides (SPOT304) Faster, Cheaper, Safer Products with AWS.
  • CMP407 Lambda as Cron: Scheduling Invocations in AWS Lambda: Granted this session lost a little luster as AWS launched Lambda Scheduled Events but it was still nice to see the creativity of folks when it came to invoking Lambda functions on a periodic basis. The TLDR; version is they used a CloudWatch alarm notification on a metric that would be either zero, one or unavailable. The notification was sent to a notification topic which triggered the Lambda function every minute (the evaluation time of the CloudWatch alarm). The function would then submit a one or zero to the metric ensuring that another alarm notification would trigger another minute later. Session video: [AWS re:Invent 2015 (CMP407) Lambda as Cron: Scheduling Invocations in AWS Lambda](https://www.youtube.com/watch?v=FhJxTIq81AU) and slides: (CMP407) Lambda as Cron: Scheduling Invocations in AWS Lambda.
  • SPOT210 Zynga’s Journey (Back) to the AWS Cloud: Zynga’s CIO shared his experience of migrating both away and back to the cloud. If you were unaware Zynga famously moved out of AWS in the 2011-2012 timeline however they are migrating back fully. The key problems they ran into was maintaining up-to-date hardware and skill sets after the initial datacenter build was completed. Most people interested in building the datacenter weren’t interested in the slow maintenance grind associated with actually running it. It turned out as well that AWS made it easier to track the cost of projects than it was in a datacenter simply because each team could have a sub-account of a master billing account. Zynga learned in a very roundabout way to value of cloud. Session video: [AWS re:Invent 2015 (SPOT210) Zynga’s Journey (Back) to the AWS Cloud](https://www.youtube.com/watch?v=BZfhhGvY3c0) and slides: (SPOT210) Zynga’s Journey (Back) to the AWS Cloud.
  • DVO305 Turbocharge Your Continuous Deployment Pipeline with Containers: Amazon had two of their own on hand to demonstrate a tightly integrated, repeatable deployment pipeline using AWS CodePipline, Jenkins and Amazon ECS. They demonstrated some great best practices such as naming the containers with a build ID for easy rollback and how to use multiple CodePipeline transition points to build, test and deploy to production a fully configured container. Session video: [AWS re:Invent 2015 (DVO305) Turbocharge Your Continuous Deployment Pipeline with Containers](https://www.youtube.com/watch?v=o4w8opVCI-Q) and slides: (DVO305) Turbocharge YContinuous Deployment Pipeline with Containers.
  • ARC309 From Monolithic to Microservices: Evolving Architecture Patterns in the Cloud: This was one of my favorite breakouts this year because it began with Adrian Trenaman, SVP, engineering of Gilt Groupe, giving a real world example of how the company Gilt slowly migrated from monolithic services to mainly using microservices. The session finished up having Derek Chiles, a manager in solutions architecture for AWS, highlight some great microservice patterns that AWS and Gilt developed together. Session video: [AWS re:Invent 2015 (ARC309) Microservices: Evolving Architecture Patterns in the Cloud](https://www.youtube.com/watch?v=C4c0pkY4NgQ) and slides: (ARC309) Getting to Microservices: Cloud Architecture Patterns.
  • SEC314-R AWS Config/Config Rules: Use AWS Config Rules to Improve Governance over Configuration Changes to Your Resources: AWS Config and Config rules is like AWS’s answer to Netflix’s Edda and Conformity Monkey tools. I love these tools (both AWS and Netflix’s offerings) because you can enforce convention and track change and relationships of your infrastructure even after it is gone. These records can really save your hide if for instance you accidentally deleted your continuous integration server which had an old kernel boot image that you have no idea which id it was. If you have no idea what I just said, you might want to enable AWS Config so that on that dark day when you need to know, it’ll tell you! Config Rules are the latest example of how AWS takes two services, Lambda and AWS Config in this case, and makes a great combination to better enable the consumer to just focus on what they need to get their job done. Session video: [AWS re:Invent 2015 (SEC314-R) New! AWS Config Rules: Improve Governance Over Configuration Changes](https://www.youtube.com/watch?v=sGUQFEZWkho) and slides: (SEC314) AWS for the Enterprise: Implementing Policy, Governance & Security.

Conclusion

AWS re:Invent 2015 was an excellent showcase and exploration of what is happening in the cloud.

Continuous Deployment is the maturation of DevOps. I am looking forward to exploring how CD methods and as well new microservice patterns will help drive forward iterative search platform enhancements. Please get in touch to talk about how OSC can help with your search application environment.