Roy Illsley, principal analyst at Ovum reviews recent open source container announcements from AWS.
Managing the rise in containers and being able to orchestrate them across different environments has become a challenge for organizations. AWS announced Blox in late 2016, a collection of open source projects looking at the challenges of managing containers, orchestrating containers, and enabling local development of containers.
Blox provides a framework that enables administrators to build custom tooling on top of Amazon ECS. One of the first such tools is custom built job schedulers or the ability to integrate third-party schedulers, while using Amazon ECS to fully manage and scale the organizations clusters. The framework makes it easy to consume events from Amazon ECS, store the cluster state locally, and query the local data store though APIs. The Blox project currently consists of two components:
Managing change in a VM is difficult enough, but as the granularity deepens with containers this magnifies the challenge. AWS have developed a cluster-state-service, which collects all the change events that occur in the container or instance across the Amazon ECS cluster. These change events are a stream of events that are stored in a local data store. The cluster-state-service then through APIs (such as search, list, filter, etc.) allows this data to be queried. Using this information an administrator can quickly obtain a view of the current state of the Amazon ECS cluster so that organizations can respond to changes in real-time. The cluster-state-service tracks the Amazon ECS cluster state locally, and manages any drift in state by periodically reconciling state with Amazon ECS.
The ability to automatically manage the execution of one task per host across the entire Amazon ECS cluster is needed if administrators are to become automated in terms of managing these tasks. For example, with the daemon-scheduler a monitoring job can be run and as new nodes are added to the cluster it can consult with the cluster-state-service and launch new tasks. The scheduler can be used as a reference for how to use the cluster-state-service to build custom scheduling logic, and AWS plan to add additional scheduling capabilities for different use cases.
Blox can be deployed locally or as an AWS deployment. Locally Blox uses the Docker Compose file, which launches the Blox components, cluster-state-service, and daemon-scheduler, along with a backing state store, etc. The AWS deployment is done via an AWS CloudFormation template to launch the Blox stack. The AWS deployed Blox stack makes use of AWS services designed to provide a secure public facing scheduler endpoint. The AWS CloudFormation template in AWS sets up a stack with the components:
An Amazon SQS queue and Amazon CloudWatch configured to deliver ECS events to the SQS queue. An Application Load Balancer (ALB) is created in front of the scheduler endpoint and an Amazon API Gateway endpoint is set up as the public facing frontend. This provides an authentication mechanism for the Blox stack with the API Gateway endpoint used to reach the scheduler and manage tasks on the ECS cluster. Finally, an AWS Lambda function can act as a simple proxy which enables the public facing API Gateway endpoint to forward requests onto the ALB listener in the VPC.