Case Studies

Challenge

Design an infrastructure with high availability, scalability and a high degree of security. A key component of the implementation was to include a disaster recovery environment, continually tested for low Recovery Point Objective (RPO) and Recovery Time Objective (RTO) recoverability.

AWS and Partner Solution

Tricolor was brought on very early to consult on the project and subsequently recommended Amazon Web Services (AWS) as the platform of choice as it met the critical high availability, scalability and security requirements.

After careful analysis, architecture was designed together with a plan of the deployment phases that included risk identification and mitigation. The plan called for the creation of a staging environment for Proof of Concept (PoC), simulating all system requirements.

AWS Virtual Private Cloud (VPC) is an excellent platform providing the required security for this workload, using Amazon Elastic Compute Cloud (EC2) instances for its application and supporting servers. Within AWS EC2, Tricolor used Elastic Load Balancing (ELB) to automatically distribute incoming traffic between multiple AWS EC2 instances - these EC2 instances form part of Auto-Scaling Groups (ASG) providing the required scalability and fault tolerance. The design also usedmultiple Availability Zones (AZ) for additional protection and redundancy of all systems. Tricolor also used Amazon Simple Storage Service (S3) to store backups and Amazon CloudWatch to proactively monitor its EC2 instances through the AWS Management Console and the alerting capability.

Tricolorused Amazon Relational Database Service (RDS) for its workload database platform. Tricolor chose to use AWS RDS for all the advantages offered by a fully managed database platform. Using RDS removes the need to perform typical administrative tasks as well as the complexities of managing a fault tolerant deployment. Tricolor used the RDS service in a Multi-Availability Zone (Multi-AZ) configuration for its redundancy and near-instant failover capability.

Tricolorleveraged the disparate AWS regions for disaster recovery with the Production systems residing in one region while the Disaster Recovery systems reside in a different region. In addition, consistency of environments was maintained across regions through the use of AWS CloudFormation templates that define the VPC configuration and resources required.

The PoC was followed by a beta period during which several locations were migrated to AWS and operated there in live production. With that success, all remaining centers were subsequently migrated to the AWS platform. During this project the Tricolor team worked closely with the client team guiding them through the process and supporting all manner of requests along the way.

Results/Benefits

The transition of all the on premise system to AWS was a complete success. Client has already benefited from the ability to quickly resize systems as needs change and this will continue to be beneficial as adoption of this system grows.

Tricolor continues to provide design and consulting services as the business continues to grow with more airports, additional application features and new services.

One Healthcare is a cloud-based software platform for the senior care market. Its goal is to help healthcare providers meet the challenges of senior care by enabling them to achieve the business results that matter – enriching the lives of their residents, improving financial and operational health, and mitigating risk. OHC has developed an Analytics platform that aggregates data from various data sources and provides its clients with insights by means of reporting and analytics tools. The platform is built using technologies such as Cloudera Hadoop for processing unstructured and large data sets, ETL for loading and transforming data coming from various SQL sources, PostgreSQL for data warehousing and data marts, RabbitMQ for data ingestion, and Tableau for BI & Analytics.

Customer Challenge

Due to the nature of data ingested into the platform - patient data collected from medical institutions around North America, deployment in AWS needed to comply with the HIPAA certification requirements. To achieve this, particular constraints on how AWS technologies are used had to be followed and OHC decided to work with Tricolor as an implementation partner to ensure this need was handled properly. Tricolor is an AWS Consulting Partner with experience in implementing environments that have to meet PCI, HIPAA and other compliance requirements.

OHC decided to adopt a Development Operations (DevOps) culture internally and a significant effort has been put into implementing deployment automation tools using Chef, an open source software tool. The expectation was that the same tool chain would be leveraged for the production deployment in AWS, while still conforming to policies like separation of duties and other HIPAA requirements.

AWS and Partner Solution

To address the specific constraints required for HIPAA compliance, AWS provides a range of capabilities, including: dedicated hosts, secure networking, VPN and VPC peering, secure and encrypted S3 storage, server side EBS volume encryption, SSL load balancing, HSM-backed key management service, private managed DNS, etc. Tricolor was able to design and implement a secure private network for hosting the infrastructure running the Analytics platform. As well, Tricolor was able to isolate all required services inside the network, while providing a secure connection with the main OHC production data center, to enable the data loading and ingestion processes as well as SSO with the rest of the OHC web application services.

To support the DevOps processes, Tricolor worked with the OHC development and build & integration teams to productize the deployment tool chain, as well as implement a continuous delivery pipeline which supports the separation of duties policies, while still allowing collaboration between the pipeline and application developers and the operations engineers. The pipeline implements a discovery mechanism for configuring the overall stack, appropriate secrets management, automated DNS management, etc. It also enables integration with the OHC release process, allowing the development team to iterate fast and push more frequent releases on their end, while allowing the operations team to take responsibility for the final validation and update of the production environment.

Some of the challenges that the Tricolor and OHC teams had to overcome in the production process of the delivery pipeline include:

Implementing a multi- availability zone (AZ) fault-tolerant and self-healing network and network services like NAT gateways, host-to-site and site-to-site VPN, etc.

Implementing a one-way only connectivity from the production network to shared resources outside that network for deploying cookbooks and packages.

Implementing a private and fully managed DNS (with reverse host lookup support) using Route 53, while sharing it across VPN for use by the integration services.

Managing credentials and other secrets required by the Chef cookbooks to configure the software without exposing those secrets outside the production environment.

Separating environment-specific and application configuration and ensuring that the discovery mechanism can handle the former properly between environments.

Providing secure and reliable delivery of keys, SSL key stores, etc. to all of the nodes to ensure appropriate in-transit encryption is used across all services.

Results & Benefits

By leveraging AWS, OHC was able to meet the HIPAA requirements fully and to ensure the safety and security of the patient data processed by the Analytics platform. As use of the platform becomes more frequent and more data is brought into it, the organization also is able to scale the platform to handle the high processing and low latency data replication demand of the Hadoop-based ingestion service. This is done by using an appropriate combination of: compute and I/O intensive instances running on dedicated hardware put close together in placement groups; the high query load on the Redshift data warehouse, by using provisioned IOPS and enhanced networking; and bandwidth needs for the data modeling and analytics operations performed by Tableau, by optimizing the instance types for running the Tableau and Analytics application servers.

In addition to the benefits on the infrastructure side, OHC was able to take advantage of various managed services and tools to reduce the operations cost and improve the automation of the deployment. These include VPC, S3, IAM, Route 53, ELB, AutoScaling and CloudFormation. By combining the AWS tools with Chef, PointClickCare was able to accomplish a high degree of automation and implement a robust continuous delivery pipeline used both by development and operations in a true DevOps fashion.

Working with OHC has been a great opportunity, as it is an agile company, which has embraced DevOps principles and is clear in its vision to extend those principles to work in a SaaS platform model under heavy compliance and IT best practice constraints.

The Challenge

SoFi wanted to create a new type of advertising campaign for customers during the Oscar 2015 TV show. The challenge at hand was how do you create and maintain over 200,000 concurrenthttp requeststo Website during the big television event to their website.

Why Tricolor

Enlisting Tricolor for their Oscar (2015) Advertising Campaign, the overall project was based around show during which data is transferred in and out of using AWS.

Prior to the show, SoFi believed there would be approximately 10,000,000 watching the Oscar TV show and 200,000 accessing the website. This was a very ambitious project which required not only developers to write the code needed but also infrastructure specialists that could help to build a scalable and elastic infrastructure that could handle a super-sized load. SoFi leveraged Tricolor for support of the necessary infrastructure needed for the big show.

SoFi has AWS environment, and they utilize AWS CloudFront (Amazon’s Content Delivery Network.

The Solution

Tricolor’s involvement in this project was to make sure that the environment could scale and handle a super-sized load at once. Due to the Oscar being such a highly watched live event, the tremendous flash of traffic from one source arrives and then disappears much quicker than normal television content. The team at Tricolor worked with SoFi developers to model this traffic load in the build up to the big game.

The total requests per second that Tricolor and SoFi tested and built this solution for was 400,000/second all being driven through CloudFront. In order to test for this, Tricolor performed a significant amount of load tests on a number of occasions, with each one increasing up to the event. To test durability, Tricolor tried to break the system and was successful in making 800,000 concurrent requests per second.

In order to perform these load tests, Tricolor used a variety of tools. The SoFi team also provided helpto simulate these load tests, which created a very realistic experience for what the Websiteload might look like during the live TV Oscar show.

As a part of this project, Tricolor also contacted and worked with partners at Amazon to not only make them aware of the project, but also have them monitor traffic internally to view testing and production in real time. With over 500,000 connections per second success, Amazon uses this case study as an example of just what their cloud infrastructure can accomplish.

Results

Utilized Cloudfront to scale an environment that could withstand heavy traffic

Built models to test incoming traffic prior to the Oscar

Acted as Lead AWS Solutions Architects