About C2i Genomics
C2i Genomics, now part of Veracyte, empowers physicians, patients, and researchers with ultra-sensitive, personalized, WGS-based MRD (Minimal Residual Disease) solution for cancer patient monitoring.
The company’s proprietary technology, implemented as a SaaS solution utilizes a cloud-based platform to perform cancer tumor burden monitoring on a global scale.
The Challenge
As C2i Genomics continued to expand, the product required highly performant and cost-effective storage, which would be used for sharing a significant volume of data between processing jobs.
The Solution
Embarking on a transformative journey with Automat-it meant collaborating on a migration project that built a robust infrastructure catering to its diverse customer base – with the AWS Premier Partner responsible for handing the critical tasks including design and implementation.
Automat-it performed a deep analysis of possible storage solutions, compare Amazon S3, Amazon FSx for NetApp ONTAP, Amazon EFS, Amazon FSx for Lustre, and found the best possible way to store, share, and process large files effectively:
- In each deployment, a single workload, performing a single sample analysis, spins dozens of Kubernetes nodes and hundreds of pods/jobs using Amazon Elastic Kubernetes Service (Amazon EKS)
- The applications utilize AWS Step Functions, Amazon EKS with Karpenter, and Amazon FSx for Lustre as shared persistent volumes, among other managed services from AWS
- Hundreds of gigabyte files are processed by multiple steps of the company’s analytical pipeline, either sequentially or in parallel, and the result is saved in an S3 bucket in the end
- ArgoCD is utilized to deploy micro services using a customized Helm chart, and Argo Workflow is used to create FSxL file systems ad hoc
- Dozens of environments are deployed, with new production deployments added monthly, while development and testing environments are continuously set up, updated, and scrubbed
Automat-it’s DevOps specialists were brought in for this situation, given their extensive experience in managing complex DevOps environments. As and when required, they were backed up by Automat-it’s CCoE (Cloud Centre of Excellence) experts.
Following the completion of this specific task, C2i Genomics moved all its DevOps tasks to the Automat-it team. Currently, four DevOps developers are committed to accomplishing all relevant tasks in C2i Genomics. This includes development, maintenance, deployment, security enhancements, and service updates.
The Results
- Previously, C2i Genomics’ software development team of 20 engineers and data scientists invested 30% of their time in DevOps activities. This is now negligible, and software developers can focus on the application’s features rather than the underlying infrastructure
- Hundreds of workflows can run in parallel in test and production environments. Each workflow works with large files (up to hundreds of gigabytes each). Large files are stored in temporary and dynamically created FSx for Lustre file systems, isolated for every workflow for security reasons
- The maximum performance was reached with FSx for Lustre and the processing job became 30% faster, with additional room for further turnaround time optimizations
- The large files remain in the same file system throughout the workflow, without the need to upload them to a persistent storage (s£) at the end of each processing step, and downloading them back to the temporary storage for the next step
- Reducing the duration of the pipeline steps leads to significant cost savings, because expensive EC2 instances are now not spending time on long downloads and uploads of large files
- Automat-it’s DevOps team constantly optimizes the development processes to match C2i Genomics’ rapidly evolving development plan
- DevOps knowledge sharing is secure within C2i Genomics by means of documentation, training sessions, and knowledge transfer
Boost Efficient Storage Today
Are you a startup like C2i Genomics looking to achieve more cost-effective and efficient storage? Automat-it’s AWS and DevOps expertise can make this possible.
