Building Private Workflows Using AWS HealthOmics
Executive Summary
This case study describes details of the work done by Clovertex in deploying a publicly available genomics analysis pipeline using AWS HealthOmics as a Private Workflow. The blog outlines the steps taken to successfully execute the analysis pipeline using AWS HealthOmics, the changes made to the public code, and benefits of HealthOmics.
About AWS HealthOmics
About wf-human-variation workflow from Oxford Nanopore
Results
Lessons Learned
- New workflow needs to be created in HealthOmics using the workflow zip and workflow parameters
- The code change needs to account for Docker containers in ECR
- Build parameters to account for ALL the parameters needed for workflow
- HealthOmics infrastructure are built securely and limits access to external sources
- HealthOmics can take inputs from S3 or OMICS datastores and can store results back to S3.
Business Benefits
- Ideal for handling production and frequently repeated workloads
- Simplified migration process with focus on challenging optimization tasks
- Requires minimal adjustments when modifying workflows
- Build and use a system that is HIPAA eligible, secure and scalable
- Able to collaborate more effectively with other scientists
Methodology
The goal of this experiment was to understand how to enable a private workflow on AWS HealthOmics. Most of the work was done using AWS console. The methodology consisted of two steps. Building a workflow using the artifacts from the wf-human-variation git repository with appropriate modifications to the code. And running a set of workflows using the workflow that was created. The workflow was also run directly using EC2 instance where Nextflow was deployed, for comparison.
Workflow Creation
- Creating an S3 bucket for uploading the artifacts and demo datasets for running the workflow
- Creating and pushing the docker images needed for the workflow to ECR
- Modifying the permissions for the user to access the ECR images
- Modifying appropriate modules in the wf-human-variation workflow artifacts and uploading the compressed file to an S3 location
- Creating the private workflow through AWS console
- Creating a run and executing it
- Fix the issues, update the workflow artifacts and repeat.
Ready to see AWS HealthOmics in action?
