How to Deploy a Parallel Cluster

Parallel Cluster is an open-source management tool supported by Amazon Web Services (AWS). It helps users deploy and manage HPC clusters in the cloud. In today’s article, we will tell you how to deploy a Parallel Cluster. Here are the steps

Step 1: Install AWS Parallel Cluster 

The first step is to access the AWS Management Console. Type Cloud9 in the search bar and select it to choose the open IDE for “instance” set up. It will take a few minutes for the IDE to open. 

Cloud9 will stop and restart the instance. That way, you can avoid paying the compute charges if you are not using the Cloud9 IDE. Make sure you use the “PIP” install command during the process. Bear in mind that the Cloud9 environment already has Python and PIP installed. 

Step 2: Configure Parallel Cluster 

Use the “pCluster Configure” command to initiate the configuration process of Parallel Cluster. It is crucial to input the requested info, which usually includes AWS Region, EC2 Instance Type, and Scheduler.  

Generate a new key pair and get your networking information. Next, create an initial Parallel Cluster “config” file.” Use the command “cat ~/.parallelcluster/config” to check the file content. Once you have the configuration file, you can create a cluster with the minimum information. You can use a default configuration file if you want to test the cluster. 

Step 3: Create a Cluster Config 

After installing the Parallel Cluster and creating a default configuration, it is time to build a configuration file. The purpose is to create a simple, high-performance system. Remember, the system will generate this file in your home directory. Use the following settings to generate the cluster. 

Head-node and computer nodes: Although you can change the instance type, doing so will cause you to run into EC2 limits. That way, you won’t be able to create instances. Use Parallel Cluster 2.9 or above to get support for multiple instance types and queues. 

Utilize a Placement Group: It is crucial to use a placement group to keep instances together in one physical data center. That way, you can keep the data in a single availability zone, and its purpose is to increase the bandwidth and decrease latencies between instances. 

Attach the GP2 EBS to the Head Node: Use the GP2 Amazon EBS volume and attach it to the head-node. Next, share it through the NFS so that the compute nodes mount it on /shared. Experts at Clovertex recommends this location for storing your scripts and applications. Remember that the NFS shares the /home directory. 

Use SLURM: It is essential to use SLURM as a job scheduler. You can use the command “distable_hyperthreading = true” in the config file to disable Intel Hyper-Threading. So, now, you are ready to launch the cluster. 

Step 4: Build an HPC Cluster 

Creating an HPC cluster is usually based on the information or specifications mentioned in the config file. Use the “pCluster create” command to create a cluster. Likewise, use the –config command if you want to deploy another configuration file than the default one. 

Remember, it will take a few minutes to build your cluster. Monitor the creation status in the terminal. Once the process completes and your cluster is ready, you will see the result on the screen. 

Leave a Comment

Your email address will not be published. Required fields are marked *