Biomedical Informatics High Performance Computing Cluster
Credit: Kevin Ernst, Matthew Weirauch, and Leah Kottyan for compiling and hosting much of this information not found on the DBS HPC website.
We have access to a RedHat 9 Linux based high performance computing (HPC) cluster computational cluster through the Data Management and Analysis Collaborative (DMAC). To request access to the cluster, use the linked web form, or send an email to Cluster Support: help-cluster@bmi.cchmc.org. After your access is enabled, you should be able to login to the cluster with your CCHMC employee network credentials.
Up to date policies are described here. Briefly, all users have a default disk (home directory) quota of 100 GB and a job walltime quota of 10000 hours per quarter, which is free. The quota of your home directory is fixed, and cannot be changed. Your home directory is accessible at the path /users/<userid> and has a default 100 GB quota. Email Cluster Support to request an increase in wall time, as described.
You also have access to a shared scratch volume for storing temporary job data only, located at /scratch/<userid>. The up-to-date policy for the scratch area is described here. In brief:
- A user’s scratch (
/scratch/<userid>) directory has a baseline limit of 500GB.
- A user can burst over this 500GB limit for up to 14 days to a maximum of 5TB.
- If a user remains over 500GB past 14 days then additional data writes are denied.
- A user can reset their burst usage clock by reducing usage under 500GB for 10 minutes.
- Data older than 60 days are cleaned every day. Be sure to move your data off this space as soon as your job completes.
Users typically do not have (read/write) access to other users’ home/scratch directories. Thus, /data/hpc-troutman/ is the place to store any data or analyses that anyone else in the lab might ever need to work with. This space is not for storing personal files.
Accessing the HPC cluster
Connect to the CCHMC cluster by following the BMI instructions here and here, or follow the examples below.
If you are in the CCHMC network or connected to CCHMC VPN:
ssh [yourusername]@bmiclusterp.chmcres.cchmc.orgIf you are outside of CCHMC network, connect to ssh.research.cchmc.org then to bmiclusterp.chmcres.cchmc.org:
ssh [yourusername]@ssh.research.cchmc.orgssh [yourusername]@bmiclusterp.chmcres.cchmc.orgAlternatively, connect through the OnDemand Portal.
Running jobs on the HPC cluster
The “head nodes” bmiclusterp1 and bmiclusterp2 are not intended to be used for computationally-intensive tasks; basic file management is about it. The head nodes are where everyone logs in and submits their jobs, so if you were to submit computationally heavy tasks, you could prevent others from logging in or submitting their own jobs to the actual cluster nodes. Processes run from the cluster head node directly (i.e., not submitted using bsub as shown below) are limited to 75% of one CPU and 10% of total system RAM (currently at 10 GB, so 1 GB) and will be terminated if they go over.
What is a “job”? A job is basically a Bash shell script that runs computationally intensive program of your making on the cluster.
What is a cluster? A cluster is a form of supercomputer, which is made up of individual servers or nodes. They share access to a large pool of disk storage through NAS (Network Attached Storage). Our server uses IBM’s “Platform LSF” workload management platform, which means, among other things, that there is a scheduler for using the cluster. You submit batch jobs (programs) to the scheduler, and it starts your jobs as soon as resources are available.
The BMI cluster offers several advantages:
- The cluster has big computer capabilities. The largest nodes have 512 GB RAM and >16 cores.
- Linux environment with many pre-installed sofware, called Modules. Refer to the BMI modules page for more information and examples.
- The cluster allows you to do pseudo-parallelism. You can break a problem into many pieces and run many jobs in a distributed fashion.
Scheduling computation jobs with LSF
The cluster batch system is managed by the LSF resource manager and scheduler. You can find some examples of submitting jobs using LSF here. Please read the job scheduling policies page for recent policy changes.
Essentially, users submit jobs to the server using the bsub command. The current state of the queue in the server can be viewed using bjobs. There are a host of other utilities that can be used by Torque users like: bkill, bmod, bstop, bmig, bresume etc. bsub can be used for batch as well as interactive submission of jobs. Interactive job submission should be used only when a user needs to run and debug his code and for short-duration jobs. In either case, you need to submit your job to a cluster node so that you can do your testing without burdening the head node.
To directly submit a script, use:
bsub -W 4:00 -n 1 -M 10000 < <script>To request an interactive session (12 hours, 16 cores, 32 GB ram):
bsub -W 12:00 -n 16 -M 32000 -Is /bin/bashJobs can be viewed using bjobs, or for more information, use bjobs -l <jobid>.
Summarize remaining wall time for all jobs using:
bjobs -o "JOBID STAT RUN_TIME TIME_LEFT" -noheaderThis displays run time in seconds and time left like 26:2L, which in the example below is in hours.
5926570 RUN 165453 second(s) 26:2 L
5929410 PEND 0 second(s) -More time can be added using bmod -W H:MM <jobid>. Or in a loop:
for i in <jobid1> <jobid2> <etc>; do bmod -W <H>:<MM> $i; doneKill a job with bkill <jobid>.
More information is available here.
Checking your cluster balance
By default, you have 10,000 hours of “wall time” (real-world computation time for the jobs you run) on the cluster, and this quota resets quarterly (ref).
BMI uses an accounting system named “Gold” to keep track of how many wall hours you have remaining; you need to load a module named gold to use this: - load the ‘gold’ module, which provides the ‘gbalance’ command - you should be on a cluster node (e.g., bmiclusterp2) when you do this.
module load gold
gbalance -h -u \$USERleave off the ‘-h’ if for some reason you want to see the output in seconds of wall time instead of the more human-readable hours
The string $USER is an environment variable that always refers to the currently-logged in user (you!). You can replace $USER with some other username if you want to spy on someone else’s cluster usage.
The columns in the output of gbalance are:
| Column Name | Meaning |
|---|---|
| Id | Unique user identifier |
| Name | Your username |
| Amount | Wall time requested for currently running jobs |
| Reserved | Wall time requested for currently pending jobs |
| Balance | Total wall time currently requested (sum of amount and reserved) |
| CreditLimit | Amount of wall time credited each quarter |
| Available | Wall time remaining after current requests |
See also:
Requesting a cluster account
Cluster accounts are now requested using a web form. The older email system below is out dated. It may still be useful if the form is broken.
Here is a boilerplate request for a BMI HPC cluster account that you can update with your own information:
To: help@bmi.cchmc.org
Subject: New HPC cluster account
Hi everyone,
May I please have an HPC cluster account? My username is usr0xy (replace with your actual username).
Please also grant me the appropriate roles so that I can use the BMI HTTP proxy to download scientific data to the cluster.
Thanks very much.
--Your NameMake sure to say “please” and “thank you,” and include your Children’s username, which is something like usr0xy (three letters of your last name, then three random characters). Also include with your request that you want BMI proxy access, and mention that they can look at account ern6xv if they need a template.