How Slurm Schedules Jobs

A description of how the Slurm schedules jobs on the cluster and decides which jobs to run where and when.

 

Fairshare Scheduling on SDSU HPC Clusters

To ensure efficient and fair access to computational resources for all users, SDSU's High-Performance Computing (HPC) cluster employs a fairshare-based job scheduling system using the Slurm workload manager. This approach helps balance usage across all research groups and users, ensuring that no single group dominates the cluster for extended periods of time.

 

How Fairshare Scheduling Works

Slurm keeps track of resource usage history and uses it to adjust job priorities dynamically. If a research group has recently consumed a large share of the system’s resources, the priority of that group’s pending jobs will be reduced. Conversely, if a group has been using fewer resources, its pending jobs will be prioritized to bring usage back toward a fair target.

In addition to fairshare, other factors influence the scheduling of jobs:

• Queue Time (AGE) – Jobs that have been waiting longer get a higher priority.

• Job Size – Larger jobs requesting more resources may get prioritized to ensure they can be scheduled efficiently.

• Requested Walltime – Jobs with shorter requested runtimes are often easier to schedule and may backfill gaps in the schedule.

 

Viewing Job Priority Components

You can inspect the factors affecting your job’s scheduling priority using:

This command shows a breakdown of three main components:

1. AGE:

   • Priority increases the longer a job waits in the queue.

   • Only a limited number of jobs per user or group receive this benefit at a time.

2. JOBSIZE:

   • Larger jobs (more CPUs, memory, GPUs) receive more priority.

   • Helps ensure large-scale simulations and analyses can be scheduled effectively.

3. FAIRSHARE:

   • Adjusted based on recent usage by your group.

   • Uses a weighted system to normalize resource types:

     • For example:

       • 1 hour on a V100 GPU ≈ 88 CPU·hours

       • 1 hour using 602 GB of memory ≈ 88 CPU·hours

   • Fairshare impact decays over time—recent usage affects your priority more than older usage.

 

Backfill Scheduling

The Slurm scheduler also uses a technique called backfilling, where smaller jobs are run in the time gaps between larger reserved jobs. This keeps resources fully utilized. To help with this:

• Provide accurate walltime estimates when submitting jobs.

• Shorter and smaller jobs with accurate time estimates are more likely to be backfilled quickly

 

Final Notes

Fairshare scheduling does not guarantee that jobs will run immediately, but it ensures a balanced workload across all SDSU HPC users. If you notice persistent delays or have questions about scheduling behavior, Please reach out to us by submitting a ticket at help.sdstate.edu