Summary
The purpose of this model is to provide a sustainable, scalable, and highly accessible High-Performance Computing (HPC) environment for the South Dakota Board of Regents (SDBOR) research and educational community. To balance accessibility with resource management, the HPC clusters (Innovator and Discovery) utilize a tiered service structure:
- Educational Support: Free access is provided for coursework, workshops, and exploratory projects, with a combined usage limit of 800 Service Units (SU)s per user.
- Standard Research: Any activities exceeding the free allocation are supported by a usage-based "pay-as-you-go" model, billed at a baseline rate of $0.015 per SU for SUs from 801 to 50,000.
- Bulk Research: Tailored cost for > 50,000 SUs and charged at a rate of $0.0075 per SU.
- Data Storage: All users receive a base allocation of 100GB for their cluster home directories and 5TB of high-speed cluster scratch space for application processing at no cost.
Scope
At this time, the cost-recovery model applies exclusively to cluster computing and associated storage on the Discovery and Innovator HPC clusters.
Definitions
Service Unit (SU): A normalized measure of computational work consumed on the HPC cluster. SU calculations are based on CPU core hours, GPU hours (weighted by the GPU model), and runtime. In practice:
- 1 SU ≈ 1 CPU core-hour (baseline)
- GPU usage is converted to equivalent SUs based on the GPU model:
- A100: SU rate equivalent to $0.84/hr.
- H100: SU rate equivalent to $1.00/hr.
CPU Core-Hour: The use of a single physical CPU core for one hour of wall-clock time. Billing is based on the number of cores requested in the job submission script, regardless of actual CPU utilization percentage.
GPU Core-Hour: The use of a single Graphics Processing Unit (GPU) for one hour of wall-clock time. Due to the high demand and cost of these units, they are billed at a premium rate.
Wall-Clock Time: The actual elapsed time from the moment a job starts running until it completes. Users are billed for the duration the resources are occupied, even if the code finishes early or sits idle.
Completion-Date Billing: The method of accounting used, where the entire cost of a job is charged to the period (day, month, or year) in which the job finishes, regardless of when it started.
Account Provisioning and Billing Requirements
To facilitate automated tracking and data collection, all users are required to provide a valid index number prior to being granted access to the systems outlined here. While Tier 1 usage is fully subsidized at no cost, an active index number must remain on file for all users to seamless, automated transition into Tier 2 or storage overage billing should the free limits be reached. If you are unsure of your index number, please work with your shared services representative to obtain that information.
Coursework: If use of the HPC cluster(s) is needed for course work, the index number will be the name of the course, for example PS787, so the index would be PS787 when you fill out the onboarding form. This only applies to SDSU coursework at this time. Charge back for non-SDSU institutions is handled differently.
To submit a new/updated onboarding form, you can click this link: https://help.sdstate.edu/TDClient/2744/Portal/Requests/Service/53041/SDBOR-HPC-Project-Intake/Request
Compute Access Tiers
The HPC clusters utilize a three-tiered access model to balance accessibility with cost recovery. Each tier is designed to support different stages of the research lifecycle. Tier 1 provides a free monthly allocation of 800 SUs for coursework, workshops, and exploratory projects. Once the free tier is exhausted, users automatically transition to a usage-based "pay-as-you-go" model for production research, with Usage up to 50,000 SUs. To provide for budget-friendly predictability, usage exceeding tier 2 limits will be billed at a reduced rate of $0.0075 per SU.
All users start each month at Tier 1. Once the 800 SU free allocation is exhausted, the system bills at the standard rate ($0.015). For the heaviest users, Level 3 acts as a "safety valve," reducing the rate by 50% for all units consumed beyond 50,000 SUs to prevent extreme billing spikes.
|
|
|
|
|
|
|
|
|
Teaching, debugging, and small pilot projects.
|
|
|
|
801 – 50,000 SUs per month
|
Standard production research.
|
|
|
|
Usage exceeding 50,000 SUs per month
|
Large-scale sims and resource heavy research.
|
Storage Cost Recovery Model
All users get 100GB of home folder space that cannot be expanded, and 5TB of /scratch space, and anything above that is charged at a rate of $0.01/GB/Month.*
|
|
|
|
|
|
|
|
|
|
|
|
|
|
High-speed temporary workspace.
|
Storage Overage (/scratch)
|
|
|
Applies to data exceeding free scratch limits.
|
*Enterprise storage and Linux/Unix file systems calculate capacity using binary notation (base-2) rather than standard decimal (base-10). While system interfaces frequently use "GB" and "TB" as shorthand labels for user familiarity, underlying storage provisioning is calculated at 1 TiB = 1,024 GiB. Consequently, a 5TB allocation is provisioned at 5120GB rather than 5,000GB.
Tools
jobstats
Jobstats is used to view the efficiency of your jobs. By using this tool you will be able to look at an after action report of your run to see just how much of the resources you requested you actually used. Jobs are billed by how much you request and not by how much you use. Therefore a tool like jobstats can really help in making you better at requesting what you need and saving you money in the long run. (note jobstats can be run on active jobs but is most accurate when used after completion. GPU can only be viewed upon completion)
Usage: jobstats jobid
hpcchargeback
hpcchargeback is a tool that can be run from the login node of the cluster to view your current usage for the month as well as previous usage for past months, in order to estimate future costs as well view current costs for the month. You can use hpcchargeback -h to view the full list of options that are listed below.
Usage: hpcchargeback [OPTIONS]
Options:
-s Include cost for current storage usage (will be current usage even for previous months)
--current (or no args) Run for the current month (1st to today)
--prev Run for the entire previous month
--dates YYYY-MM-DD YYYY-MM-DD
Run for a custom start and end date
Questions/Concerns
We recognize that transitioning to a cost recovery model requires careful planning, and you may have questions regarding upcoming grant proposals, unique compute workloads, or specific workflow exceptions. We are fully committed to working with individual labs to ensure your active research continues without disruption.
- Policy and Billing Inquiries: For questions regarding financial tracking, obtaining index numbers, and other billing related questions please contact your shared services representative or the Division of Budget and Finance.
- Technical and Account Assistance: If you need assistance checking your current monthly SU utilization, modifying jobs scripts to better optimize your workflow, or updating/changing index number, please contact the Research Computing Team at sdsu.hpc@sdstate.edu.
- Stray Process Cleanup: Scripts are ran on all cluster nodes to remove stray processes once jobs are completed. Doing so ensures the nodes are free and available for use when jobs start and not competing with rogue/stray processes from other users.
Frequently Asked Questions
How are GPU charges calculated compared to CPUs?
While 1 CPU core-hour equals 1 SU, GPUs are weighted higher due to their performance and cost. An NVIDIA A100 consumes 56 SUs per hour, and an NVIDIA H100 consumes 66.6 SUs per hour.
What if my job finishes early? Am I still billed for the full requested time?
Billing is based on Wall-Clock Time, which is the actual elapsed time from the start to the completion of the job. However, you are billed for the number of cores/GPUs requested in your script, regardless of whether your code utilizes 100% of those resources.
Is my data in /scratch safe and free?
You receive 5TB of /scratch space at no cost. However, anything exceeding 5TB is billed at $0.01/GB per month. Note that while the cluster /home folder (100GB limit) is for permanent files and is backed up daily, the cluster /scratch folder is intended as a high-speed temporary workspace that is NOT backed up.
How was the "Service Unit" (SU) rate of $0.015 determined?
The rate is a normalized measure of computational work based on CPU core hours and weighted GPU models. It is designed to facilitate cost recovery for cluster operations, based on other charges from similar institutions and HPC centers.
How do we justify the "Free" Tier 1 allocation?
Tier 1 is an institutional investment in "Educational Support." By providing 800 SUs per month at no cost, the university lowers the barrier for coursework, workshops, and pilot projects that do not yet have external grant funding.
How is the "Service Unit" (SU) rate adjusted over time?
The $0.015/SU baseline is subject to an annual review. Adjustments may be made based on fluctuating utility costs (power/cooling), changes in vendor maintenance contract pricing, or institutional subsidy levels.
Is there a purge policy for the 5TB of free scratch space?
To ensure the 5TB free tier remains "high-speed temporary workspace," a purge policy will be enforced. Files in /scratch not accessed for 12 months are subject to removal to prevent the file system from reaching capacity.
Can I "bank" or carry over unused Tier 1 SUs to the next month?
No. The 800 SU allocation is a "use it or lose it" monthly allowance that resets on the 1st of every month. This action ensures that the "Free" tier remains available for new educational and research exploratory projects.
What happens if my storage exceeds 5TB mid-month?
Storage overages are calculated based on monthly Usage. If your /scratch usage exceeds 5TiB, you will be billed $0.01/GiB/month for the excess. A monthly average will be implemented should "gaming" of storage occur.
Can I request a temporary increase to my Tier 1 free allocation?
No. To maintain a sustainable environment, the 800 SU limit is fixed and resets monthly. Projects requiring more than 800 SUs per month are considered production-level and will transition to Tier 2 for billing.
Is SDSU still subsidizing the HPC?
Yes, at this rate, SDSU continues to subsidize HPC use. The rates charged for the HPC currently reflect about 40% of total costs (note: total costs also exclude depreciation of any federally funded equipment).