• 1. Introduction
• 2. HPCToolkit Overview
  • 2.1. Call Path Profiling
  • 2.2. Recovering Program Structure
  • 2.3. Attributing Performance to Code
  • 2.4. Interactive Performance Analysis
• 3. Quick Start
  • 3.1. Guided Tour
    • 3.1.1. Compiling an Application
    • 3.1.2. Measuring Application Performance
      • 3.1.2.1. Specifying CPU Sample Sources
      • 3.1.2.2. Measuring GPU Computations
    • 3.1.3. Recovering Program Structure
      • 3.1.3.1. Caching Structure Results
    • 3.1.4. Attributing Performance to Code
    • 3.1.5. Interactive Performance Analysis
    • 3.1.6. Effective Analysis Strategies
  • 3.2. Additional Guidance
• 4. Installing with Spack
  • 4.1. Config Files
    • 4.1.1. Config.yaml
    • 4.1.2. Modules.yaml
  • 4.2. Installing HPCToolkit
    • 4.2.1. Basic Installation
    • 4.2.2. Configuration Options
      • 4.2.2.1. CUDA (+cuda)
      • 4.2.2.2. Level Zero (+level_zero)
      • 4.2.2.3. ROCm (+rocm)
      • 4.2.2.4. OpenCL (+opencl)
      • 4.2.2.5. MPI (+mpi)
      • 4.2.2.6. PAPI (+papi)
      • 4.2.2.7. Python (+python)
  • 4.3. Installing Hpcviewer
  • 4.4. Spack for Beginners
• 5. Meson for Developers
  • 5.1. Quickstart
  • 5.2. Configuration
  • 5.3. Installing Dependencies without Root
    • 5.3.1. Meson Wraps
    • 5.3.2. Dev Containers (BETA)
  • 5.4. Installing Dependencies as Root
    • 5.4.1. Debian/Ubuntu and Derivatives
    • 5.4.2. Fedora/RHEL and Derivatives
    • 5.4.3. SUSE Leap/SLES 15 and Derivatives
  • 5.5. Custom Dependencies
  • 5.6. Links to Meson Documentation
• 6. Monitoring Applications
  • 6.1. Using hpcrun
    • 6.1.1. If hpcrun causes your application to fail
      • 6.1.1.1. hpcrun causes failures related to loading or using shared libraries
      • 6.1.1.2. hpcrun causes your application to fail when gprof instrumentation is present
  • 6.2. Experimental Python Support
    • 6.2.1. Known Limitations
  • 6.3. Process Fraction
  • 6.4. Start and Stop API
  • 6.5. Environment Variables
• 7. Monitoring CPU
  • 7.1. REALTIME and CPUTIME
  • 7.2. IO
  • 7.3. MEMLEAK
  • 7.4. PAPI
    • 7.4.1. Proxy Sampling
  • 7.5. Linux perf_events
    • 7.5.1. Frequency-based sampling
    • 7.5.2. Multiplexing
    • 7.5.3. Profiling with perf_events
    • 7.5.4. Notes
    • 7.5.5. Thread blocking
  • 7.6. CPU Top-down Profiling
    • 7.6.1. TMA Metric Tree
    • 7.6.2. Event: topdown
    • 7.6.3. Event: topdown:retiring
    • 7.6.4. Event: topdown:frontend
    • 7.6.5. Event: topdown:backend
    • 7.6.6. Event: topdown:all
    • 7.6.7. Combining Events
    • 7.6.8. Workflow Example
      • 7.6.8.1. Step 1 — Performance Overview
      • 7.6.8.2. Step 2 — Backend Bound Subtree
      • 7.6.8.3. Step 3 — Verify After Optimization
    • 7.6.9. Interpretation Notes
    • 7.6.10. Summary of Events
• 8. Monitoring MPI
  • 8.1. Measuring MPI Ranks
• 9. Monitoring OpenMP
  • 9.1. Monitoring OpenMP on the Host
  • 9.2. Monitoring OpenMP Offloading on GPUs
    • 9.2.1. AMD OpenMP
    • 9.2.2. Intel OneAPI OpenMP
    • 9.2.3. NVIDIA OpenMP
    • 9.2.4. HPE’s Cray OpenMP
    • 9.2.5. GCC OpenMP
• 10. Monitoring GPUs
  • 10.1. GPU Measurement Quickstart
  • 10.2. Profiling GPU Operations
  • 10.3. Tracing GPU Operations
  • 10.4. PC Sampling on GPUs
    • 10.4.1. Attributing PC Samples to Source Code
  • 10.5. GPU Top-down Analysis
    • 10.5.1. Metrics Hierarchy
    • 10.5.2. Presentation
    • 10.5.3. A Case Study
    • 10.5.4. Inspecting Metrics
    • 10.5.5. Under the Hood
  • 10.6. Vendor-specific Details
    • 10.6.1. AMD GPUs
      • 10.6.1.1. AMD PC Sampling
      • 10.6.1.2. Source Attribution
      • 10.6.1.3. HW Counters
    • 10.6.2. NVIDIA GPUs
      • 10.6.2.1. NVIDIA PC Sampling
      • 10.6.2.2. Source Attribution
      • 10.6.2.3. Binary analysis
    • 10.6.3. Intel GPUs
      • 10.6.3.1. Intel PC Sampling
      • 10.6.3.2. Source Attribution
• 11. Hpcviewer
  • 11.1. Overview
    • 11.1.1. Downloading
    • 11.1.2. Building from Source
    • 11.1.3. Launching
    • 11.1.4. Menus
      • 11.1.4.1. File
      • 11.1.4.2. Filter
      • 11.1.4.3. View
      • 11.1.4.4. Help (Linux only)
    • 11.1.5. Limitations
  • 11.2. Profile View
    • 11.2.1. Panes
      • 11.2.1.1. Source Pane
      • 11.2.1.2. Navigation Pane
        • 11.2.1.2.1. Control Panel
        • 11.2.1.2.2. Context menus
      • 11.2.1.3. Metric Pane
      • 11.2.1.4. Top-down Pane
    • 11.2.2. Understanding Metrics
      • 11.2.2.1. How Metrics are Computed
      • 11.2.2.2. Example
    • 11.2.3. Derived Metrics
      • 11.2.3.1. Formulae
      • 11.2.3.2. Examples
      • 11.2.3.3. Creating Derived Metrics
    • 11.2.4. Plotting Metrics for Execution Contexts
      • 11.2.4.1. Plot Graphs
      • 11.2.4.2. Thread View
    • 11.2.5. Filtering Tree Nodes
    • 11.2.6. Convenience Features
      • 11.2.6.1. Source Code Pane
      • 11.2.6.2. Metric Pane
  • 11.3. Trace view
    • 11.3.1. Action and Information Pane
    • 11.3.2. Customizing the Color Map
    • 11.3.3. Filtering Execution Contexts
      • 11.3.3.1. Filtering Suggestions
  • 11.4. Accessing Remote Databases
    • 11.4.1. Installing hpcserver
    • 11.4.2. Opening a Remote Database
• 12. Strategies for Insight
  • 12.1. Monitoring High-Latency Events
  • 12.2. Computing Derived Metrics
  • 12.3. Pinpointing and Quantifying Inefficiencies
  • 12.4. Analyzing Scalability Bottlenecks
    • 12.4.1. Scalability Analysis Using Expectations
      • 12.4.1.1. Weak Scaling
      • 12.4.1.2. Exploring Scaling Losses
• 13. Known Issues
  • 13.1. Limited cross-version compatibility with ROCm
  • 13.2. No support for PC sampling with CUDA 13+
  • 13.3. When monitoring applications that use ROCm, using LD_AUDIT in hpcrun may cause it to fail to elide OpenMP runtime frames
  • 13.4. When using instrumentation on Intel GPUs, hpcrun may report that substantial time is spent in a partial call path consisting of only an unknown procedure
  • 13.5. hpcrun reports partial call paths for code executed by a constructor prior to entering main
  • 13.6. hpcrun may fail to measure a program execution on a CPU with hardware performance counters
  • 13.7. hpcrun may associate several profiles and traces with rank 0, thread 0
  • 13.8. hpcrun sometimes enables writing of read-only data in memory
  • 13.9. A confusing label for GPU theoretical occupancy
• 14. FAQ and Troubleshooting
  • 14.1. General Measurement Failures
    • 14.1.1. Profiling setuid programs
    • 14.1.2. Problems loading dynamic libraries
    • 14.1.3. Problems using allocator replacement libraries
    • 14.1.4. Problems caused by gprof instrumentation
  • 14.2. Measurement Failures using NVIDIA GPUs
    • 14.2.1. Deadlock while monitoring a program that uses IBM Spectrum MPI and NVIDIA GPUs
    • 14.2.2. Ensuring permission to use GPU performance counters
    • 14.2.3. Avoiding the error cudaErrorUnknown
    • 14.2.4. Avoiding the error CUPTI_ERROR_NOT_INITIALIZED
    • 14.2.5. Avoiding the error CUPTI_ERROR_HARDWARE_BUSY
    • 14.2.6. Avoiding the error CUPTI_ERROR_UNKNOWN
  • 14.3. General Measurement Issues
    • 14.3.1. How do I choose sampling periods?
    • 14.3.2. Why do I see partial unwinds?
    • 14.3.3. Measurement with HPCToolkit has high overhead! Why?
    • 14.3.4. Some of my syscalls return EINTR
    • 14.3.5. My application spends a lot of time in C library functions with names that include mcount
  • 14.4. Problems Recovering Loops in NVIDIA GPU binaries
  • 14.5. Graphical User Interface Issues
    • 14.5.1. hpcviewer fails to launch
    • 14.5.2. Fail to run hpcviewer: executable launcher was unable to locate its companion shared library
    • 14.5.3. Launching hpcviewer is very slow on Windows
    • 14.5.4. When executing hpcviewer, it complains cannot create “Java Virtual Machine”
    • 14.5.5. hpcviewer fails to launch due to java.lang.NoSuchMethodError exception.
    • 14.5.6. hpcviewer fails due to java.lang.OutOfMemoryError exception.
    • 14.5.7. hpcviewer writes a long list of Java error messages to the terminal!
    • 14.5.8. hpcviewer attributes performance information only to functions and not to source code loops and lines! Why?
    • 14.5.9. hpcviewer hangs trying to open a large database! Why?
    • 14.5.10. hpcviewer runs glacially slowly! Why?
    • 14.5.11. hpcviewer does not show my source code! Why?
      • 14.5.11.1. An explanation how HPCToolkit finds source files
    • 14.5.12. hpcviewer’s reported line numbers do not exactly correspond to what I see in my source code! Why?
    • 14.5.13. hpcviewer claims that there are several calls to a function within a particular source code scope, but my source code only has one! Why?
    • 14.5.14. hpcviewer’s Trace view shows lots of white space on the left. Why?
  • 14.6. Debugging
    • 14.6.1. How do I debug HPCToolkit’s measurement?
    • 14.6.2. Tracing HPCToolkit’s Measurement Subsystem
    • 14.6.3. Using a debugger to inspect an execution being monitored by HPCToolkit
• 15. Environment Variables
  • 15.1. Environment Variables for Users
  • 15.2. Environment Variables that May Avoid a Crash
  • 15.3. Environment Variables for Developers
• 16. Getting Help
• 17. Information for Experts
  • 17.1. Hardware Counter Event Names
    • 17.1.1. Examples