6. Monitoring Applications

This chapter describes the mechanics of using hpcrun to profile a single process application and collect performance data. For advice on how to choose events, perform scaling studies, etc., see Chapter 12 Strategies for Insight.

6.1. Using hpcrun

The hpcrun launcher is used to run an application and collect call path profiles and call path traces data for dynamically linked binaries. For dynamically linked programs, this requires no change to the program source and no change to the build procedure. You should build your application natively with full optimization. hpcrun inserts its profiling code into the application at runtime via LD_AUDIT (preferred in most situations) or LD_PRELOAD (still available since most versions of glibc LD_AUDIT have significant bugs in corner cases).

hpcrun monitors the execution of applications on a CPU using asynchronous sampling and, upon request, monitors GPU activity using vendor runtime capabilities. If hpcrun is used without any arguments to measure a program

hpcrun app arg ...

it will the measure the program’s execution by sampling its CPUTIME and collect a call path profile for each thread in the execution. More about the CPUTIME metric can be found in Section 7.1.

In addition to a call path profile, hpcrun can collect a call path trace of an execution if the -t (or --trace) option is used turn on tracing. The following use of hpcrun will collect both a call path profile and a call path trace of CPU execution using the default CPUTIME sample source.

hpcrun -t app arg ...

The -t will collect a call path trace based on asynchronous sampling of each CPU thread using a time-based sampling metric. Besides the default CPUTIME metric, other time-based metrics include REALTIME and CYCLES. How to use these metrics is described a bit later.

The -t option also traces any GPU operations if a -e gpu=* option is used to enable measurement of GPU activities.

Traces are most useful for understanding the execution dynamics of multithreaded or multi-process applications; however, you may find a trace of a single-threaded application to be useful to understand how an execution unfolds over time.

For programs that launch many GPU operations per second, the -tt will collect a more useful trace than -t. Like the -t option, -tt records a call path trace based on asynchronous sampling of each CPU thread using a time-based sampling metric. Unlike -t, -tt also records a sample for a thread as it launches each GPU operation, if any. Without -tt, the activity seen on a CPU trace line at the time it launches a GPU operation is often from far earlier in the execution, which can make it difficult to relate to concurrent CPU and GPU activity. Since additional non-sample elements are added, any statistical properties of the CPU traces are disturbed.

While CPUTIME is used as the default sample source if no other sample source is specified, many other sample sources are available. Typically, one uses the -e (or --event) to specify a sample source and sampling rate.[7] Sample sources are specified as ‘event@howoften’ where event is the name of the source and howoften is either a number specifying the period (threshold) for that event, or f followed by a number, e.g., @f100 specifying a target sampling frequency for the event in samples/second.[8] Note that a higher period implies a lower rate of sampling. The -e option may be used multiple times to specify that multiple sample sources be used for measuring an execution.

The basic syntax for profiling an application with hpcrun is:

hpcrun -t -e event@howoften ... app arg ...

For example, to profile an application using hardware counter sample sources provided by Linux perf_events and sample cycles 300 times/second (the default sampling frequency) and sample every 4,000,000 instructions, you would use:

hpcrun -e CYCLES -e INSTRUCTIONS@4000000 app arg ...

The units for timer-based sample sources (CPUTIME and REALTIME are microseconds, so to sample an application with tracing every 5,000 microseconds (200 times/second), you would use:

hpcrun -t -e CPUTIME@5000 app arg ...

hpcrun stores its raw performance data in a measurements directory with the program name in the directory name. On systems with a batch job scheduler (eg, PBS) the name of the job is appended to the directory name.

hpctoolkit-app-measurements[-jobid]

It is best to use a different measurements directory for each run. So, if you’re using hpcrun on a local workstation without a job launcher, you can use the ‘-o dirname’ option to specify an alternate directory name.

For programs that use their own launch script (eg, mpirun or mpiexec for MPI), put the application’s run script on the outside (first) and hpcrun on the inside (second) on the command line. For example,

mpirun -n 4 hpcrun -e CYCLES mpiapp arg ...

Note that hpcrun is intended for profiling dynamically linked binaries. It will not work well if used to profile a shell script. At best, you would be profiling the shell interpreter, not the script commands, and sometimes this will fail outright. Profiling statically-linked binaries is no longer supported by HPCToolkit.

6.1.1. If hpcrun causes your application to fail

hpcrun can cause applications to fail in certain circumstances. Here, we describe two kind of failures that may arise and how to sidestep them.

6.1.1.1. hpcrun causes failures related to loading or using shared libraries

Unfortunately, the Glibc implementations used today on most platforms have known bugs monitoring loading and unloading of shared libraries and calls to a shared library’s API. While the best approach for coping with these problems is to use a system running Glibc 2.35 or later, for most people, this is not an option: the system administrator picks the operating system version, which determines the Glibc version available to developers.

To understand what kinds of problems that you may encounter with shared libraries and how you can work around them, it is helpful to understand how HPCToolkit monitors shared libraries. On Power and x86_64 architectures, by default hpcrun uses LD_AUDIT to monitor an application’s use of dynamic libraries. Use of LD_AUDIT is the only strategy for monitoring shared libraries that will not cause a change in application behavior when libraries contain a RUNPATH. However, Glibc’s implementation of LD_AUDIT has a number of bugs that may crash the application:

• Until Glibc 2.35, most applications running on ARM will crash. This was caused by a fatal flaw in Glibc’s PLT handler for ARM, where an argument register that should have been saved was instead replaced with a junk pointer value. This register is used to return C/C++ struct values from functions and methods, including some C++ constructors.

• Until Glibc 2.35, applications and libraries using dlmopen will crash. While most applications do not use dlmopen, an example of a library that does is Intel’s GTPin, which hpcrun uses to instrument Intel GPU code.

• Applications and libraries using significant amounts of static TLS space may crash with the message “cannot allocate memory in static TLS block.” This is caused by a flaw in Glibc causing it to allocate insufficient static TLS space when LD_AUDIT is enabled. For Glibc 2.35 and newer, setting the environment variable

  export GLIBC_TUNABLES=glibc.rtld.optional_static_tls=0x1000000
  

  will instruct Glibc to allocate 16MB of static TLS memory per thread, in our experience this is far more than any application will use (however the value can be adjusted freely). For older Glibc, the only option is to disable hpcrun’s use of LD_AUDIT.

The following options direct hpcrun to adjust the strategy it uses for monitoring dynamic libraries. We suggest that you don’t consider using any of these options unless your program fails using hpcrun’s defaults.

--disable-auditor

This option instructs hpcrun to track dynamic library operations by intercepting dlopen and dlclose instead of using LD_AUDIT. Note that this alternate approach can cause problem with libraries and applications that specify a RUNPATH.

--enable-auditor

This option is default, except on ARM or when Intel GTPin instrumentation is enabled. Passing this option instructs hpcrun to use LD_AUDIT in all cases.

--disable-auditor-got-rewriting

When using an LD_AUDIT, Glibc unnecessarily intercepts every call to a function in a shared library. hpcrun avoids this overhead by rewriting each shared library’s global offset table (GOT). Such rewriting is tricky. This option can be used to disable GOT rewriting if it is believed that the rewriting is causing the application to fail.

--namespace-single

dlmopen may load a shared library into an alternate namespace, which crashes on Glibc until 2.35. This option instructs hpcrun to override dlmopen to instead load all shared libraries within the application namespace. This may significantly change application behavior, but may be helpful to avoid crashing. This option is default when Intel GTPin instrumentation is enabled.

--namespace-multiple

This option is the opposite of --namespace-single, and will instruct hpcrun to not override dlmopen and thus retain its normal function. This option is default except when Intel GTPin instrumentation is enabled.

If your code fails to find libraries when it is monitoring your code by wrapping dlopen and dlclose rather than using LD_AUDIT, you can sidestep this problem by adding any library paths listed in the RUNPATH of your application or library to your LD_LIBRARY_PATH environment variable before launching hpcrun.

6.1.1.2. hpcrun causes your application to fail when gprof instrumentation is present

When an application has been compiled with the compiler flag -pg, the compiler adds instrumentation to collect performance measurement data for the gprof profiler. Measuring application performance with HPCToolkit’s measurement subsystem and gprof instrumentation active in the same execution may cause the execution to abort. One can detect the presence of gprof instrumentation in an application by the presence of __monstartup and _mcleanup symbols in a executable. One can disable gprof instrumentation when measuring the performance of a dynamically-linked application by using the --disable-gprof argument to hpcrun.

6.2. Experimental Python Support

This section provides a brief overview of how to use HPCToolkit to analyze the performance of Python-based applications. Normally, hpcrun will attribute performance to the CPython implementation, not to the application Python code, as shown in Figure 6.2. This usually is of little interest to an application developer, so HPCToolkit provides experimental support for attributing to Python callstacks.

NOTE: Python support in HPCToolkit is in early days. If you compile HPCToolkit to match the version of Python being used by your application, in many cases you will find that you can measure what you want. However, other cases may not work as expected; crashes and corrupted performance data are not uncommon. For the aforementioned reasons, use at your own risk.

Figure 6.2: Example of a simple Python application measured without (left) and with (right) Python support enabled via hpcrun -a python. The left database has no source code, since sources were not provided for the CPython implementation.

If HPCToolkit has been compiled with Python support enabled, hpcrun is able to replace segments of the C callstacks with the Python code running in those frames. To enable this transformation, profile your application the additional -a python flag:

hpcrun -a python -e event@howoften python3 app arg ...

As shown in Figure 6.2, passing this flag removes the CPython implementation details, replacing it with the much smaller Python callstack. When Python calls an external C library, HPCToolkit will report both the name of the Python function object and the C function being called, in this example sleep and Glibc’s clock_nanosleep respectively.

6.2.1. Known Limitations

This section lists a number of known limitations with the current implementation of the Python support. It is recommended that users are aware of these limitations before attempting to use the Python support in practice.

1. Pythons older than 3.8 are not supported by HPCToolkit. Please upgrade any applications and Python extensions to use a recent version of Python before attempting to enable Python support.

2. The application should be run with the same Python that was used to compile HPCToolkit. The CPython ABI can change between patch versions and due to certain build configuration flags. To ensure hpcrun will not unwittingly crash the application, it is best to use a single Python for both HPCToolkit and the application.

   Despite this recommendation, it is worth noting that we have had some minor success with cross-version compatibility (e.g. building HPCToolkit with Python 3.11.8 support and using it to measure a program running under Python 3.11.6) However, you should be surprised if cross-version compatibility works rather than expecting it to work, even across patch releases.

3. The bottom-up and flat views of hpcviewer may not correctly present Python callstacks, particularly those that call C/C++ extensions. Some Python functions may be missing, and the metrics attributed to them may be suspect. In these cases, refer to the top-down view as the known-good source of truth.

4. Threads spawned by Python’s threading and subprocess modules are not fully supported. Only the main Python thread will attribute performance to Python callstacks, all others will attribute performance to the CPython implementation. If Python threading is a performance bottleneck, consider implementing the parallelism in a C/C++ extension instead of in Python to avoid contention on the GIL.

5. Applications using signals and signal handlers, for example Python’s signal module, will experience crashes when run under hpcrun. The current implementation fails to process the non-sequential modifications to the Python stack that take place when Python handles signals.

6.3. Process Fraction

Although hpcrun can profile parallel jobs with thousands or tens of thousands of processes, there are two scaling problems that become prohibitive beyond a few thousand cores. First, hpcrun writes the measurement data for all of the processes into a single directory. This results in one file per process plus one file per thread (two files per thread if using tracing). Unix file systems are not equipped to handle directories with many tens or hundreds of thousands of files. Second, the sheer volume of data can overwhelm the viewer when the size of the database far exceeds the amount of memory on the machine.

The solution is to sample only a fraction of the processes. That is, you can run an application on many thousands of cores but record data for only a few hundred processes. The other processes run the application but do not record any measurement data. This is what the process fraction option (-f or --process-fraction) does. For example, to monitor 10% of the processes, use:

hpcrun -f 0.10 -e event@howoften app arg ...
hpcrun -f 1/10 -e event@howoften app arg ...

With this option, each process generates a random number and records its measurement data with the given probability. The process fraction (probability) may be written as a decimal number (0.10) or as a fraction (1/10) between 0 and 1. So, in the above example, all three cases would record data for approximately 10% of the processes. Aim for a number of processes in the hundreds.

6.4. Start and Stop API

HPCToolkit supports an API for the application to start and stop sampling. This is useful if you want to profile only a subset of a program and ignore the rest. The API supports the following functions.

void hpctoolkit_sampling_start(void);
void hpctoolkit_sampling_stop(void);

For example, suppose that your program has three major phases: it reads input from a file, performs some numerical computation on the data and then writes the output to another file. And suppose that you want to profile only the compute phase and skip the read and write phases. In that case, you could stop sampling at the beginning of the program, restart it before the compute phase and stop it again at the end of the compute phase.

This interface is process wide, not thread specific. That is, it affects all threads of a process. Note that when you turn sampling on or off, you should do so uniformly across all processes, normally at the same point in the program. Enabling sampling in only a subset of the processes would likely produce skewed and misleading results.

And for technical reasons, when sampling is turned off in a threaded process, interrupts are disabled only for the current thread. Other threads continue to receive interrupts, but they don’t unwind the call stack or record samples. So, another use for this interface is to protect syscalls that are sensitive to being interrupted with signals. For example, some Gemini interconnect (GNI) functions called from inside gasnet_init() or MPI_Init() on Cray XE systems will fail if they are interrupted by a signal. As a workaround, you could turn sampling off around those functions.

Also, you should use this interface only at the top level for major phases of your program. That is, the granularity of turning sampling on and off should be much larger than the time between samples. Turning sampling on and off down inside an inner loop will likely produce skewed and misleading results.

To use this interface, put the above function calls into your program where you want sampling to start and stop. Remember, starting and stopping apply process wide. For C/C++, include the following header file from the HPCToolkit include directory.

#include <hpctoolkit.h>

Compile your application with libhpctoolkit with -I and -L options for the include and library paths. For example,

gcc -I /path/to/hpctoolkit/include app.c ... \
    -L /path/to/hpctoolkit/lib/hpctoolkit -lhpctoolkit ...

The libhpctoolkit library provides weak symbol no-op definitions for the start and stop functions. For dynamically linked programs, be sure to include -lhpctoolkit on the link line (otherwise your program won’t link).

To run the program, set the LD_LIBRARY_PATH environment variable to include the HPCToolkit lib/hpctoolkit directory.

export LD_LIBRARY_PATH=/path/to/hpctoolkit/lib/hpctoolkit

Note that sampling is initially turned on until the program turns it off. If you want it initially turned off, then use the -ds (or --delay-sampling) option for hpcrun.

hpcrun -ds -e event@howoften app arg ...

6.5. Environment Variables

For most systems, hpcrun requires no special environment variable settings. There are two situations, however, where hpcrun, to function correctly, must refer to environment variables. These environment variables, and corresponding situations are:

HPCTOOLKIT

To function correctly, hpcrun must know the location of the HPCToolkit top-level installation directory. The hpcrun script uses elements of the installation lib and libexec subdirectories. On most systems, the hpcrun can find the requisite components relative to its own location in the file system. However, some parallel job launchers copy the hpcrun script to a different location as they launch a job. If your system does this, you must set the HPCTOOLKIT environment variable to the location of the HPCToolkit top-level installation directory before launching a job.

Note to system administrators: if your system provides a module system for configuring software packages, then constructing a module for HPCToolkit to initialize these environment variables to appropriate settings would be convenient for users.

---

[7]

GPU and OpenMP measurement events don’t accept a rate.

[8]

Frequency-based sampling and the frequency-based notation for howoften is only available for sample sources managed by Linux perf_events. For Linux perf_events, HPCToolkit uses a default sampling frequency of 300 samples/second.