# Monitoring Applications This chapter describes the mechanics of using `hpcrun` to profile a single process application and collect performance data. For advice on how to choose events, perform scaling studies, etc., see Chapter [12](#chpt:effective-performance-analysis) *Strategies for Insight*. (hpcrun)= ## Using `hpcrun` The `hpcrun` launcher is used to run an application and collect call path profiles and call path traces data for *dynamically linked* binaries. For dynamically linked programs, this requires no change to the program source and no change to the build procedure. You should build your application natively with full optimization. `hpcrun` inserts its profiling code into the application at runtime via `LD_AUDIT` (preferred in most situations) or `LD_PRELOAD` (still available since most versions of glibc `LD_AUDIT` have significant bugs in corner cases). `hpcrun` monitors the execution of applications on a CPU using asynchronous sampling and, upon request, monitors GPU activity using vendor runtime capabilities. If `hpcrun` is used without any arguments to measure a program ``` hpcrun app arg ... ``` it will the measure the program's execution by sampling its CPUTIME and collect a call path profile for each thread in the execution. More about the CPUTIME metric can be found in Section [7.1](#linux-timers). In addition to a call path profile, `hpcrun` can collect a call path trace of an execution if the `-t` (or `--trace`) option is used turn on tracing. The following use of `hpcrun` will collect both a call path profile and a call path trace of CPU execution using the default CPUTIME sample source. ``` hpcrun -t app arg ... ``` The `-t` will collect a call path trace based on asynchronous sampling of each CPU thread using a time-based sampling metric. Besides the default `CPUTIME` metric, other time-based metrics include `REALTIME` and `CYCLES`. How to use these metrics is described a bit later. The `-t` option also traces any GPU operations if a `-e gpu=*` option is used to enable measurement of GPU activities. Traces are most useful for understanding the execution dynamics of multithreaded or multi-process applications; however, you may find a trace of a single-threaded application to be useful to understand how an execution unfolds over time. For programs that launch many GPU operations per second, the `-tt` will collect a more useful trace than `-t`. Like the `-t` option, `-tt` records a call path trace based on asynchronous sampling of each CPU thread using a time-based sampling metric. Unlike `-t`, `-tt` also records a sample for a thread as it launches each GPU operation, if any. Without `-tt`, the activity seen on a CPU trace line at the time it launches a GPU operation is often from far earlier in the execution, which can make it difficult to relate to concurrent CPU and GPU activity. Since additional non-sample elements are added, any statistical properties of the CPU traces are disturbed. While CPUTIME is used as the default sample source if no other sample source is specified, many other sample sources are available. Typically, one uses the `-e` (or `--event`) to specify a sample source and sampling rate.[^7] Sample sources are specified as '`event@howoften`' where `event` is the name of the source and `howoften` is either a number specifying the period (threshold) for that event, or `f` followed by a number, e.g., `@f100` specifying a target sampling frequency for the event in samples/second.[^8] Note that a higher period implies a lower rate of sampling. The `-e` option may be used multiple times to specify that multiple sample sources be used for measuring an execution. The basic syntax for profiling an application with `hpcrun` is: ``` hpcrun -t -e event@howoften ... app arg ... ``` For example, to profile an application using hardware counter sample sources provided by Linux `perf_events` and sample cycles 300 times/second (the default sampling frequency) and sample every 4,000,000 instructions, you would use: ``` hpcrun -e CYCLES -e INSTRUCTIONS@4000000 app arg ... ``` The units for timer-based sample sources (`CPUTIME` and `REALTIME` are microseconds, so to sample an application with tracing every 5,000 microseconds (200 times/second), you would use: ``` hpcrun -t -e CPUTIME@5000 app arg ... ``` `hpcrun` stores its raw performance data in a *measurements* directory with the program name in the directory name. On systems with a batch job scheduler (eg, PBS) the name of the job is appended to the directory name. ``` hpctoolkit-app-measurements[-jobid] ``` It is best to use a different measurements directory for each run. So, if you're using `hpcrun` on a local workstation without a job launcher, you can use the '`-o dirname`' option to specify an alternate directory name. For programs that use their own launch script (eg, `mpirun` or `mpiexec` for MPI), put the application's run script on the outside (first) and `hpcrun` on the inside (second) on the command line. For example, ``` mpirun -n 4 hpcrun -e CYCLES mpiapp arg ... ``` Note that `hpcrun` is intended for profiling dynamically linked *binaries*. It will not work well if used to profile a shell script. At best, you would be profiling the shell interpreter, not the script commands, and sometimes this will fail outright. Profiling statically-linked binaries is no longer supported by HPCToolkit. ### If `hpcrun` causes your application to fail `hpcrun` can cause applications to fail in certain circumstances. Here, we describe two kind of failures that may arise and how to sidestep them. (hpcrun-audit)= #### `hpcrun` causes failures related to loading or using shared libraries Unfortunately, the Glibc implementations used today on most platforms have known bugs monitoring loading and unloading of shared libraries and calls to a shared library's API. While the best approach for coping with these problems is to use a system running Glibc 2.35 or later, for most people, this is not an option: the system administrator picks the operating system version, which determines the Glibc version available to developers. To understand what kinds of problems that you may encounter with shared libraries and how you can work around them, it is helpful to understand how HPCToolkit monitors shared libraries. On Power and `x86_64` architectures, by default `hpcrun` uses `LD_AUDIT` to monitor an application's use of dynamic libraries. Use of `LD_AUDIT` is the only strategy for monitoring shared libraries that will not cause a change in application behavior when libraries contain a `RUNPATH`. However, Glibc's implementation of `LD_AUDIT` has a number of bugs that may crash the application: - Until Glibc 2.35, most applications running on ARM will crash. This was caused by a fatal flaw in Glibc's PLT handler for ARM, where an argument register that should have been saved was instead replaced with a junk pointer value. This register is used to return C/C++ `struct` values from functions and methods, including some C++ constructors. - Until Glibc 2.35, applications and libraries using `dlmopen` will crash. While most applications do not use `dlmopen`, an example of a library that does is Intel's GTPin, which `hpcrun` uses to instrument Intel GPU code. - Applications and libraries using significant amounts of static TLS space may crash with the message "`cannot allocate memory in static TLS block`." This is caused by a flaw in Glibc causing it to allocate insufficient static TLS space when `LD_AUDIT` is enabled. For Glibc 2.35 and newer, setting the environment variable ``` export GLIBC_TUNABLES=glibc.rtld.optional_static_tls=0x1000000 ``` will instruct Glibc to allocate 16MB of static TLS memory per thread, in our experience this is far more than any application will use (however the value can be adjusted freely). For older Glibc, the only option is to disable `hpcrun`'s use of `LD_AUDIT`. The following options direct `hpcrun` to adjust the strategy it uses for monitoring dynamic libraries. We suggest that you don't consider using any of these options unless your program fails using `hpcrun`'s defaults. `--disable-auditor` : This option instructs hpcrun to track dynamic library operations by intercepting `dlopen` and `dlclose` instead of using `LD_AUDIT`. Note that this alternate approach can cause problem with libraries and applications that specify a `RUNPATH`. `--enable-auditor` : This option is default, except on ARM or when Intel GTPin instrumentation is enabled. Passing this option instructs `hpcrun` to use `LD_AUDIT` in all cases. `--disable-auditor-got-rewriting` : When using an `LD_AUDIT`, Glibc unnecessarily intercepts every call to a function in a shared library. `hpcrun` avoids this overhead by rewriting each shared library's global offset table (GOT). Such rewriting is tricky. This option can be used to disable GOT rewriting if it is believed that the rewriting is causing the application to fail. `--namespace-single` : `dlmopen` may load a shared library into an alternate namespace, which crashes on Glibc until 2.35. This option instructs `hpcrun` to override `dlmopen` to instead load all shared libraries within the application namespace. This may significantly change application behavior, but may be helpful to avoid crashing. This option is default when Intel GTPin instrumentation is enabled. `--namespace-multiple` : This option is the opposite of `--namespace-single`, and will instruct `hpcrun` to *not* override `dlmopen` and thus retain its normal function. This option is default except when Intel GTPin instrumentation is enabled. If your code fails to find libraries when it is monitoring your code by wrapping `dlopen` and `dlclose` rather than using `LD_AUDIT`, you can sidestep this problem by adding any library paths listed in the `RUNPATH` of your application or library to your `LD_LIBRARY_PATH` environment variable before launching `hpcrun`. #### `hpcrun` causes your application to fail when `gprof` instrumentation is present When an application has been compiled with the compiler flag `-pg`, the compiler adds instrumentation to collect performance measurement data for the `gprof` profiler. Measuring application performance with HPCToolkit's measurement subsystem and `gprof` instrumentation active in the same execution may cause the execution to abort. One can detect the presence of `gprof` instrumentation in an application by the presence of `__monstartup` and `_mcleanup` symbols in a executable. One can disable `gprof` instrumentation when measuring the performance of a dynamically-linked application by using the `--disable-gprof` argument to `hpcrun`. ## Experimental Python Support This section provides a brief overview of how to use HPCToolkit to analyze the performance of Python-based applications. Normally, `hpcrun` will attribute performance to the CPython implementation, not to the application Python code, as shown in Figure [6.2](#fig:python-support). This usually is of little interest to an application developer, so HPCToolkit provides experimental support for attributing to Python callstacks. **NOTE: Python support in HPCToolkit is in early days. If you compile HPCToolkit to match the version of Python being used by your application, in many cases you will find that you can measure what you want. However, other cases may not work as expected; crashes and corrupted performance data are not uncommon. For the aforementioned reasons, use at your own risk.** ```{figure-md} fig:python-support  Figure 6.2: Example of a simple Python application measured without (left) and with (right) Python support enabled via `hpcrun` `-a python`. The left database has no source code, since sources were not provided for the CPython implementation. ``` If HPCToolkit has been compiled with Python support enabled, `hpcrun` is able to replace segments of the C callstacks with the Python code running in those frames. To enable this transformation, profile your application the additional `-a python` flag: ``` hpcrun -a python -e event@howoften python3 app arg ... ``` As shown in Figure [6.2](#fig:python-support), passing this flag removes the CPython implementation details, replacing it with the much smaller Python callstack. When Python calls an external C library, HPCToolkit will report both the name of the Python function object and the C function being called, in this example `sleep` and Glibc's `clock_nanosleep` respectively. ### Known Limitations This section lists a number of known limitations with the current implementation of the Python support. It is recommended that users are aware of these limitations before attempting to use the Python support in practice. 1. Pythons older than 3.8 are not supported by HPCToolkit. Please upgrade any applications and Python extensions to use a recent version of Python before attempting to enable Python support. 1. The application should be run with the same Python that was used to compile HPCToolkit. The CPython ABI can change between patch versions and due to certain build configuration flags. To ensure `hpcrun` will not unwittingly crash the application, it is best to use a single Python for both HPCToolkit and the application. Despite this recommendation, it is worth noting that we have had some minor success with cross-version compatibility (e.g. building HPCToolkit with Python 3.11.8 support and using it to measure a program running under Python 3.11.6) However, you should be surprised if cross-version compatibility works rather than expecting it to work, even across patch releases. 1. The bottom-up and flat views of `hpcviewer` may not correctly present Python callstacks, particularly those that call C/C++ extensions. Some Python functions may be missing, and the metrics attributed to them may be suspect. In these cases, refer to the top-down view as the known-good source of truth. 1. Threads spawned by Python's `threading` and `subprocess` modules are not fully supported. Only the main Python thread will attribute performance to Python callstacks, all others will attribute performance to the CPython implementation. If Python `threading` is a performance bottleneck, consider implementing the parallelism in a C/C++ extension instead of in Python to avoid [contention on the GIL](https://docs.python.org/3/glossary.html#term-global-interpreter-lock). 1. Applications using signals and signal handlers, for example Python's `signal` module, will experience crashes when run under `hpcrun`. The current implementation fails to process the non-sequential modifications to the Python stack that take place when Python handles signals. ## Process Fraction Although `hpcrun` can profile parallel jobs with thousands or tens of thousands of processes, there are two scaling problems that become prohibitive beyond a few thousand cores. First, `hpcrun` writes the measurement data for all of the processes into a single directory. This results in one file per process plus one file per thread (two files per thread if using tracing). Unix file systems are not equipped to handle directories with many tens or hundreds of thousands of files. Second, the sheer volume of data can overwhelm the viewer when the size of the database far exceeds the amount of memory on the machine. The solution is to sample only a fraction of the processes. That is, you can run an application on many thousands of cores but record data for only a few hundred processes. The other processes run the application but do not record any measurement data. This is what the process fraction option (`-f` or `--process-fraction`) does. For example, to monitor 10% of the processes, use: ``` hpcrun -f 0.10 -e event@howoften app arg ... hpcrun -f 1/10 -e event@howoften app arg ... ``` With this option, each process generates a random number and records its measurement data with the given probability. The process fraction (probability) may be written as a decimal number (0.10) or as a fraction (1/10) between 0 and 1. So, in the above example, all three cases would record data for approximately 10% of the processes. Aim for a number of processes in the hundreds. ## Start and Stop API HPCToolkit supports an API for the application to start and stop sampling. This is useful if you want to profile only a subset of a program and ignore the rest. The API supports the following functions. ``` void hpctoolkit_sampling_start(void); void hpctoolkit_sampling_stop(void); ``` For example, suppose that your program has three major phases: it reads input from a file, performs some numerical computation on the data and then writes the output to another file. And suppose that you want to profile only the compute phase and skip the read and write phases. In that case, you could stop sampling at the beginning of the program, restart it before the compute phase and stop it again at the end of the compute phase. This interface is process wide, not thread specific. That is, it affects all threads of a process. Note that when you turn sampling on or off, you should do so uniformly across all processes, normally at the same point in the program. Enabling sampling in only a subset of the processes would likely produce skewed and misleading results. And for technical reasons, when sampling is turned off in a threaded process, interrupts are disabled only for the current thread. Other threads continue to receive interrupts, but they don't unwind the call stack or record samples. So, another use for this interface is to protect syscalls that are sensitive to being interrupted with signals. For example, some Gemini interconnect (GNI) functions called from inside `gasnet_init()` or `MPI_Init()` on Cray XE systems will fail if they are interrupted by a signal. As a workaround, you could turn sampling off around those functions. Also, you should use this interface only at the top level for major phases of your program. That is, the granularity of turning sampling on and off should be much larger than the time between samples. Turning sampling on and off down inside an inner loop will likely produce skewed and misleading results. To use this interface, put the above function calls into your program where you want sampling to start and stop. Remember, starting and stopping apply process wide. For C/C++, include the following header file from the HPCToolkit `include` directory. ``` #include ``` Compile your application with `libhpctoolkit` with `-I` and `-L` options for the include and library paths. For example, ``` gcc -I /path/to/hpctoolkit/include app.c ... \ -L /path/to/hpctoolkit/lib/hpctoolkit -lhpctoolkit ... ``` The `libhpctoolkit` library provides weak symbol no-op definitions for the start and stop functions. For dynamically linked programs, be sure to include `-lhpctoolkit` on the link line (otherwise your program won't link). To run the program, set the `LD_LIBRARY_PATH` environment variable to include the HPCToolkit `lib/hpctoolkit` directory. ``` export LD_LIBRARY_PATH=/path/to/hpctoolkit/lib/hpctoolkit ``` Note that sampling is initially turned on until the program turns it off. If you want it initially turned off, then use the `-ds` (or `--delay-sampling`) option for `hpcrun`. ``` hpcrun -ds -e event@howoften app arg ... ``` (sec:env-vars)= ## Environment Variables For most systems, `hpcrun` requires no special environment variable settings. There are two situations, however, where `hpcrun`, to function correctly, *must* refer to environment variables. These environment variables, and corresponding situations are: `HPCTOOLKIT` : To function correctly, `hpcrun` must know the location of the HPCToolkit top-level installation directory. The `hpcrun` script uses elements of the installation `lib` and `libexec` subdirectories. On most systems, the `hpcrun` can find the requisite components relative to its own location in the file system. However, some parallel job launchers *copy* the `hpcrun` script to a different location as they launch a job. If your system does this, you must set the `HPCTOOLKIT` environment variable to the location of the HPCToolkit top-level installation directory before launching a job. **Note to system administrators:** if your system provides a module system for configuring software packages, then constructing a module for HPCToolkit to initialize these environment variables to appropriate settings would be convenient for users. [^7]: GPU and OpenMP measurement events don't accept a rate. [^8]: Frequency-based sampling and the frequency-based notation for `howoften` is only available for sample sources managed by Linux `perf_events`. For Linux `perf_events`, HPCToolkit uses a default sampling frequency of 300 samples/second.