Score-P OPEN ISSUES v9.0 ======================== Effective: Mar 2025 This file lists known limitations and unimplemented features of various Score-P components. -------------------------------------------------------------------------------- * Platform support - Score-P is intended to run on ppc64, x86_64, aarch64, or riscv architectures. - Score-P has been tested on the following platforms: + HPE/Cray EX systems with PrgEnvs cray, intel, nvidia, gnu, amd, aocc, cray-amd. PrgEnv-cray with Cray PE prior 21.05 fails to compiler-instrument functions. + various Linux (Intel, AMD, ARM) clusters with GNU, Intel, NVIDIA, and AMD compilers. Note that some compilers are based on Clang. Note that Fujitsu compilers haven't been tested thoroughly, Fujitsu cross-compilers haven't been tested at all. The provided configure options (see INSTALL) may provide a good basis for building and testing the toolset on other systems. - The following platforms have not been tested recently: + Intel Xeon Phi (KNL) + IBM Blue Gene and IBM Power systems + Cray XC, XT, XE, XK However the supplied build system might still work on these systems. - Each toolset installation can only support one MPI implementation (because MPI is only source-code but not binary compatible). If your systems support more than one MPI implementation (e.g. Linux clusters often do), separate builds for each MPI implementation have to be installed. The same applies for SHMEM. - The same is true if your system features more than one compiler supporting automatic function instrumentation. Additionally if the GNU compiler plug-in based instrumentation is used, then different major version of the GNU compiler suite are also incompatible. I.e., for each major version a separate Score-P installation must be provided. - To build Score-P, a C and C++ compiler as well as C and, optionally, Fortran compiler wrappers (e.g., MPI or SHMEM) are used. Compilers and compiler wrappers must be compatible. - Note (dated): PGI changed its default C++ compiler from 'pgCC' (which was entirely removed in 16.1) to 'pgc++'. For building Score-P with '--with-nocross-compiler-suite=pgi', 'CXX=pgc++' is used. If your MPI and SHMEM compiler wrappers still use the old 'pgCC', please add 'CXX=pgCC' to your configure line to force Score-P to use 'pgCC' as well. This will prevent link failures. - (dated) Shared libraries on Cray systems: building Score-P's shared libraries works as normal through the '--enable-shared' configure option. But note that if the default Cray link mode is static (which is the case for CCE prior to 9.0.0) or if CRAYPE_LINK_TYPE=static is provided via the environment, you need to pass the '-dynamic' flag (if available) to the Cray compiler wrapper when building your application to link to Score-P's shared libraries. - Experimental platforms: The following platforms are considered experimental and don't get much attention. They may break without notification. Help to maintain these is always welcomed. + MinGW - Address-to-line lookup might not work as link.h and dl_iterate_phdr might not be available. + macOS - Building shared/dynamic libraries may not work. - Address-to-line lookup (e.g., for clang-based compiler instrumentation) does not work as link.h and dl_iterate_phdr is not available. -------------------------------------------------------------------------------- * Automatic compiler instrumentation via "scorep" based on (often undocumented) compiler switches - GNU : tested with GCC 4.9 and higher - NVHPC : tested with version 20.7 and higher - PGI : tested with version 10.1 and higher Note that PGI 13.8 is currently not supported as it is not recognized as an PGI compiler anymore. PGI 16 supports only 64-bit platforms. - IBM : only works for xlc/xlC version 7.0 and xlf version 9.1 and higher and corresponding bgxl compilers on Blue Gene systems - Intel : only works with Intel icc/icpc/ifort version 11 and higher compilers or the Clang-based ixc/icpx compilers from Intel oneAPI toolkit. Note that the Fortran ifx compiler does not support compiler instrumentation. - Cray : tested with version 8.1.8 and later, uses the GNU interface. - Fujitsu: tested with the language environment 1.2.40 in traditional and Clang mode on Fugaku, uses the GNU interface. - Clang : tested with Clang 4.0 and higher. Note that several compiler vendors now use Clang for their C and C++ compilers while the Fortran compiler stays vendor-specific. - The Intel compiler provides the function name and the source code location in one string separated by a colon. Thus, if the path name of the source code file contains colons, Score-P will split the source file name and the function name improperly. Additionally, the provided function name makes it impossible to distinguish overloaded functions in C++. Thus, functions which differ only in the argument list will be mapped to the same function definition by Score-P. - When using any of the intrinsic headers (e.g., xmmintrin.h and friends) from GCC version up to 6.3, it is known that the result of the compiler instrumentation will fail at link time because of undefined reference. A work around is to pass -finstrument-functions-exclude-file-list=intrin.h as flag to GCC on the command line. This issue does not apply if the GNU compiler plug-in based instrumentation is used. - Measurement filtering can only be applied to functions instrumented by the IBM, GNU, Intel, NVHPC, Clang, or PGI compilers as well as functions instrumented by Kokkos and user functions. Filtering of MPI, SHMEM, and OpenMP runtime functions is always ineffective. - The GCC plug-in is supported from GCC 4.9 onwards. Though due to constant changes in the plug-in API of the GNU compiler infrastructure, it is unlikely that the Score-P instrumentation plug-in builds with versions newer than GCC snapshot 11-20210307. - The pgCC compiler versions 13.9 and higher preinclude omp.h for OpenMP codes. This results in multiple defined symbols if the source file is preprocessed before compilation. Since version 14.1 an option is available to avoid preinclusion, which we can use for preprocessed source files. For the pgCC versions 13.9 until 14.0, preprocessing is not possible for C++ codes. - Some compiler interfaces provide function pointers that get translated to region meta data via libbfd from the binutils package. If Score-P uses a system libbfd and region meta data is implausible, consider reinstalling Score-P while providing a recent libbfd via the configure option `--with-libbfd=download`. - The LLVM plug-in is supported from LLVM 13.0 and onwards. Due to regular changes in the plug-in API, bleeding edge releases of LLVM might not be able to compile the plug-in. In our testing, the plug-in did compile with LLVM 13.0 to LLVM 18.1. - When no support for LLVM demangle is detected during configure, compile-time filtering will only work for mangled names. - Compiling HIP code with the LLVM plug-in might cause error messages of the linker `lld` because plug-in parameters are passed to the linker. This issue is fixed in LLVM 17.0 and newer, as well as in ROCm 6.1 and newer. -------------------------------------------------------------------------------- * MPI support - Most functions added with MPI-3.0 and MPI-3.1 only have Enter and Exit event records, i.e., Score-P can measure the time spent in these routines, but no further analysis is possible. - C++ bindings for MPI are not supported. These were deprecated as of MPI-2.2 and removed with MPI-3.0. When using C++ bindings for MPI, Score-P will most likely indicate the failed library wrapping with the following warning: "If you are using MPICH1, please ignore this warning. If not, it seems that your interprocess communication library (e.g., MPI) hasn't been initialized. Score-P cannot generate output." If your MPI implements the C++ bindings in a separate library on top of the C bindings, the following workaround might work for you: 1) Instrument your application with 'scorep --keep-files -v' It will make the instrumenter not remove the intermediate files and output the commands it executes. 2) Copy the link command (the last command from the scorep output). 3) Insert a link flag (e.g. -lmpicxx) for your C++ bindings library right before '-lscorep_adapter_mpi_events' and execute the modified link command. - When using derived data types in non-blocking communications, and no support for MPI_Type_dup() was detected, please ensure that the MPI_Datatype handle is still valid at the time the request finishes. - Currently, Score-P cannot handle MPI_Finalize() calls that occur after the end of main(), e.g., via a destructor of a static C++ object. Please call MPI_Finalize() before the end of main(). The issue will be resolved in a future version of Score-P. - The IBM Platform MPI (not mpixlc!) compiler wrapper (the formerly HP-MPI) does not append its libraries at the very end of the original link command. Thus, instrumenting applications with Score-P fails at link time due to unresolved symbols in the Score-P libraries. - Multi-threaded MPI communication is limited/experimental in Score-P. It is not that it is not possible to measure communication from other threads. But it is nearly impossible to do reasonable analysis when only ranks but not the real communication partners (threads) are known. There are efforts in the MPI forum to address this issue (endpoints). The current version of Score-P is designed to stop the instrumented application from crashing under multi-threaded MPI communication but there are some limitations. When thread-local storage is not detected during the configure step `MPI_THREAD_MULTIPLE` is not supported. All recorded MPI communication events will still be reported on the master thread. MPI implementations optimize small messages and create non-unique requests, e.g., empty requests in OpenMPI, and can not be distinguished by Score-P. A workaround is implemented. However, the ordering and, therefore, association to the MPI calls is lost. This effect was also present in previous versions of Score-P but went unnoticed. - Open MPI 4.0.0 is known broken to build with Score-P, because it accidentally removed deprecated-only MPI functions, without removing the prototypes in `mpi.h`. - Score-P leaks memory in the MPI requests management as the Score-P memory management does not allow freeing individual allocations. We try to reduce this leakage. Gradually increasing the number of requests will increase this effect. However, in practice, it does not seem to have any impact on the measurement. - The MPI sessions model is not supported and `MPI_Session_init` aborts the measurement. - Fortran 2008 bindings: - When instrumented with a Score-P that lacks F08 support, applications that initialize MPI with the `mpi` module, but call functions from `mpi_f08`, might crash or produce inconsistent measurements. Verify with `scorep-info config-summary | grep "Fortran 2008 bindings"` that Score-P is built with F08 support. - The deprecated functions `MPI_KEYVAL_CREATE`, `MPI_KEYVAL_FREE`, `MPI_ATRR_PUT`, `MPI_ATTR_GET`, `MPI_ATTR_DELETE` are not supported. These functions are deprecated since MPI 2.0 and the MPI standard does not define an F08 binding for them. Please update your program to use the appropriate MPI 2.0 replacements. - Due to the delayed call to `MPI_FINALIZE`, user-defined attribute deletion callbacks that are invoked during MPI finalization might see invalid values for `extrastate`. Either invoke the deletion callback explicitly before `MPI_FINALIZE`, or declare the actual argument for `extrastate` as `SAVE`. - When sampling is active, the Score-P internal function `scorep_in_measurement_increment_fromF08` might be recorded in the profile. - `MPI_ALLTOALLW` allocates temporary arrays for type conversions from F08 to C. On large communicators, this might consume a large amount of memory. - Compilation of Score-P fails with oneAPI's mpiifx (several versions affected) due to an error in their F08 bindings. A workaround is described here: https://community.intel.com/t5/Intel-MPI-Library/MPI-f08-with-polymorphic-argument-CLASS/m-p/1590421/highlight/true#M11660 -------------------------------------------------------------------------------- * OpenMP support Applies to OPARI2 and OMPT instrumentation: - Changes introduced with the OpenMP 6.0 specification are not supported. Applies to OPARI2 instrumentation only: - Device directives, introduced with OpenMP 4 and later, are not supported. - Function instrumentation using the Intel compiler version 11.1 for codes using OpenMP tasking is erroneous. - When compiling with the PGI compiler version 10.1, local variables that are defined after a OpenMP for construct share the same memory address. This breaks the OPARI2 instrumentation for task tracking and may lead to segmentation faults in the measurement system. Our recommendation is to use a newer compiler version, According to our tests, late compiler versions have fixed this issues. We tested with PGI compiler version 11.7. - Currently, the instrumenter allows to switch off OPARI2 instrumentation for OpenMP programs with the --noopenmp option. However parallel regions still need to be instrumented to ensure thread-safe execution of the measurement system. Currently combined constructs, such as parallel for/do, are still instrumented fully, i.e. the for/do appears in the measurements. - Due to a bug in the Cray compilers OpenMP instrumentation is broken if an OpenMP parallel pragma is used in combination with an if clause. - OPARI2-instrumented Fortran OpenMP codes that use compiler options to change the default name-mangling (XL compilers: -qextname, GNU: -fno-underscoring' and '-fsecond-underscore') might crash if OpenMP 3.0 ancestry API functions are not available (check with 'scorep-info config-summary | grep ancestry'). In this case OPARI2 instruments a thread private variable that, due to non-standard name-mangling, does not match the one used in the Score-P libraries. On AIX, a workaround is to manually rename it at link time (-brename:pomp_tpd_,pomp_tpd). - With PGI C++ v13.10 compiler, preprocessing of OpenMP codes using OPARI2 is not possible any longer as the compiler itself adds a '--preinclude omp.h' option to the call of pgcpp1. This leads to 'invalid redeclaration' errors. As a workaround, use the '--nopreprocess' instrumenter option. - The OPARI2 instrumentation and the preprocessing for OPARI2 instrumentation prepend some headers to source files which include stdint.h in C/C++ files. The behavior of this header changes whether macros the __STDC_CONSTANT_MACROS or __STDC_LIMIT_MACROS are defined. If these macros are defined in a header file or source file, the instrumentation will prepend the include directive before the macro definition. Thus, macros like UINT32_C are left undefined. As workaround, pass macros like __STDC_CONSTANT_MACROS or __STDC_LIMIT_MACROS on the command line. See also the Open Issues of OPARI2. - OPARI2-instrumented OpenMP code using user-defined reductions (UDR), introduced with OpenMP 4, might crash when the UDR includes execution of instrumented routines, which Score-P currently reports as events on invalid threads or mis-matching enter/exit events. - Score-P 5.0 introduced program-begin/-end events. In profiling mode, these manifest as enter/exit of an additional region which acts as the new root node for the program. Artificial regions like THREADS and TASKS are now recorded as child nodes of this new root node. When recording OpenMP tasks, the tasks are recorded as children of the TASKS node as well as children in the program's calltree. Metrics, time in particular, are double accounted. As a consequence, the new root node shows negative time values when expanded (the absolute value equals the inclusive TASKS value). The values shown for TASKS and the program's calltree are correct though. This situation will be improved in a future release of Score-P and/or Cube. - In C++, OpenMP clauses like private, firstprivate, and lastprivate call constructors, copy-constructors, and destructors for list elements of class type. Given OPARI2 instrumentation, if these functions trigger any event, Score-P will/might fail. For constructors and destructors we might see `TPD == 0` errors, for destructors it will be `Assertion 'parent != ((void *)0)' failed`. Preventing all events from these functions by runtime/compile-time filtering is a workaround. Future OpenMP instrumentation based on OMPT instead of OPARI2 does not show this problems. - Like other compilers, the NVIDIA HPC SDK compilers add outlined functions to implement OpenMP directives. These functions get potentially instrumented but need to be ignored by Score-P to prevent 'TPD==0' errors. Unfortunately, NVIDIA's naming convention for these functions isn't documented and changed between versions of the SDK. However, since version 21.2 of the SDK, the outlined functions appear to follow this naming scheme: __nv_*_F[0-9]*L[0-9]*_[0-9]* Outlined functions following this scheme are automatically filtered by Score-P. For SDKs predating 21.2, the outlined functions seem to be of the form _L. This isn't handled automatically by Score-P, but could be filtered with this very generic EXCLUDE rule: *_*L[0-9]* - With CMake, OpenMP might get enabled at link time not via a compiler flag like `-fopenmp`, but by explicit linking of the runtime, e.g., `-lgomp`. In this case, Score-P would not automatically add its OpenMP support libraries and the link would fail. As a workaround, please provide `--thread=omp` to the `scorep-wrapper` via `make SCOREP_WRAPPER_INSTRUMENTER_FLAGS="--thread=omp"`. - With CMake and NVHPC, the preparations for OPARI2 instrumentation might create empty source files due to the `-MD` flag added by CMake during compilation. The compilation will abort with the message `undefined reference to main`. For NVHPC 23.7 and newer, please use OMPT via `--thread=omp:ompt` instead. - With Cray compiler wrappers and an accelerator architecture loaded, CPE 23.09 or lower are not supported due to Clang-based compilers failing to compile the pre-processed output generated by the wrappers. Trying to build an OpenMP application with OPARI2 enabled will fail with compilation errors. Applies to OMPT instrumentation only: - When instrumenting an application using `#pragma omp sections` with Intel oneAPI compilers, enabling OpenMP via `-fiopenmp` or `-qopenmp` may result in an error reporting an `inconsistent profile`. This happens because the runtime reports the end of the sections construct incorrectly. For C / C++ only, you may switch to `-fopenmp` for instrumentation. For Fortran applications, please use OPARI2 via `--thread=omp:opari2`. - When instrumenting an application using `#pragma omp sections` with NVHPC 23.7, running the instrumented application will abort reporting an `inconsistent profile`. This happens because the runtime does not report the end of the sections construct. If you want to instrument applications using OpenMP sections with NVHPC, please use OPARI2 via `--thread=omp:opari2` instead. - When measuring an application using `#pragma omp ordered` with NVHPC, running the application may result in an error message like "Bug: 'task->workshare_regions_current == 0'" or crash with a segmentation fault. As this is a runtime issue and cannot be worked around, please use OPARI2 via `--thread=omp:opari2` to instrument applications using `#pragma omp ordered`. - When configuring Score-P with a Cray compiler and an accelerator architecture loaded on a node without the respective accelerator, configure checks might fail because no GPU was found. Please build on a node with the accelerator being present. Applies to OpenMP target offloading: - When using an LLVM-based compiler in an application using OpenMP target directives, the `scorep` compile command may fail with a linker error and the following error message: Undefined reference to '__cyg_profile_func_enter' Undefined reference to '__cyg_profile_func_exit' The error is caused by the `-finstrument-functions-after-inlining` flag. As a work around, you may try to add the following function declarations to each of your compile units: #pragma omp begin declare target device_type(nohost) void __cyg_profile_func_enter(void *a, void *b) __attribute__((no_instrument_function)); void __cyg_profile_func_exit(void *a, void *b) __attribute__((no_instrument_function)); void __cyg_profile_func_enter(void *a, void *b) { } void __cyg_profile_func_exit(void *a, void *b) { } #pragma omp end declare target On a compiler using LLVM 16.0 or newer, you may try to use these flags: scorep --nocompiler [llvm-based compiler] [args] \ -Xarch_host -finstrument-functions-after-inlining - Runtimes have been reported creating helper threads for certain target directives. These helper threads may dispatch events in a non-conforming way. In addition, non-conforming tasking behavior has been observed. In these cases, Score-P can't ignore the device events, but aborts. As a workaround, please turn off these helper threads by setting the environment variable `LIBOMP_USE_HIDDEN_HELPER_TASK` to `0`. Known nonconforming runtimes include LLVM/Clang (12.0.0 to 18.1.7) and oneAPI (2024.1 and earlier). - For instrumentation of target directives, only runtimes having the OpenMP Tools Interface device tracing interface available are supported. This includes recent ROCm and AOMP compilers. For other runtimes, please try to use the native accelerator adapters for instrumentation. - For best support, it is advised to use the latest compiler versions to ensure the best support of the device tracing interface. In our testing, AMDs ROCm 6.2.x and AOMP 20.0-0 offer the best support, with the following limitations: - AOMP 19.0-0 will report incorrect times for data transfers between devices. - AOMP 18.0-1 may deadlock for short programs when multiple accelerators are initialized. - ROCm 6.1.3 and earlier and AOMP 18.0-0 and earlier do not support multiple devices per rank. If kernels are executed on more than one device per process, execution may abort. Otherwise events may be associated with the wrong accelerator. - AOMP 18.0-0 incorrectly maps identifiers between callbacks and the device tracing interface. This leads to data transfers being shown incorrectly between the host threads and devices. - ROCm 5.7.1 and earlier and AOMP 17.0-3 and earlier do not support accessing a device from multiple threads. This may lead to issues where events are associated with the incorrect host thread. - ROCm 5.6 to 5.7.1 do not dispatch all callbacks for `#pragma omp target enter/exit data`. Score-P will abort due to timestamp issues. - When utilizing multiple accelerators with ROCm 5.5, execution will dead lock at the end of the program execution when Score-P calls `stop_trace` for the device tracing interface. - ROCm 5.4 and earlier are not supported due to not having a way to translate the device time to host time. In addition, Cray CCE 17.0.1 and newer support the device tracing interface, with the following known limitations. These limitations will be improved in a future release of Score-P: - Cray CCE 17.0.1 may not dispatch outstanding buffers at the end of the program, and may not dispatch a device finalization event before the measurement has already finished. This may lead to missing events. - Cray CCE 17.0.1 may dispatch buffers twice for the same event, when explicit flushes are done. This will lead to the measurement aborting with an error message since too many events for one target region are processed. If the application crashes or hangs because of the mentioned issues, please try to disable the device tracing interface by setting `SCOREP_OPENMP_TARGET_DEVICE_TRACING_ENABLE=false` and record events with the respective native accelerator adapter. - Not all runtimes offering support for the OpenMP Tools Interface support dispatching events for target directives and/or the device tracing interface to report accelerator events. In our testing, LLVM/Clang 17.0, oneAPI 2024.1, NVHPC 23.7, ROCm 5.4, and newer versions have at least partial support for target directives. If the device tracing interface cannot be enabled, Score-P will report a warning. In this case, please try to record events using the respective native accelerator adapter, e.g., CUDA. - When instrumenting an application using OpenMP tasks within a teams region, using `#pragma omp single nowait` or `#pragma omp masked` might cause Score-P to abort when using LLVM based compilers due to missing implicit barriers in the runtime. As a workaround, please use `#pragma omp single` instead. - When instrumenting an application with NVHPC 24.11 or 25.1, the OpenMP runtime may dispatch a target-begin event before device initialization. This was observed with codes, where no data transfer is done, or small variables on the stack were copied. Since Score-P requires having OpenMP target devices initialized when a target-begin event is dispatched, the program will abort. Please disable the instrumentation of target directives via `SCOREP_OPENMP_TARGET_ENABLE=no` and record events with the CUDA adapter instead. -------------------------------------------------------------------------------- * CUDA support - Support for CUDA 4.2 and prior versions was deprecated in Score-P 7.0 and removed in Score-P 8.0. Since 8.0, Score-P requires CUDA 7.0 or newer. If an older CUDA version is used, the CUDA adapter will not be built. This requirement will be bumped to CUDA 11.6 in a future release. - The `nvcc` compiler preincludes its CUDA runtime header. This results in multiple defined symbols if the source file is preprocessed before compilation. Thus enabling preprocessing and CUDA simultaneously is not allowed by the instrumenter. - Score-P's NVTX implementation does not support RangeStart/Stop APIs. They are supposed to cross locations and can overlap, which does not fit into Score-P's event model. - NVTX does consider different domains as disjoint range stacks, thus this does not fit into Score-P's event model as well. Score-P expect that regions from disjoint domains are properly nested. - Score-P's NVTX implementation might require setting `LD_PRELOAD` for instrumentation to work correctly if Score-P is built with NVTX v2 or lower (CUDA 9.0 and lower). This happens because `libnvToolsExt.so` might be loaded before Score-P's CUDA adapter. The required flags can be obtained via `scorep-preload-init --cuda [app]`. - Thread migration of a context is not supported. Affected memory copies are not recorded. This will be improved in future Score-P versions. - Unified memory is not supported. The CUDA adapter might report that memory allocations could not be found or memory was freed that has not been allocated. Data transfers are not shown. This will be improved in future Score-P versions. - CUDA graphs are not supported. The CUDA adapter might report that hash values could not be retrieved or matching failed. Kernels are still recorded, but allocations and data transfers are not. This will be improved in future Score-P versions. - CUDA calls transferring memory introduced after CUDA 7.0 might not be supported correctly. The CUDA adapter might report that the memory copy kind could not be identified. This includes 2D/3D memory transfers. This will be improved in future Score-P versions. - The CUDA adapter is not able to record kernels associated with OpenMP target directives to NVIDIA GPUs with LLVM 18.1 or newer, as the runtime is creating streams before the Score-P CUDA adapter can be initialized. Future versions of Score-P will handle this more gracefully. -------------------------------------------------------------------------------- * SHMEM support - When using the OpenSHMEM reference implementation and building Score-P as a shared library, ensure that the GASNet library is build with -fPIC on platforms which need this flag for shared library code. For example, run make with the MANUAL_LIBCFLAGS and MANUAL_CFLAGS variables set: make MANUAL_LIBCFLAGS="-fPIC -DPIC" MANUAL_CFLAGS="-fPIC -DPIC" on GNU/Linux platforms. Also, if you encounter segmentation faults when running the instrumented application with Score-P attached, but the error disappears magically on subsequent runs, then the OpenSHMEM reference implementation may not consider some of your global or static variables as symmetric objects. Please allocate these objects with shmalloc() to ensure that they are in the symmetric heap. - Since version 1.0g the OpenSHMEM library is compiled as a statically linked archive, rather than as a shared object. If you want to build Score-P as a shared library, ensure that the OpenSHMEM archieve is build with -fPIC. - Currently tested SHMEM implementations: + OpenSHMEM reference implementation 1.0h + Open MPI implementation up to 4.1.1 + SGI SHMEM + Cray SHMEM - When using the Cray SHMEM versions 7.2.2 or 7.2.3, then Score-P fails to detect a number of SHMEM functions. Please use a newer version of the module to circumvent this problem. -------------------------------------------------------------------------------- * POSIX threads support - Pthread support is currently not available on systems that use linkers other than the GNU linker, e.g., AIX systems. - Please note that on systems where Pthreads or runtimes based on Pthreads don't need extra flags (like e.g., -pthread) to be compiled and linked (e.g., Cray, Blue Gene/Q), Score-P cannot instrument Pthreads automatically. You need to enable Pthread instrumentation manually via scorep's --thread=pthread option. -------------------------------------------------------------------------------- * Sampling support - Sampling is mostly tested on x86-64 CPUs. The compiler needs to support thread-local storage, either via the language extension __thread or the C11 keyword _Thread_local. This excludes PGI compilers prior to 'Version 2015'. - Sampling uses libunwind as external library. Please use version 1.2 or later as earlier versions occasionally segfaulted. - libunwind used in combination with Intel MPI, even when sampling is not active, may mysteriously alter the application output just by linking libunwind. Thus, sampling/libunwind support is disabled when Intel MPI is detected. - It is possible to activate sampling for programs that were instrumented with compiler instrumentation. In this case the events originating from the compiler instrumentation will be suppressed. Please note that depending on the compiler instrumentation used, the overhead can still be significant as the compiler instrumentation might affect inlining. If this is the case, consider rebuilding your application using Score-P's --nocompiler switch. - Thread management events and intermediate trace buffer flushes may result in unexpected callpaths. - CUDA API calls are represented twice in the calling context. - If PAPI is used as the interrupt source for sampling, then it is not possible to measure also PAPI metrics. - The timer interrupt source only works for non-threaded applications. - Flushing the trace buffer from inside a sample signal is not allowed and thus aborts the measurement immediately. - Untested feature combinations with sampling: - OpenMP tasks - Trace rewinding - SCOREP_RECORDING_ON/SCOREP_RECORDING_OFF user instrumentation macros - Selective recording (SCOREP_SELECTIVE_CONFIG_FILE) -------------------------------------------------------------------------------- * User library wrapping - Functions returning function pointers are not supported, e.g.: int ( *fn_x_return_function_pointer( void ) )( double ); A typedef works around this limitation, e.g.: typedef int ( function )( void ); function fn_x_return_function_pointer( double ); - Nested Function pointer arguments are not supported, e.g.: int sion_generic_register_gather_and_execute_cb( int aid, int gather_execute_cb( const void *, long long*, int, long long, void *, int, int, int, int, int process_cb( const void *, long long *, int ) ) ); Typedefs work around this limitation, e.g.: typedef int ( process_function )( const void *, long long *, int ); typedef int ( gather_execute_function )( const void *, long long*, int, long long, void *, int, int, int, int, process_function process_cb ); int sion_generic_register_gather_and_execute_cb( int aid, gather_execute_function gather_execute_cb ); - Typedefs from classes need to be fully qualified when used, e.g., convert: class C { typedef int mytype; void f(mytype a); }; into: class C { typedef int mytype; void f(C::mytype a); }; - LLVM upto version 3.9 does not support the ABI tag attribute from GCC 5 and onwards. Thus creating library wrappers for symbols with ABI tags is impossible. The issue is tracked in LLVM here: https://llvm.org/bugs/show_bug.cgi?id=23529 - Building and using library wrappers for Intel Xeon Phi co-processors is untested. -------------------------------------------------------------------------------- * I/O recording - ISO C I/O support - fsetpos: Seek events provides only the resulting offset - ungetc not supported yet - POSIX I/O support - Redirecting stdout/stderr to a file via dup2 will not affect the ISO C I/O. The recorded I/O of e.g. puts will target the stdout instead of the file. - O_NONBLOCK and O_NDELAY flags are not handled in open operations because of the following note (see manpage): Note that this flag has no effect for regular files and block devices; that is, I/O operations will (briefly) block when device activity is required, regardless of whether O_NONBLOCK is set. Since O_NONBLOCK semantics might eventually be implemented, applications should not depend upon blocking behavior when specifying this flag for regular files and block devices. - The mode of an open operation will not be tracked. e.g. open(..., mode_t mode); - putc will not be recorded because on most system it is macro. Furthermore, it is difficult to detect it as a macro on Cray systems. -------------------------------------------------------------------------------- * Kokkos support - Currently only one accelerator device is supported. To ensure that for this device a Score-P location was created before first deep-copy, enable recording of memory allocations in the corresponding accelerator adapter of Score-P. - Kokkos/OpenMP support should be considered experimental until OMPT support is added to Score-P. This affects principally the Kokkos adapter. - Kokkos/Pthreads should be considered unsupported. It will produce a measurement, but probably not a meaningful one. This affects instrumentation of Kokkos-based codes in general, with or without use of the Kokkos tools interface. - In order for non-GPU-based back ends to support Kokkos-based allocation tracking and memory copy instrumentation, it is necessary to ensure that the threads where memory is allocated or copied are instrumented. For best chances of success there, we recommend using Score-P's '--thread=pthread' option for the OpenMP back end and a static build of Kokkos. This affects use of the SCOREP_KOKKOS_ENABLE=memcpy,malloc settings in the Kokkos adapter. If instrumenting a different configuration (e.g. a shared Kokkos library) is important for your use case, please contact support@score-p.org for help. -------------------------------------------------------------------------------- * HIP support - Score-P is only able to use one compiler suite for one installation. Because `hipcc` is Clang based, it also only accepts the Clang compiler instrumentation. Thus Score-P must be build with a Clang-based compiler suite. If the compile step for device code errors out with: lld: error: undefined symbol: __cyg_profile_func_enter >>> referenced by : then the heuristic to detect the usage of `hipcc` failed and must be enabled explicity with `scorep --hip`. - Device-to-device data transfers are only recorded on the API level, not as a device activity. - Setting compile flags via the environment variable `HIPCC_COMPILE_FLAGS_APPEND` or link flags via the environment variable `HIPCC_LINK_FLAGS_APPEND` may interfere with correct instrumentation, most likely resulting in an empty experiment directory when MPI is used. Please pass the desired compile/link flags explicitly instead. If this is not possible, try to preload the Score-P libraries via `LD_PRELOAD=/lib/libscorep_adapter_mpi_event.so`. -------------------------------------------------------------------------------- * OpenACC support - Score-P may generate OpenACC related events if the compiler supports the OpenACC Profiling (and Error) Callback Interface. It is a Score-P requirement that the OpenACC runtime registers a tool by linking a library that contains the function `acc_register_library`. Tool registration via `ACC_PROFLIB` or `LD_PRELOAD` is not supported. Currently, only NVIDIA compilers are known to support the interface and to provide events to Score-P. With GNU compilers, a tool can be compiled, but it won't be registered by linking `acc_register_library` to the program. -------------------------------------------------------------------------------- * Score-P misc. - Score-P does not support MPMD style programs where the executables are instrumented differently. I.e., it is best to compile/link all sources with the same set of flags for the Score-P instrumenter, so that the instrumenter itself cannot automatically decides what paradigms the program uses. This applies to the --mpp and --thread flags. - There might be a performance impact when instrumenting code without explicitly given optimization flags. The instrumenter adds compiler flags to enable additional debugging information. Depending on the compiler this may turn off optimization unless optimization flags are explicitly specified. - For autotools or CMake based build systems, please consult the usage of `scorep-wrapper` to learn about the recommended way to apply instrumentation. - Literal file-filter rules like "INCLUDE bt.f" for Fortran files that will be processed by OPARI2 (i.e., files containing OpenMP or POMP user pragmas) do not work as expected as OPARI2 changes the file name (here to bt.opari.f) - Rusage-based metrics are not supported on Blue Gene systems. - Traces generated by applications compiled with the CCE Fortran compiler on Cray X series systems are inconsistent as there is no exit event for 'main'. I.e., such traces are not analyzable by Scalasca. - Running make check fails if SCOREP_EXPERIMENT_DIRECTORY is set to 'scorep'. One workaround is to run '(unset SCOREP_EXPERIMENT_DIRECTORY; make check;)'. - In some cases the PGI 11.7 compiler fails to link C++ programs that use user-instrumentation macros due to an undefined reference to '_Unwind_Resume'. Adding -lstdc++ to the original link line solves this issue. - The Cube metric hierarchy remapping specification file shipped with Score-P currently classifies all MPI request finalization calls (i.e., MPI_Test[all|any|some] and MPI_Wait[all|any|some]) as point-to-point, regardless of whether they actually finalize a point-to-point request, a file I/O request, or a non-blocking collective operation. This will be addressed in a future release. - When instrumenting memory API calls with the Cray CCE, Score-P uses the '-h system_alloc' flag and therefore not the default tcmalloc library (note that -h tcmalloc was deprecated in CCE-15). - Tracking memory allocations in Fortran programs may only work, if the compiler generate calls to malloc/free. This is only known for the GNU compiler. - Score-P uses an 'atexit' handler to finalize the measurement and produces the output files. In case a run-time system also uses an 'atexit' handler but does not allow previous registered 'atexit' handlers to run, then one can try to re-install the 'atexit' handler again, so that Score-P's will run before the malicious handler. If that still does not work one can also trigger the finalization manually. Here are the C prototypes for these two functions: void SCOREP_RegisterExitHandler( void ); void SCOREP_FinalizeMeasurement( void ); One known malicious run-time system is NetDRMS. - Instrumentation of files that include system headers via local headers of the same name fails for compilers that don't support the '-iquote' option (e.g., PGI). In this case, renaming of the local header is the only workaround. - Recording arguments of the executable was disabled due to unreliable values on some old HPC platforms now decommissioned. If there are program aborts inside Score-P, please do make check and installcheck and exercise the constructor checks: $ make $ make -C build-backend check-build $ make -C build-backend check-serial TESTS_SERIAL="test_constructor_check_c test_constructor_check_cxx test_constructor_f" $ ./build-backend/test_constructor_check_c 1 2 3 $ ./build-backend/test_constructor_check_cxx 1 2 3 $ ./build-backend/test_constructor_check_f 1 2 3 $ make install $ make -C build-backend constructor-checks $ ./installcheck/constructor_checks/bin/run_constructor_checks.sh - For AMD's AOCC 4.1.0 and newer (tested till 5.0.0), AOCC's clang compiler prepends its library directory to user provided library directories if -fopenmp is given. As a consequence, AOCC's incompatible libunwind is linked instead of the one provided to Score-P's configure. This fails with undefined symbol errors. One could configure --without-libunwind to work around this compiler bug. Still, there might be subtle differences, as not only libunwind is found and silently linked in AOCC's library directory. The same applies to AMD's ROCm compiler amdclang in version 6.1.2 and newer. -------------------------------------------------------------------------------- Please report bugs, wishes, and suggestions to .