Skip to content
ParCoreLabPublic

About

ParcoreLab's GPU Communication Profiling and Visualisation Suite. Comshu in Turkish means neighbor. comshu monitors what GPU neighbors are saying to each other.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

comshu

comshu in Turkish means neighbor. With communication tools, under one roof, we can observe what GPU neighbors are saying to each other.

comshu records how a GPU program communicates. It captures InfiniBand traffic, GPU-initiated InfiniBand, UCX, NCCL collectives, CUDA kernels and NVLink peer load/store from a running workload, then reads the capture back as tables you can export, query, or open in a browser.

Install

comshu carries its own spack package repository, so spack needs no fork:

spack repo add https://github.com/ParCoreLab/comshu.git
spack install comshu +ibverbs +ucx

Every capture provider is a variant, and all of them are off by default:

Variant Captures
+ibverbs InfiniBand verbs, host side
+mlx5dv GPU-initiated InfiniBand (IBGDA)
+ucx UCX, both the application and the transport layer
+nccl NCCL collectives
+cupti CUDA kernels and memory copies
+duckdb Parquet export and SQL over a capture
+ui The panels in a browser, with comshu ui

+nccl builds against NCCL, which asks for a GPU target:

spack install comshu +nccl ^nccl cuda_arch=80

The NVLink provider (gpumem) is not in the package. It needs NVBit and its own CUDA build tree, so build that one from source.

Build

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
ctest --test-dir build

The core and its tests build on a host with no CUDA and no verbs. Providers are turned on one at a time and need their vendor SDK:

cmake -B build -DCOMSHU_PROVIDER_IBVERBS=ON -DCOMSHU_PROVIDER_UCX=ON

cmake --install build writes a tree that locates its own libraries, so the installed comshu needs no environment variables.

Usage

Capture a run. Providers are picked by name, and most of them share one pass:

comshu list                              # what this build can capture
comshu run -p ib      -- ./app args      # InfiniBand traffic
comshu run -p gpumem  -- ./app args      # NVLink peer load/store
comshu run -p ib,nccl -- ./app args      # both, in one run

Read the capture back:

comshu read comshu-out                    # what is in it
comshu export comshu-out --csv out.csv    # tables as CSV
comshu export comshu-out --parquet out    # tables as Parquet
comshu sql out "select count(*) from ib_post_send"

Visualize it:

comshu ui comshu-out                      # panels at 127.0.0.1:8777

sql needs +duckdb, and ui needs +ui, which turns on +duckdb too. The page is compiled into the binary, so comshu ui needs no other files. This is the usual way to read a cluster capture: record it there, copy the directory home, open it here.

For the layout, the gates, and how to add a record type or a provider, see docs/development.md.

Related Papers

Comshu brings together our communication profiling tools published in the following papers:

About

ParcoreLab's GPU Communication Profiling and Visualisation Suite. Comshu in Turkish means neighbor. comshu monitors what GPU neighbors are saying to each other.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages