A testing environment for the Yggdrasil workflow. It wires the submodules in this repository into a single Docker Compose stack, starts Yggdrasil only after its dependencies are up and running, and (by default) fires an end-to-end test scenario so you can see the whole event-driven pipeline work: CouchDB change → plan generation → plan persistence → plan execution → event/ops artifacts.
┌────────────────────────── compose network ──────────────────────────┐
│ │
:5984 ───────────►│ statusdb CouchDB 3.4, seeds projects/gs_users/... on first boot │
│ ▲ │
│ │ http://statusdb:5984 │
:8765 ───────────►│ biomate BioMate web interface (only with --profile full) │
│ │
:9761 ───────────►│ genomics-status (default stack) │
│ ▲ │
│ │ (stub config → statusdb) │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ yggdrasil daemon (yggdrasil --dev daemon) │ │
│ │ depends_on: statusdb, dind, slurmctld, slurmd (healthy) │ │
│ │ entrypoint: wait CouchDB → wait seeded DBs → create ygg DBs │ │
│ │ → inject test scenario → start daemon │ │
│ └────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────┘
├── docker-compose.yml # the stack (core: statusdb + yggdrasil + dind + genomics-status + slurmdbd-db + slurmdbd + slurmctld + slurmd; profile full adds biomate)
├── deploy/
│ ├── Dockerfile.yggdrasil # Yggdrasil daemon image (no Dockerfile in the submodule)
│ ├── yggdrasil-entrypoint.sh # readiness gate + DB bootstrap + scenario injection
│ ├── yggdrasil-config/main.json
│ ├── seed/test_scenario_happy_path.json
│ ├── seed/test_scenario_slurm.json
│ ├── slurm/slurm.conf # shared config of the local SLURM cluster
│ ├── slurm/entrypoint-slurm-ctld.sh
│ ├── slurm/entrypoint-slurmd.sh
│ ├── Dockerfile.slurm-ctld # SLURM controller (Debian trixie slurmctld)
│ ├── Dockerfile.slurmd # SLURM compute node (Debian trixie slurmd)
│ ├── seed/flowcell_status_biomate.json
│ ├── seed/demux_sample_info_biomate.json
│ ├── seed/demux_sample_info/ # test-flowcell seeds, bind-mounted into statusdb
│ ├── seed/flowcell_status/ # test-flowcell seeds, bind-mounted into statusdb
│ ├── Dockerfile.genomics-status # builds on top of the submodule's conda-env image
│ └── genomics-status/ # stub settings/credentials for genomics-status
├── Makefile # submodule management + compose-* targets
├── pyproject.toml / pixi.lock # pixi workspace aggregating submodule deps
├── extract_deps.py
├── BioMate/ # submodule
├── demux_realm/ # submodule (Yggdrasil realm dmx_realm, installed in the Yggdrasil image)
├── genomics-status/ # submodule
├── StatusDB_NGI/ # submodule
└── Yggdrasil/ # submodule
At runtime the submodules are never modified by this repository: images are
built from their Dockerfiles (or, for Yggdrasil, from a Dockerfile in
deploy/), and all runtime configuration is baked into images or bind
mounts rather than written into submodule working trees.
- Docker Engine with the Compose plugin (v2+). Tested with Docker 29.8 and Compose v5.5.
- git (submodules)
- Optional: pixi, only for the submodule dependency workflow.
First-time builds pull base images and install dependencies, so expect a several-minute build. Subsequent builds are fast (layer caching).
git clone [email protected]:NationalGenomicsInfrastructure/dataflow_testing.git
cd dataflow_testing
make init # clone/update the submodules
make compose-up # build + start statusdb, yggdrasil, genomics-statusThat's it. What happens automatically:
statusdbstarts CouchDB and (on a fresh volume) seeds theprojects,gs_users,gs_configs,server_statusdatabases in the background — plusflowcell_statusanddemux_sample_infowith four test flowcells each, fromdeploy/seed/(bind-mounted into the container; seedocker-compose.yml).yggdrasilstarts only afterstatusdbreports healthy (depends_on: condition: service_healthy), then its entrypoint:- waits until CouchDB answers
/_upand the seededprojectsDB exists (CouchDB being "up" is not enough — the seed runs afterwards), - creates the
yggdrasil,yggdrasil_plans,yggdrasil_ops,yggdrasil_testdocsdatabases (Yggdrasil fails fast at startup if any of its databases are missing), - injects a
happy_pathtest scenario document (skip withYGG_INJECT_TEST_SCENARIO=false), - starts
yggdrasil --dev daemon(--devenables the built-intest_realmused for testing).
- waits until CouchDB answers
- The daemon replays the changes feed from the beginning, picks up the
scenario, generates a plan from the
happy_pathrecipe, persists it toyggdrasil_plans, executes it via the Engine, and writesplan_statusdocuments toyggdrasil_ops. Step artifacts land in theygg-workandygg-eventsvolumes.
(The biomate web interface on port 8765 only comes up with
make compose-up-full — profile full; nothing in the core stack needs
it, so it is no longer a yggdrasil dependency.)
make compose-ps # all services "healthy", yggdrasil "Up"
# (the one-shot slurmdbd-bootstrap has exited with 0)
make compose-logs # follow the daemonExpected log lines:
Generating plan from recipe 'happy_path' for scenario 'test_scenario:compose-demo'
Persisted plan 'test_realm:test_scenario:compose-demo' to yggdrasil_plans
Eligible plan detected: test_realm:test_scenario:compose-demo (authority=daemon, ...)
Executing plan ...
✓ Plan 'test_realm:test_scenario:compose-demo' execution completed
And from the host:
# the executed plan document
curl -s -u "${STACK_COUCH_USER:-admin}:${STACK_COUCH_PASSWORD:-secret}" \
"http://localhost:5984/yggdrasil_plans/_all_docs?include_docs=true"
# step artifacts and the event spool
docker compose exec yggdrasil find /work /events -type fUIs on the host:
| Port | Service |
|---|---|
| 5984 | CouchDB (StatusDB) — _all_docs, _changes, etc. |
| 8765 | BioMate web interface (only in the full profile) |
| 9761 | Genomics Status (status dashboard UI) |
The daemon stays running and watches CouchDB, so you can inject more scenarios any time:
make compose-scenario # PUTs a fresh test_scenario:manual-<timestamp> docThe running daemon picks it up within a few seconds (watch make compose-logs).
Each injection gets a unique document id, so it always executes; the initial
test_scenario:compose-demo is injected only once and is not re-executed
on restarts (plans are tracked by run_token/executed_run_token and
watcher checkpoints are persisted in CouchDB). Use make compose-reset for a
completely fresh state.
The stack runs a real single-node SLURM cluster (slurmctld + slurmd
services; see deploy/slurm/slurm.conf) as a stand-in for an HPC, and the
yggdrasil container carries the SLURM client (sbatch/sacct) plus a
main.json setting of yggdrasil.slurm.backend = "slurm". The
slurm_job test recipe submits a batch job through Yggdrasil's
SlurmJobManager and waits for its terminal state:
make compose-scenario-slurm # PUTs a fresh test_scenario:slurm-<timestamp> docWatch the plan run with make compose-logs (the step records the
job_id and final_state in its metrics); from inside the stack you can
inspect the job the usual SLURM way, e.g.
docker compose exec slurmctld sacct -j <job_id>.
By default dmx_realm runs each branch's Nextflow invocation as a
subprocess of the yggdrasil daemon process. With DMX_EXECUTOR=slurm the
realm instead submits each branch as an sbatch job to the stack's
SLURM cluster — the same SlurmJobManager (sbatch/sacct) the
slurm_job scenario uses, and the shape of a production HPC deployment:
make compose-demux-slurm # recreate yggdrasil with DMX_EXECUTOR=slurm
make compose-sync-flowcell FLOWCELL_ID=B98QWERTY6 # arm one flowcell (adds transferred_to_hpc)
make compose-demux-local # back to local execution (the default)The entrypoint materialises the flag as dev_dmx_realm.json
(nextflow.executor=slurm; in dev mode the realm's ConfigLoader prefers
dev_*.json over dmx_realm.json, and the file is removed again on a
local run), and the execute_demux step writes the branch's batch script
to its work directory, submits it and blocks until the job reaches a
terminal state — COMPLETED is expected, anything else fails the step
with the tail of the run log (on the shared volume). The node runs the
real pipeline: deploy/Dockerfile.slurmd provides the JRE, a pinned
Nextflow launcher and a static docker CLI pointing at the dind sidecar
(container execution for -profile docker still happens there), and
NXF_HOME lives on the shared /work volume, so the pipeline checkout
and the tool images are shared with the yggdrasil container. sbatch
resource defaults are configurable in the nextflow object of
deploy/yggdrasil-config/dmx_realm.json (sbatch_cpus, sbatch_time,
sbatch_partition); no --mem is requested, so the cluster's default
memory applies.
While the plan runs, the usual SLURM tools apply, and the per-branch script and log live on the shared volume under the plan directory:
docker compose exec slurmctld squeue # dmx-<fcid>-lane_N_settings_I
docker compose exec yggdrasil sacct -n -X -o JobID,JobName,State
ls /work/dmx_realm:<FCID>:demux/lane_*__execute_demux/ # nextflow_slurm.sh + nextflow_run.logOn a fresh stack the first run also git-fetches the pipeline and pulls the tool images into the dind daemon (one-time, ~3 GB); later runs reuse them. From a completely clean state:
make compose-reset
DMX_EXECUTOR=slurm make compose-up
make compose-sync-flowcell FLOWCELL_ID=B98QWERTY6The demux workflow (dmx_realm) operates on the documents in the
flowcell_status and demux_sample_info databases. A fresh stack is
seeded at boot with four test flowcells per database (from
deploy/seed/), and any change to one of those documents fires the
workflow. Two BioMate seed documents additionally live in
deploy/seed/*_biomate.json; compose-seed-status PUTs them only if
absent (idempotent) — on a fresh stack the documents already exist under
the same _ids, so it is a no-op unless the docs have been removed:
make compose-seed-status # PUTs both seed docs; idempotent (skips if present)Once a flowcell carries the transferred_to_hpc event (see below), any new
revision of either document re-triggers the workflow, which the recipes below
exploit:
make compose-trigger # re-touch both docs (repeatable)
make compose-trigger DBS=flowcell_status # or just one of them
make compose-inject # +1 random flowcell (fresh ids)
make compose-inject COUNT=3 LANES=4 # 3 flowcells, 4 lanes each
make compose-inject DIR=/path/to/runfolder # docs built from that run folder's
# SampleSheet.csv (upserted by flowcell_id)
make compose-sync-flowcell # add transferred_to_hpc to trigger demux
make compose-sync-flowcell FLOWCELL_ID=B98QWERTY6 # specify which flowcell to finalizecompose-inject generates the documents on the fly from the live docs in
the database (fetched as templates by deploy/seed/generate_flowcell.py):
random mode derives new flowcell/runfolder ids, remaps projects/samples and
rebases all timestamps to now (LANES= beyond the template's lane count
copies the template's lanes round-robin with fresh sample names); DIR= mode takes a real run folder (or a
directory containing one), so the injected documents carry that run's actual
samplesheet content instead of random data. Re-injecting the same run folder
replaces the existing documents and re-triggers the workflow. Injected
flowcells get the transferred_to_hpc event added, so they start right away.
The four seeded flowcells deliberately omit the transferred_to_hpc event, so
the demux workflow does not start on a fresh stack. Use compose-sync-flowcell
(with an optional FLOWCELL_ID= parameter) to add the event and trigger the
workflow for one flowcell. The default flowcell ID is A12GGASZR5.
Each test runfolder ships the expected FASTQs. The yggdrasil image moves
them from the runfolder's Demultiplexing/ to original_FASTQs/, since the
demux workflow writes its own output to Demultiplexing/. Once the plan has
finished, compare the two:
make compose-check-demux # default flowcell (A12GGASZR5)
make compose-check-demux FLOWCELL_ID=C36MNBVCX7 # one flowcell
make compose-check-demux FLOWCELL_ID= # all fourdeploy/seed/check_demux_results.py runs inside the yggdrasil container. It
reads original_FASTQs/ from the image copy (/Illumina/NextSeq) and
Demultiplexing/ from the staged runfolder (/work/Illumina/NextSeq), and
matches files on sample name, lane and read (directory layout and _S<n>
are ignored). Per file it compares the read count and the sorted sequences;
headers and qualities are not compared. Index-read FASTQs (I1/I2) are
skipped. Run one flowcell at a time: two concurrent demux plans fail on a
shared Nextflow session lock.
A /work volume staged by an image older than this layout still holds the
expected FASTQs in Demultiplexing/; the check then reports duplicates.
Restage it once with make compose-reset-work CONFIRM=1.
Limitation: Undetermined FASTQs are not checked, so reads wrongly left
unassigned only show up as a read-count mismatch in a sample file. Two
reasons: nf-core/demultiplex does not publish Undetermined_S0_* to the
results directory (they stay in the Nextflow work dir), and BCL Convert
writes them at full cycle length while the expected ones are trimmed per
OverrideCycles (shortened reads lose their tail, U8Y143 reads their UMI
head). Exit status is non-zero on any
mismatch, or when a flowcell has no results.
Scenario documents select a recipe via the recipe field
(deploy/seed/test_scenario_happy_path.json is the template; change recipe
and PUT to http://localhost:5984/yggdrasil_testdocs/<your-id>):
| Recipe | Behaviour |
|---|---|
happy_path |
all steps succeed (echo → sleep → echo) |
fail_fast |
first step fails |
fail_mid_plan |
fails partway through execution |
long_running |
extended sleep (timeout testing) |
artifact_write |
tests artifact registration |
genomics-status (the status dashboard UI on port 9761) is part of the
default stack: make compose-up builds its images and starts the container
alongside statusdb and yggdrasil. The yggdrasil entrypoint waits for it
before starting the daemon (soft gate, up to 300 s). It is not a compose
depends_on because it is a UI the daemon does not depend on — a hard gate
would couple daemon startup to its slower conda-python boot.
It is the heaviest service to build: the submodule Dockerfile only installs
the conda environment, so deploy/Dockerfile.genomics-status builds on top
of it and bakes in the app source plus stub configuration
(deploy/genomics-status/: settings.yaml, .genologicsrc, .genosqlrc.yaml,
lims_backend_cred.yaml, orderportal_cred.yaml). The stubs point LIMS /
order portal / Zendesk / Jira / Slack at mock URLs, and the app runs with
--testing_mode (no Google auth). It talks to the real statusdb CouchDB.
make compose-up-full additionally starts the biomate web interface
(profile "full").
Copy .env.example to .env to override any of these (the stack runs with
the defaults otherwise):
| Variable | Default | Purpose |
|---|---|---|
STACK_COUCH_USER |
admin |
CouchDB admin user for the stack. The STACK_ prefix deliberately avoids colliding with COUCHDB_USER that may exist in your shell for other purposes. |
STACK_COUCH_PASSWORD |
secret |
CouchDB admin password. |
HOST_UID / HOST_GID |
1000 |
User the biomate container runs as (upstream convention). |
YGG_INJECT_TEST_SCENARIO |
true |
Inject the happy_path scenario on container start. |
YGG_SCENARIO_ID |
test_scenario:compose-demo |
Document id of the injected scenario. |
DMX_EXECUTOR |
local |
dmx_realm's Nextflow executor (local or slurm). make compose-demux-slurm sets it for one recreate only; set it in .env to make it stick. |
The Makefile loads .env too, so the compose-* targets that talk to
CouchDB use the same credentials as the stack.
Where things live:
- Yggdrasil config:
deploy/yggdrasil-config/main.json(CouchDB endpoint, connectionsprojects_db/yggdrasil_db/yggdrasil_testdocs/demux_sample_info_db/flowcell_status_db, poll intervals). Bind-mounted over the image-baked copy at the path Yggdrasil'sConfigLoaderexpects. If you change it, restart the container:docker compose restart yggdrasil. - Genomics Status stubs:
deploy/genomics-status/(baked in the same way). Note the CouchDB credentials are hardcoded to the defaults there — if you changeSTACK_COUCH_*, updatedeploy/genomics-status/settings.yamltoo. - Volumes:
couchdb-data(CouchDB, incl. seed marker and Yggdrasil state),ygg-work(plan step artifacts),ygg-events(event JSON spool),ygg-logs(daemon log files),dind-data(pipeline tool images),slurm-munge(shared munge key),slurm-stateandslurmdbd-data(SLURM controller and accounting state).
scenario doc (type=ygg_test_scenario) in the `yggdrasil_testdocs` DB
→ WatcherManager (CouchDB changes feed, replays from start_seq=0)
→ test_realm handler generates a plan from the recipe
→ plan persisted to `yggdrasil_plans` (auto_run ⇒ approved)
→ PlanWatcher emits PLAN_EXECUTION
→ Engine executes the steps (work under YGG_WORK_ROOT=/work)
→ step events spooled to YGG_EVENT_SPOOL=/events
→ OpsConsumerService writes plan_status to `yggdrasil_ops`
Only test_realm is registered in this environment (it is the dev-only
testing realm; the production smartseq3/tenx realms are not part of the
test stack).
This repository doubles as a pixi workspace that aggregates the Python
dependencies of all submodules (see pyproject.toml / pixi.lock). Useful
targets:
| Target | Purpose |
|---|---|
make init / make init-<TARGET> |
Register missing submodules and initialize root module(s) at parent-pinned commits; clone missing genomics-status/nvd3 |
make deinit CONFIRM=1 / make deinit-<TARGET> CONFIRM=1 |
Remove all/one submodule checkout and local Git database |
make status / make status-<TARGET> |
Show submodule state |
make update-<TARGET> / make update-all |
Advance submodules without switching the parent branch |
make use-dev |
Switch the parent repo to dev and apply its pinned submodule commits |
make use-main |
Switch the parent repo to main (or master) and apply its pinned submodule commits |
make sync |
Sync submodule remotes with .gitmodules |
make reset CONFIRM=1 |
Destructively clean and reset submodules to their recorded commits |
make extract-deps |
Print a pixi add ... one-liner from the submodules' pyproject.toml files |
Deinitialization is destructive: it removes the submodule worktree (including
ignored files) and its local Git database under .git/modules/, including
local-only commits and refs. It refuses tracked or untracked changes and
requires CONFIRM=1; .gitmodules registrations are retained so the init
targets can restore them. Initialization only adds missing registrations and
does not overwrite existing URLs. A newly added submodule gitlink or repaired
.gitmodules entry may leave parent-repository changes to review and commit.
Initialization does not update submodules recursively. The nested
genomics-status/run_dir/static/nvd3 checkout is initialized only when absent;
once cloned, later make init runs leave its commit unchanged.
After updating submodules, make compose-up rebuilds the images from the new
code, including the genomics-status base (conda env) image.
| Target | Purpose |
|---|---|
make compose-up |
Build + start statusdb, yggdrasil, genomics-status |
make compose-up-full |
Same, plus biomate (profile full) |
make compose-up-statusdb |
Build + start only statusdb |
make compose-up-biomate |
Build + start only biomate (profile full) |
make compose-up-yggdrasil |
Build + start yggdrasil (also starts its deps: statusdb, dind and the SLURM services) |
make compose-up-genomics-status |
Build + start genomics-status (also starts statusdb) |
make compose-build-gs-base |
Build only the genomics-status base image (conda env) |
make compose-logs |
Follow yggdrasil logs |
make compose-ps |
Stack status |
make compose-scenario |
Inject a fresh test scenario into the running stack |
make compose-seed-status |
PUT the BioMate flowcell_status + demux_sample_info seed docs if absent (idempotent; a no-op on a fresh stack) |
make compose-trigger |
Re-touch the seeded docs (timestamp → now) to re-fire the demux workflow; DBS= selects docs |
make compose-inject |
Inject fresh generated flowcells; COUNT= (random mode), LANES=, DIR= (build docs from a run folder) |
make compose-sync-flowcell |
Add transferred_to_hpc event to trigger the demux workflow for a seeded flowcell (default: A12GGASZR5) |
make compose-sync-flowcell FLOWCELL_ID=<id> |
Specify which flowcell to finalize (e.g., B98QWERTY6, C36MNBVCX7, D74LKJHGF8) |
make compose-check-demux |
Compare the FASTQs in Demultiplexing/ with the expected ones in original_FASTQs/; FLOWCELL_ID=<id>, empty = all |
make compose-down |
Stop the stack (volumes kept) |
make compose-reset |
Stop the stack and remove all volumes (fresh CouchDB/plan state) |
make compose-reset-work CONFIRM=1 |
Clear the /work volume, then restart yggdrasil so its entrypoint restages the test data |
Network … Resource is still in use(ondown) orfailed to set up container networking: network … not found(onup) — a profile-gated container (e.g.biomatefrom a previouscompose-up-full) was left behind by a default-profiledown, keeping the shared project network alive (or stranding the container without it).make compose-down/make compose-resetnow pass--profile fulland cover all services; if you hit this with an older checkout, rundocker compose --profile full down --remove-orphansonce, thenmake compose-up/make compose-up-full.- Port already in use (5984/8765/9761) — something else is bound to the
host port. Change the host-side port in
docker-compose.yml(left of the colon) or stop the conflicting process. - yggdrasil stuck at "Waiting for CouchDB" — check
docker compose psfor statusdb health. Credentials are taken fromSTACK_COUCH_USER/STACK_COUCH_PASSWORDand mirrored into the yggdrasil container asYGG_COUCH_USER/YGG_COUCH_PASS; if you changed one side only, they no longer match. - Stale plans/scenarios from earlier runs —
make compose-resetwipes the CouchDB volume (and all Yggdrasil state); the nextupre-seeds. Configuration file 'module_registry.json' not foundin the logs — the yggdrasil image is older than the current setup; rebuild withmake compose-up.- Genomics Status 500s on
/— its stub config is baked into the image; after editing anything underdeploy/genomics-status/, rebuild withmake compose-up-full. - Submodules have uncommitted local changes —
use-*andupdate-*leave dirty submodules at their current commits and report that the requested pin or update was skipped.make reset CONFIRM=1is the explicit destructive option: it cleans untracked files and checks out the commits recorded by the parent. The compose stack never writes into submodule trees, so any dirt there came from manual edits.
| Submodule | Upstream | Role in the stack |
|---|---|---|
StatusDB_NGI |
NationalGenomicsInfrastructure/StatusDB_NGI | CouchDB image + core seed data (statusdb); the test-flowcell seeds for flowcell_status / demux_sample_info live in this repo's deploy/seed/ and are bind-mounted in |
Yggdrasil |
NationalGenomicsInfrastructure/Yggdrasil | The orchestration daemon under test |
demux_realm |
NationalGenomicsInfrastructure/demux_realm | Yggdrasil realm for demux planning; installed into the Yggdrasil image as realm dmx_realm (--no-deps), watches the demux_sample_info + flowcell_status DBs (bootstrapped by the yggdrasil entrypoint). Requires the Yggdrasil dev branch (handler API generate_plan_drafts) |
BioMate |
NationalGenomicsInfrastructure/BioMate | Bio-data tooling + web interface (biomate) |
genomics-status |
NationalGenomicsInfrastructure/genomics-status | Status dashboard UI (genomics-status) |