Skip to content

Template System Reference

This page documents the template YAML format and resolution mechanics. For browsing available templates and quick start guides, see the Template Catalog.

What is a Template?

A template is a template.yaml file that ships inside Depictio under depictio/projects/<pipeline>/<version>/. It extends a standard project YAML with:

  • A template: metadata block — variable declarations, conditionals, dashboard references
  • {VAR_NAME} placeholders throughout the project config — resolved at runtime
  • References to bundled dashboard YAML files — imported automatically
  • References to recipes for transformed data collections

Template Variables

Every template declares its own variables. Variable names are template-specific — each pipeline decides what it needs.

DATA_ROOT is the only universal variable (always required, set via --data-root). All others are passed via --var KEY=VALUE.

Auto-detected variables: When a metadata file is provided, the system reads its headers and auto-populates:

  • GROUP_COL — first non-ID annotation column (overridable)
  • GROUP_COL_DISPLAY — title-cased version for chart labels
  • ANNOTATION_COLS — comma-separated list of all annotation columns

Template YAML Structure

template:
  template_id: "org/pipeline/version"
  description: "Human-readable description"
  version: "1.0.0"

  variables:
    - name: "DATA_ROOT"
      description: "Pipeline output root directory"
      required: true
    - name: "OPTIONAL_VAR"
      description: "An optional variable"
      required: false

  dashboards:
    - "dashboards/base.yaml"

  conditional:
    - if_var_absent: "OPTIONAL_VAR"
      remove_dc_tags: ["optional_dc"]
      dashboards: ["dashboards/base.yaml"]
    - if_var_present: "OPTIONAL_VAR"
      dashboards: ["dashboards/base.yaml", "dashboards/extended.yaml"]

Below the template: block is a standard project configuration with {VAR_NAME} placeholders.

Conditionals

Conditionals adapt the project based on which optional variables the user provides. Each rule can:

  • Remove data collections (remove_dc_tags) — and automatically prune any links referencing them
  • Override the dashboard list (dashboards) — select different dashboard variants

Rules fire on if_var_absent (variable not provided) or if_var_present (variable provided).

Source Overrides

When a recipe's source paths depend on a template variable, use source_overrides:

transform:
  recipe: "org/pipeline/my_recipe.py"
  source_overrides:
    input_file:
      path: "results/Category-{GROUP_COL}-level-2/data.csv"

The recipe Python code stays generic — path resolution happens via variable substitution in the YAML.

CLI Flags

Flag Type Required Description
--template string yes Template ID. Pin a version (nf-core/ampliseq/2.16.0), or use nf-core/ampliseq/latest, or just nf-core/ampliseq, to resolve the newest shipped version (v1.5.2+)
--data-root path yes Root directory substituted for {DATA_ROOT}
--var KEY=VALUE depends on template Pass template-specific variables; repeatable
--dashboard path no Override default dashboard(s); repeatable
--skip-dashboard-import flag no Skip automatic dashboard import
--project-name string no Custom project name

Resolution Workflow

When --template is set, depictio run inserts Step 0: Template resolution before the standard pipeline:

Step Name Description
0 Template resolution Load YAML, substitute variables, auto-detect metadata columns, apply conditionals, then materialise recipe seeds
1 Config validation Pydantic validation of the resolved project config
2 Authentication Login + fetch JWT token
3 Project sync Create or update project on server
4 File scan Discover data collection files
5 Data process Execute recipes, write to Delta Lake
6 Join computation Compute cross-DC joins
7 Finalize Mark project as ready
8 Dashboard import Import bundled dashboard YAML (with variable substitution)

Dashboard YAML files also undergo variable substitution (e.g. {GROUP_COL} in filter columns, chart titles).

Recipe seeds (v1.6.0+)

A reference template ships each recipe's output as a committed {DATA_ROOT}/{dc_tag}.tsv, so the bundled projects can be explored without the pipeline run behind them. Since v1.6.0 --template reads those seeds, the way first-boot seeding always did: where a seed exists for a source: transformed data collection, the recipe is replaced by a plain scan of that file. The collection keeps source: transformed, so the viewer still shows its lineage.

A data collection with no seed is left alone and its recipe runs as before. Seed coverage is uneven: nf-core/viralrecon/3.0.0 ships one for all ten of its recipe collections, nf-core/ampliseq/2.16.0 for sixteen of nineteen.

This runs after conditionals, so a collection a conditional gated out stays gated out even with a seed sitting beside it.

A seed is matched by name, not assumed

Only a source: transformed collection is redirected. A {dc_tag}.tsv next to a source: native collection of the same tag is that collection's own input, and is scanned normally.

Templates ship with the repository, not the wheel

The bundled projects live under depictio/projects/, which is not part of the published depictio-cli package. Re-ingesting a bundled project from its own directory works from a repository checkout or a container image that carries them.

Template Origin

Once a project is created from a template, the UI shows a template badge on dashboard cards and in the project data manager. This provenance is stored in the template_origin field:

  • template_id — e.g. "nf-core/ampliseq/2.16.0"
  • template_version — schema version
  • data_root — the resolved path
  • variables — all resolved variable values
  • applied_at — timestamp
  • config_snapshot — frozen copy of the resolved config

Templates define links between data collections using tags (not IDs):

links:
  - source_dc_tag: "metadata"
    source_column: "ID"
    target_dc_tag: "alpha_diversity"
    target_type: "table"
    link_config:
      resolver: "direct"
      target_field: "sample"

Tags are resolved to MongoDB ObjectIds after the project is synced to the server. Links referencing removed DCs (via conditionals) are auto-pruned.

Additional Resources