YAML Configuration Reference

Full Configuration Template Warning

Do not copy-paste this entire configuration blindly. This reference shows all available options with their defaults and descriptions. Only specify values that differ from defaults or are required for your specific use case.

This is a complete reference of all configuration options available in Depictio YAML files. For getting started and examples, see the YAML Examples Guide.

# =============================================================================
# DEPICTIO PROJECT CONFIGURATION REFERENCE
# =============================================================================

# Required: Project identification
name:
  "My Project Name" # Required: Human-readable project name
  # Must be non-empty string
  # Example: "Multi-omics Cancer Study"

# Required: Project type determines configuration structure
project_type:
  "basic" # Required: Options are "basic" or "advanced" (default: "basic")
  # basic: Direct file upload/processing
  # advanced: Workflow-integrated projects

# Optional: Project visibility settings
is_public:
  false # Optional: Project visibility (default: false)
  # true: Visible to all users (read-only)
  # false: Restricted to project members

# Optional: CLI integration path
yaml_config_path:
  null # Optional: Path to this YAML file (default: null)
  # Auto-populated by CLI, rarely set manually

# Optional: External project management integration
data_management_platform_project_url:
  null # Optional: URL to external project system (default: null)
  # Must start with http:// or https://
  # Example: "https://labid.embl.org/projects/123"

# =============================================================================
# LINKS (Cross-DC Interactive Filtering)
# =============================================================================

links:  # Optional: Define relationships for cross-DC filtering
  - source_dc_id: "sample_metadata"    # Required: DC containing the filter
    source_column: "sample_id"         # Required: Column to filter on
    target_dc_id: "multiqc_fastqc"     # Required: DC to update when filtered
    target_type: "multiqc"             # Required: "table" or "multiqc"
    link_config:
      resolver: "sample_mapping"       # Required: "direct", "sample_mapping", "pattern"
      target_field: "sample_name"      # Optional: Field to match in target DC
      # Resolvers:
      #   "direct" - Same value in both DCs
      #   "sample_mapping" - Canonical ID → MultiQC sample variants
      #   "pattern" - Template substitution (e.g., "{sample}.bam")

# =============================================================================
# WORKFLOWS (Required for Advanced Projects)
# =============================================================================

workflows:
  - # Required: Workflow identification
    name:
      "rnaseq_pipeline" # Required: Unique workflow identifier
      # Example: "rnaseq_pipeline", "variant_calling"

    # Required: Execution engine information
    engine:
      name:
        "nextflow" # Required: Engine name
        # Examples: "nextflow", "snakemake", "python", "galaxy", "cwl", "shell", "r"
      version:
        "24.10.3" # Optional: Engine version for reproducibility
        # Example: "24.10.3", "7.32.0"

    # Optional: Workflow metadata
    version:
      "3.18.0" # Optional: Workflow version
      # Example: "1.0.0", "v2.3.1"
    description: "nf-core RNA-seq pipeline for expression quantification" # Optional: Workflow description
    repository_url:
      "https://github.com/nf-core/rnaseq" # Optional: Source code repository
      # Example: "https://github.com/user/workflow"
    workflow_tag:
      "nextflow/rnaseq_pipeline" # Optional: Auto-generated workflow identifier
      # Format: "engine/name" or "catalog/name"

    # Optional: Workflow registry information
    catalog:
      name:
        "nf-core" # Optional: Catalog name
        # Options: "nf-core", "smk-wf-catalog", "workflowhub"
      url:
        "https://nf-co.re/rnaseq" # Optional: Catalog URL
        # Example: "https://nf-co.re/rnaseq"

    # Required: Data location configuration
    data_location:
      structure:
        "sequencing-runs" # Required: Directory organization pattern
        # Options:
        #   "flat" - All files in single directory level
        #   "sequencing-runs" - Hierarchical run-based structure

      locations: # Required: Root directories to search
        - "{DATA_ROOT}/rnaseq_studies/cohort_2024" # Supports environment variable expansion: {VAR_NAME}
        - "/backup/rnaseq_data" # Examples: "/absolute/path/to/data", "{DATA_ROOT}/project1"

      runs_regex:
        "batch_[A-C]" # Required for "sequencing-runs" structure
        # Optional for "flat" structure
        # Regex pattern to identify individual runs
        # Example: "run_\\d+", "sample_.*", "batch[A-Z]"

    # Required: Data collections for this workflow
    data_collections:
      - # Required: Unique identifier for this data collection
        data_collection_tag:
          "gene_expression" # Required: Unique within project
          # Used for referencing in links and dashboards
          # Example: "gene_counts", "quality_metrics"

        # Optional: Human-readable description
        description:
          "Per-sample gene expression quantification" # Optional: Description of the data collection
          # Used in UI tooltips and documentation

        # Required: Data collection configuration
        config:
          # Required: Type of data collection
          type:
            "table" # Required: Data collection type
            # Options:
            #   "table" - Tabular data (CSV, TSV, Excel, Parquet, Feather)
            #   "MultiQC" - MultiQC quality control reports
            #   "Image" - Image files with metadata
            #   "geojson" - GeoJSON boundary files for choropleth maps
            #   "phylogeny" - Newick / Nexus trees for the phylogenetic viz
            #   "indexed_file" - Indexed genomic files (VCF, BAM, bigWig,
            #     bigBed, GFF3, FASTA, bgzip + tabix), range-read by the
            #     browser; see the guide's "Indexed-file DCs" example
            #   "jbrowse2" - Genome browser tracks

          # Required: Data aggregation strategy
          metatype:
            "aggregate" # Required: Data collection metatype
            # Options:
            #   "metadata" - Single annotation/metadata file per project
            #   "aggregate" - Multiple files combined into unified dataset

          # Required: File discovery configuration
          scan:
            # Required: File scanning strategy
            mode:
              "recursive" # Required: Scanning strategy
              # Options:
              #   "single" - Single file per project/run
              #   "recursive" - Pattern-based file discovery
              #   "url" - One remote file at an s3:// or https:// URL
              #   "s3_prefix" - Every object under an s3:// prefix matching a glob
              #   "manifest" - The files a Data Manifest lists for this collection
              # mode and scan_parameters must agree: a parameter block that
              # belongs to another mode is rejected with a message naming both.
              # The three remote modes are detailed under "Remote scan modes" below.

            # Required: Mode-specific scan parameters
            scan_parameters:
              # For mode: "recursive" - specify search pattern
              regex_config: # Required for recursive mode: Pattern matching configuration
                pattern:
                  "salmon/{sample_id}/quant.sf" # Required: Regex pattern for file discovery
                  # Example: "stats/.*_stats\\.tsv"
                wildcards: # Optional: what each {name} placeholder of the pattern stands for
                  - name: "sample_id" # The placeholder, and since v1.16.0 a column of the table holding what it matched
                    wildcard_regex: "[^/]+" # Regex the placeholder is replaced by

              # For mode: "single" - specify exact file (alternative to regex_config)
              # filename: "metadata/sample_info.csv"  # Required for single mode: File path
              # Can be absolute or relative path

              # For mode: "url" - one remote file, fetched by the server
              # url: "https://data.example.org/run42/metadata.tsv"  # Required: absolute s3:// or https:// URL

              # For mode: "s3_prefix" - list a bucket prefix (the remote counterpart of recursive)
              # prefix: "s3://my-bucket/run42/"          # Required: s3:// prefix, bucket included
              # pattern: "*.samples.csv"                 # Optional: glob on the key relative to the prefix, or on the file name (default "*")
              # pattern_syntax: "glob"                   # Optional: "glob" (default) or "regex"
              # id_regex: "^([^/]+?)\\.samples\\.csv$"   # Optional: one capture group, searched in the key relative to the prefix; the capture becomes depictio_manifest_id
              # max_files: 10000                         # Optional: ceiling on matching objects (default 10000, max 100000)

              # For mode: "manifest" - consume the entries of a Data Manifest
              # manifest_url: "https://data.example.org/run42/manifest.json"  # Required: https:// manifest URL (a local path in CLI context)
              # manifest_type: "samples"                 # Required: manifest `type` this collection consumes (its tag, by convention)
              # id_field: "id"                           # Optional: manifest column holding the entity id
              # url_field: "url"                         # Optional: manifest column holding the file URL
              # type_field: "type"                       # Optional: manifest column holding the type
              # run_field: "run"                         # Optional: manifest column holding the run grouping

          # Required: Type-specific configuration
          dc_specific_properties:
            # Required: File format specification
            format:
              "tsv" # Required: File format
              # Values: "csv", "tsv", "xlsx", "xls", "parquet", "feather"
              # Case-insensitive, normalized to lowercase

            # Required: Data reading configuration using Polars
            polars_kwargs:
              # Common options
              separator:
                "\t" # Required for CSV/TSV: Column separator character
                # Default: "," for CSV, "\t" for TSV
                # Example: ",", "\t", "|", ";"
              has_header:
                true # Required: First row contains column names
                # true: First row is headers, false: First row is data
              skip_rows:
                0 # Optional: Number of rows to skip at file beginning (default: 0)
                # Useful for files with metadata headers

              # Advanced options
              column_types: # Optional: Explicit column type mapping
                Name: "String" # Forces specific data types
                Length: "Int64" # Example types: "String", "Int64", "Float64", "Boolean"
                EffectiveLength: "Float64"
                TPM: "Float64"
                NumReads: "Float64"
              column_names:
                null # Optional: Override column names when has_header: false
                # Example: ["sample", "gene", "expression"]
              null_values:
                ["", "NULL", "null", "None"] # Optional: Values to treat as null/missing
                # Default: ["", "NULL", "null", "None"]
              n_rows:
                null # Optional: Limit number of rows to read
                # Useful for testing configurations
              encoding:
                "utf8" # Optional: File encoding (default: "utf8")
                # Example: "utf8", "latin1", "ascii"

              # Excel-specific options (for .xlsx, .xls files)
              sheet_name:
                null # Optional: Excel sheet name to read
                # Example: "Results", "Sheet1"
              sheet_id:
                null # Optional: Excel sheet index (0-based)
                # Example: 0 (first sheet), 1 (second sheet)

            # Optional: Column filtering for performance
            keep_columns: # Optional: Column filtering
              - "Name" # If specified, only these columns are retained
              - "TPM" # Improves performance for large datasets
              - "NumReads" # Example: ["sample_id", "expression", "p_value"]

            # Optional: Column documentation
            columns_description: # Optional: Column documentation
              Name: "Gene/transcript identifier" # Human-readable column descriptions
              TPM: "Transcripts per million" # Used in dashboard tooltips and documentation
              NumReads: "Estimated read count" # Example format: column_name: "Description"

        # Optional: Data joining configuration (client-side pre-computed)
        # NOTE: Joins combine Table DCs during depictio ingest
        # Use project-level "links" for runtime cross-DC filtering in UI
        join: # Optional: Join this collection with others
          on_columns:
            ["sample_id"] # Column names for joining
            # Must exist in both datasets
          how:
            "inner" # Join type: "inner", "outer", "left", "right"
          with_dc:
            ["qc_summary"] # Target data collection tags

      # --- MultiQC DATA COLLECTION (v0.5.0+) ---
      - data_collection_tag:
          "multiqc_data" # Required: Unique identifier for MultiQC data

        description:
          "MultiQC quality control report data" # Optional: Description

        config:
          # Required: Type specification for MultiQC
          type:
            "MultiQC" # Required: Identifies this as a MultiQC data collection
            # NOTE: MultiQC type has special handling:
            #   - Automatically detects multiqc_data/multiqc.parquet in each run
            #   - No scan configuration needed (auto-detected)
            #   - No metatype specification required
            #   - No dc_specific_properties needed
            #   - Requires MultiQC 1.29+ to generate parquet format

        # NOTE: For MultiQC, use project-level "links" instead of joins
        # Links enable interactive filtering with sample_mapping resolver
        # Example link configuration (at project level):
        #   links:
        #     - source_dc_id: "sample_metadata"
        #       source_column: "sample_id"
        #       target_dc_id: "multiqc_data"
        #       target_type: "multiqc"
        #       link_config:
        #         resolver: "sample_mapping"

      # --- GeoJSON DATA COLLECTION ---
      - data_collection_tag:
          "europe_geojson" # Required: Unique identifier for GeoJSON data

        description:
          "GeoJSON boundaries for European regions" # Optional: Description

        config:
          # Required: Type specification for GeoJSON
          type:
            "geojson" # Required: Identifies this as a GeoJSON data collection
            # NOTE: GeoJSON type stores FeatureCollection files in S3
            #   - Used as boundary source for choropleth_map components
            #   - Referenced by geojson_dc_tag in dashboard YAML
            #   - Requires a single .geojson file

          metatype:
            "GeoJSON" # Required: Metatype identifier

          # Required: File discovery configuration
          scan:
            mode: single
            scan_parameters:
              filename: "path/to/boundaries.geojson"

          # Required: GeoJSON-specific configuration
          dc_specific_properties:
            feature_id_key:
              "properties.NAME" # Required: GeoJSON property path to match locations_column
              # Maps to Plotly's featureidkey parameter
              # Common values: "id", "properties.NAME", "properties.ISO_A3"

      # --- PHYLOGENY DATA COLLECTION ---
      - data_collection_tag:
          "bacterial_tree" # Required: Unique identifier for the tree

        description: "Bacterial phylogeny, Newick file" # Optional

        config:
          type:
            "phylogeny" # Required: Identifies this as a tree data collection
            # NOTE: file-backed, nothing is parsed or converted
            #   - Read where it was found; served as raw Newick via
            #     /advanced_viz/phylogeny/{dc_id}/newick
            #   - Referenced by tree_dc_id in an advanced_viz component
            #     of kind "phylogenetic"

          metatype: "Aggregate" # Required: Metatype identifier

          scan:
            mode: single
            scan_parameters:
              filename: "path/to/tree.nwk"

          dc_specific_properties:
            format: "newick" # Optional: "newick" (default) or "nexus"

            ladderize:
              true # Optional: ladderise the tree by default (default true)
              # Viewers can toggle this in the viz controls

            # Tip annotations live in a SEPARATE Table DC, keyed by taxon
            # name, the same pattern Microreact uses. All three keys are
            # optional; without them the tree renders unannotated.
            metadata_dc_tag:
              "bacterial_metadata" # Optional: tag of the metadata Table DC
            metadata_taxon_column:
              "taxon" # Optional: column joining to tip labels (default "taxon")
            tip_label_strategy:
              "taxon" # Optional: "taxon" (leaf name IS the taxon id, default)
              #                   or "first_token" (take the first
              #                   underscore-separated token of the leaf name)

Remote scan modes (v1.16.0+)

Three scan modes fetch data from where it already is instead of walking a local directory. Whoever ingests, the CLI or the server, reads the files there; the server goes through a gateway that validates every URL and decides which buckets it may read (see Security). The files are materialised to Delta Lake exactly like scanned local files. Remote data and manifests explains when to use which.

mode and scan_parameters are cross-checked: declaring mode: url with a filename block, for instance, is rejected with a message naming the mode the parameters belong to and the parameters the declared mode expects. The scan bounds max_depth and ignore belong to recursive: a remote mode that declares them is rejected the same way.

url: one remote file

Parameter	Required	Default	Description
url	yes		Absolute s3:// or https:// URL of the file. http:// is accepted only with DEPICTIO_REMOTE_ALLOW_HTTP=true, set on the server and, for a CLI ingestion, in the CLI's environment too. Validation here is syntactic; reachability and address checks happen at fetch time.

scan:
  mode: url
  scan_parameters:
    url: "https://data.example.org/run42/metadata.tsv"

s3_prefix: every object under a prefix

The remote counterpart of recursive: list the objects under an s3:// prefix and keep the keys matching a glob. It is S3-only by construction, since plain HTTPS exposes no listing operation.

Parameter	Required	Default	Description
prefix	yes		s3:// prefix, bucket included, e.g. s3://my-bucket/run42/
pattern	no	"*"	Glob matched against the key relative to the prefix and against the file name, so *.csv rather than the full path. Its * also crosses /, so *.csv reaches the keys of sub-prefixes too
pattern_syntax	no	"glob"	How pattern is read: glob (fnmatch) or regex. A regex is matched from the start of the file name, and of the relative key when it contains a /, as in a recursive scan. A run folder on S3 turns each recursive collection into an s3_prefix scan that keeps the template's regex, with regex here
id_regex	no	null	Regex with exactly one capture group, searched in the key relative to the prefix, then in the file name. The capture is stored as the file's entity id and read back as the depictio_manifest_id column, which gives a prefix scan the same cross-DC join key as a manifest. An object it does not match is kept without an id, and the scan warns how many there are
max_files	no	10000	Ceiling on the number of matching objects (at most 100000), a backstop against pointing a collection at a bucket root. The listing also stops after 10 × max_files keys. Either limit is reported as a warning

scan:
  mode: s3_prefix
  scan_parameters:
    prefix: "s3://my-bucket/run42/"
    pattern: "*.samples.csv"
    id_regex: "^([^/]+?)\\.samples\\.csv$"

manifest: the entries of a Data Manifest

The collection consumes every manifest entry whose type equals manifest_type. The manifest contract (id, type, url, optional run) is described on Remote data and manifests.

Parameter	Required	Default	Description
manifest_url	yes		https:// URL of the manifest (JSON or CSV). In CLI context a local path is accepted too. An s3:// URL passes validation but is not read yet
manifest_type	yes		Which manifest type this collection consumes. By convention, the collection's own tag
id_field	no	"id"	Manifest column holding the entity id
url_field	no	"url"	Manifest column holding the file URL
type_field	no	"type"	Manifest column holding the type
run_field	no	"run"	Manifest column holding the run grouping. Files of a manifest without that column get remote as their run

The *_field overrides remap a manifest with non-canonical column names onto the contract without rewriting it. In a template, every manifest collection reads the same manifest, so they must all name the same columns: From Manifest refuses a template whose collections disagree.

scan:
  mode: manifest
  scan_parameters:
    manifest_url: "{MANIFEST_URL}"
    manifest_type: "samples"

See Also

• YAML Examples - Complete configuration examples and patterns
• Project Guide - Comprehensive project management guide
• Remote data and manifests - Choosing between a URL, a prefix and a manifest
• CLI Reference - Command-line interface documentation