Depictio CLI Usage¶

Note about the CLI

The depictio-cli is a command line interface that allows you to interact with the Depictio backend. It is used to register projects information including workflow files metadata. The depictio-cli is currently in development and is not yet ready for production use.

Installation¶

See the installation guide for instructions on how to install the depictio-cli.

Quick Reference¶

Command	Description	Access Level
`version`	Show CLI version	All users
`run`	Execute complete workflow	All users
`recipe list`	List all bundled recipes	All users
`recipe info <name>`	Show recipe sources and schema	All users
`recipe run <name>`	Execute a recipe locally with validation	All users
`config show-cli-config`	Display CLI configuration	All users
`config check-s3-storage`	Validate S3 storage setup	All users
`config check-server-accessibility`	Test server connection	All users
`config validate-project-config`	Validate project configuration	All users
`config sync-project-config-to-server`	Sync project config to server	All users
`data scan`	Scan project files	All users
`data process`	Process data collections	All users
`dashboard validate`	Validate dashboard YAML file locally	All users
`dashboard import`	Import dashboard YAML to server	All users
`dashboard export`	Export dashboard to YAML file	All users
`backup create`	Create system backup	Admin only
`backup list`	List available backups	Admin only
`backup validate`	Validate backup against models	Admin only
`backup restore`	Restore from backup	Admin only
`backup check-coverage`	Check validation coverage	Admin only
`migrate`	Migrate a project to another instance	Admin only

Global Options¶

Option	Short	Type	Default	Description
`--verbose`	`-v`	`boolean`	`false`	Enable verbose logging
`--verbose-level`	`-vl`	`string`	`"INFO"`	Set verbose logging level

🚀 Commands¶

🏃 Run Command¶

🎬 🖥️ `depictio-cli run` command example

Execute the complete Depictio workflow: validate → sync → scan → process

depictio-cli run [OPTIONS]

Quick Start:

depictio-cli run --project-config-path ./config.yaml

Pipeline Steps:

✅ Check server accessibility
✅ Check S3 storage configuration
✅ Validate project configuration
✅ Sync project configuration to server
✅ Scan data files
✅ Process data collections

📋 Basic Configuration

Parameter	Type	Default	Description
`--CLI-config-path`	`string`	`~/.depictio/CLI.yaml`	CLI configuration file path
`--project-config-path`	`string`	`""`	Pipeline configuration file path (mutually exclusive with `--template`)
`--workflow-name`	`string`	`null`	Specific workflow to process
`--data-collection-tag`	`string`	`null`	Data collection tag to process

🍳 Template Options

Use these flags to run from a pre-packaged template instead of a project YAML. --template and --project-config-path are mutually exclusive.

Parameter	Type	Default	Description
`--template`	`string`	`null`	Template ID (e.g. `nf-core/ampliseq/2.16.0`)
`--data-root`	`path`	`null`	Root directory substituted for `{DATA_ROOT}` in template. Required when `--template` is set.
`--project-name`	`string`	`null`	Custom project name (auto-generated from template if omitted)
`--dashboard`	`path`	`null`	Override default dashboard(s) to import. Repeatable.
`--skip-dashboard-import`	`flag`	`false`	Skip the automatic dashboard import step (Step 8)

See Templates for full documentation.

⚙️ Flow Control Options

Parameter	Type	Default	Description
`--skip-server-check`	`boolean`	`false`	Skip server accessibility check
`--skip-s3-check`	`boolean`	`false`	Skip S3 storage validation
`--skip-sync`	`boolean`	`false`	Skip config sync to server
`--skip-scan`	`boolean`	`false`	Skip data scanning step
`--skip-process`	`boolean`	`false`	Skip data processing step

🔄 Sync & Scan Options

Parameter	Type	Default	Description
`--update-config`	`boolean`	`false`	Update project configuration on server
`--rescan-folders`	`boolean`	`false`	Reprocess all runs for data collection
`--sync-files`	`boolean`	`false`	Update files for data collection
`--overwrite`	`boolean`	`false`	Overwrite workflow if it already exists

🖥️ Output & Control

Parameter	Type	Default	Description
`--rich-tables`	`boolean`	`false`	Show detailed execution summary
`--continue-on-error`	`boolean`	`false`	Continue execution on step failure
`--dry-run`	`boolean`	`false`	Show execution plan without running

Examples:

Basic Usage

# Complete workflow execution
depictio-cli run --project-config-path ./config.yaml

Template (nf-core/ampliseq)

# Set up a complete ampliseq project from raw pipeline output
depictio-cli run \
  --template nf-core/ampliseq/2.16.0 \
  --data-root /data/my_ampliseq_run

Development

# With detailed output and error handling
depictio-cli run \
  --project-config-path ./config.yaml \
  --continue-on-error \
  --rich-tables

Dry Run

# Preview execution without running
depictio-cli run \
  --project-config-path ./config.yaml \
  --dry-run

Skip Steps

# Skip server checks for faster execution
depictio-cli run \
  --project-config-path ./config.yaml \
  --skip-server-check \
  --skip-s3-check

🍳 Recipe Commands¶

Command Group: depictio-cli recipe

All commands in this section are part of the recipe command family. Use them to discover, inspect, and locally test data transformation recipes before running them in a project. For full recipe documentation, see Recipes.

Discover and execute data transformation recipes locally.

`recipe list`¶

List all bundled recipes.

depictio-cli recipe list

Output:

Available recipes (5):
  nf-core/ampliseq/alpha_diversity.py
  nf-core/ampliseq/alpha_rarefaction.py
  nf-core/ampliseq/ancombc.py
  nf-core/ampliseq/taxonomy_composition.py
  nf-core/ampliseq/taxonomy_rel_abundance.py

`recipe info <name>`¶

Show recipe details: description, sources, and expected output schema.

depictio-cli recipe info <recipe_name>

Parameter	Type	Default	Description
`recipe_name`	`string`	required	Recipe name (e.g. `nf-core/ampliseq/alpha_diversity.py`)

depictio-cli recipe info nf-core/ampliseq/alpha_diversity.py

Output:

Recipe: nf-core/ampliseq/alpha_diversity.py
Description: Transform QIIME2 alpha diversity vector to per-sample Faith PD table.

Sources (1):
  faith_pd: qiime2/diversity/alpha_diversity/faith_pd_vector/metadata.tsv (TSV)

Expected output schema (3 columns):
  sample: Utf8
  habitat: Utf8
  faith_pd: Float64

`recipe run <name>`¶

Execute a recipe against a local data directory with all 4 validation checkpoints.

depictio-cli recipe run <recipe_name> [OPTIONS]

Parameter	Short	Default	Description
`recipe_name`	—	required	Recipe name (e.g. `nf-core/ampliseq/alpha_diversity.py`)
`--data-dir`	`-d`	required	Root directory with workflow output files
`--output`	`-o`	`null`	Save result to `.parquet` or `.csv` file
`--head`	`-n`	`20`	Number of rows to display

# Run with validation output
depictio-cli recipe run nf-core/ampliseq/alpha_diversity.py \
  --data-dir /data/ampliseq_results

# Save output and show first 5 rows
depictio-cli recipe run nf-core/ampliseq/alpha_diversity.py \
  --data-dir /data/ampliseq_results \
  --output alpha_diversity.parquet \
  --head 5

📋 Config Commands¶

Command Group: depictio-cli config

All commands in this section are part of the config command family. Use them to manage Depictio configurations and validate connections.

Manage Depictio configurations and validate connections.

`config show-cli-config`¶

Display the current CLI configuration.

depictio-cli config show-cli-config [OPTIONS]

Parameter	Type	Default	Description
`--CLI-config-path`	`string`	`~/.depictio/CLI.yaml`	CLI configuration file path

depictio-cli config show-cli-config

`config check-s3-storage`¶

Validate S3 storage configuration.

depictio-cli config check-s3-storage [OPTIONS]

Parameter	Type	Default	Description
`--CLI-config-path`	`string`	`~/.depictio/CLI.yaml`	CLI configuration file path

depictio-cli config check-s3-storage

`config check-server-accessibility`¶

Test connection to Depictio server.

depictio-cli config check-server-accessibility [OPTIONS]

Parameter	Type	Default	Description
`--CLI-config-path`	`string`	`~/.depictio/CLI.yaml`	CLI configuration file path

depictio-cli config check-server-accessibility

`config show-depictio-project-metadata-on-server`¶

Display metadata for registered projects.

depictio-cli config show-depictio-project-metadata-on-server [OPTIONS]

Parameter	Type	Default	Description
`--CLI-config-path`	`string`	`~/.depictio/CLI.yaml`	CLI configuration file path
`--project-name`	`string`	`""`	Specific project name

depictio-cli config show-depictio-project-metadata-on-server --project-name my-project

`config validate-project-config`¶

Validate project configuration file.

depictio-cli config validate-project-config [OPTIONS]

Parameter	Type	Default	Description
`--CLI-config-path`	`string`	`~/.depictio/CLI.yaml`	CLI configuration file path
`--project-config-path`	`string`	`""`	Pipeline configuration file path

depictio-cli config validate-project-config --project-config-path ./config.yaml

`config sync-project-config-to-server`¶

Sync project configuration to server.

depictio-cli config sync-project-config-to-server [OPTIONS]

Parameter	Type	Default	Description
`--CLI-config-path`	`string`	`~/.depictio/CLI.yaml`	CLI configuration file path
`--project-config-path`	`string`	`""`	Pipeline configuration file path
`--update`	`boolean`	`false`	Update existing project configuration

depictio-cli config sync-project-config-to-server --project-config-path ./config.yaml --update

📊 Data Commands¶

Command Group: depictio-cli data

All commands in this section are part of the data command family. Use them to manage data scanning and processing operations.

Manage data scanning and processing operations.

`data scan`¶

Scan project files for data collections.

depictio-cli data scan [OPTIONS]

Parameter	Type	Default	Description
`--CLI-config-path`	`string`	`~/.depictio/CLI.yaml`	CLI configuration file path
`--project-config-path`	`string`	`""`	Pipeline configuration file path
`--workflow-name`	`string`	`null`	Specific workflow to scan
`--data-collection-tag`	`string`	`null`	Data collection tag to scan
`--rescan-folders`	`boolean`	`false`	Reprocess all runs for data collection
`--sync-files`	`boolean`	`false`	Update files for data collection

depictio-cli data scan --project-config-path ./config.yaml --workflow-name my-workflow

`data process`¶

Process data collections for workflow execution.

depictio-cli data process [OPTIONS]

Parameter	Type	Default	Description
`--CLI-config-path`	`string`	`~/.depictio/CLI.yaml`	CLI configuration file path
`--project-config-path`	`string`	`""`	Pipeline configuration file path
`--overwrite`	`boolean`	`false`	Overwrite existing workflow

depictio-cli data process --project-config-path ./config.yaml --overwrite

📈 Dashboard Commands¶

Command Group: depictio-cli dashboard

All commands in this section are part of the dashboard command family. Use them to manage dashboard YAML files - validate, import to server, and export from server.

Manage dashboard YAML files for the Dashboard YAML Management feature.

Command	Description	Server Required
`validate`	Validate YAML schema locally	No
`import`	Import YAML to server	Yes (unless `--dry-run`)
`export`	Export dashboard to YAML	Yes

`dashboard validate`¶

Validate a dashboard YAML file. Runs in two passes: schema + domain constraints (always), then server schema validation (when --config is provided).

depictio-cli dashboard validate <yaml_file> [OPTIONS]

Parameter	Type	Default	Description
`yaml_file`	`path`	required	Path to YAML dashboard file
`--config` / `-c`	`string`	`null`	CLI config file — enables server schema validation (Pass 2)
`--offline`	`boolean`	`false`	Skip server schema check even when `--config` is provided
`--verbose` / `-v`	`boolean`	`false`	Show detailed validation output
`--api`	`string`	from config	API base URL

# Schema + domain only (no server needed)
depictio-cli dashboard validate my_dashboard.yaml

# Full validation including server column check
depictio-cli dashboard validate my_dashboard.yaml --config ~/.depictio/admin_config.yaml

# Force offline even when config is provided
depictio-cli dashboard validate my_dashboard.yaml --config ~/.depictio/admin_config.yaml --offline

Example Output (Success):

Validating: my_dashboard.yaml
  Pass 1: schema + domain constraints
  ✓ Schema + domain OK
  Pass 2: server schema validation
  ✓ Server schema OK

✓ Validation passed

Example Output (Failure — per-component error table):

Validating: my_dashboard.yaml
  Pass 1: schema + domain constraints
✗ Schema/domain validation failed
                         Validation Errors
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Component     ┃ Field     ┃ Message                                            ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ pie-chart     │ -         │ Invalid visu_type 'pie' for mode='ui'.             │
│               │           │ Valid values: scatter, line, bar, box, histogram   │
└───────────────┴───────────┴────────────────────────────────────────────────────┘

`dashboard import`¶

Import a dashboard YAML file to the server. The project is determined from the YAML project_tag field or the --project option.

depictio-cli dashboard import <yaml_file> [OPTIONS]

Parameter	Type	Default	Description
`yaml_file`	`path`	required	Path to YAML dashboard file
`--config` / `-c`	`string`	`null`	Path to CLI config file (required unless `--dry-run`)
`--project` / `-p`	`string`	`null`	Project ID (overrides `project_tag` in YAML)
`--overwrite`	`boolean`	`false`	Update existing dashboard with same title
`--dry-run`	`boolean`	`false`	Validate schema + domain only, don't import (no server needed)
`--offline`	`boolean`	`false`	Skip server schema check (column names not verified)
`--api`	`string`	from config	API base URL

# Validate locally without server (no config needed)
depictio-cli dashboard import dashboard.yaml --dry-run

# Import to server
depictio-cli dashboard import dashboard.yaml --config ~/.depictio/admin_config.yaml

# Update existing dashboard with same title
depictio-cli dashboard import dashboard.yaml --config ~/.depictio/admin_config.yaml --overwrite

# Override project from YAML
depictio-cli dashboard import dashboard.yaml --config ~/.depictio/admin_config.yaml --project 646b0f3c1e4a2d7f8e5b8c9a

Example Output:

Validating: dashboard.yaml
✓ Validation passed
  Title: Iris Dashboard Demo
  Components: 7
  Project: Iris_Dataset_Project (from YAML project_tag)

Loading CLI configuration...
✓ Configuration loaded
  API URL: localhost:8058

Importing dashboard (project: Iris_Dataset_Project)...
✓ Dashboard imported successfully!
  Dashboard ID: 6824cb3b89d2b72169309737
  Title: Iris Dashboard Demo
  Project ID: 650a1b2c3d4e5f6a7b8c9d0e

View at: localhost:8058/dashboard/6824cb3b89d2b72169309737

`dashboard export`¶

Export a dashboard from the server to a YAML file.

depictio-cli dashboard export <dashboard_id> [OPTIONS]

Parameter	Type	Default	Description
`dashboard_id`	`string`	required	Dashboard ID to export
`--config` / `-c`	`string`	required	Path to CLI config file
`--output` / `-o`	`path`	`dashboard.yaml`	Output file path
`--api`	`string`	from config	API base URL

# Export to default file
depictio-cli dashboard export 6824cb3b89d2b72169309737 --config ~/.depictio/admin_config.yaml

# Export to specific file
depictio-cli dashboard export 6824cb3b89d2b72169309737 --config ~/.depictio/admin_config.yaml -o iris_dashboard.yaml

Example Output:

Loading CLI configuration...
Exporting dashboard 6824cb3b89d2b72169309737...
✓ Dashboard exported to: iris_dashboard.yaml

For more information about dashboard YAML format and workflows, see Dashboard YAML Management.

💾 Backup Commands¶

Command Group: depictio-cli backup

All commands in this section are part of the backup command family. Use them to backup and restore MongoDB database and S3 data. Admin access required.

Backup and restore system data and configurations.

`backup create`¶

Create a backup of the MongoDB database.

depictio-cli backup create [OPTIONS]

Parameter	Type	Default	Description
`--CLI-config-path`	`string`	`~/.depictio/CLI.yaml`	CLI configuration file path
`--dry-run`	`boolean`	`false`	Validate without creating backup
`--include-s3-data`	`boolean`	`false`	Include S3 deltatable data in backup
`--s3-backup-prefix`	`string`	`backup`	Prefix for S3 backup location

depictio-cli backup create --include-s3-data

`backup list`¶

List available backup files on the server.

depictio-cli backup list [OPTIONS]

Parameter	Type	Default	Description
`--CLI-config-path`	`string`	`~/.depictio/CLI.yaml`	CLI configuration file path

depictio-cli backup list

`backup validate`¶

Validate a backup file against Pydantic models.

depictio-cli backup validate <backup_id> [OPTIONS]

Parameter	Type	Default	Description
`backup_id`	`string`	required	Backup ID to validate (YYYYMMDD_HHMMSS)
`--CLI-config-path`	`string`	`~/.depictio/CLI.yaml`	CLI configuration file path

depictio-cli backup validate 20240115_143052

`backup restore`¶

Restore data from a backup file. Warning: destructive operation.

depictio-cli backup restore <backup_id> [OPTIONS]

Parameter	Type	Default	Description
`backup_id`	`string`	required	Backup ID to restore (YYYYMMDD_HHMMSS)
`--CLI-config-path`	`string`	`~/.depictio/CLI.yaml`	CLI configuration file path
`--dry-run`	`boolean`	`false`	Simulate restore without making changes
`--collections`	`string`	`null`	Comma-separated list of collections
`--force`	`boolean`	`false`	Skip confirmation prompt

depictio-cli backup restore 20240115_143052 --dry-run

`backup check-coverage`¶

Check validation coverage for all MongoDB collections.

depictio-cli backup check-coverage [OPTIONS]

Parameter	Type	Default	Description
`--CLI-config-path`	`string`	`~/.depictio/CLI.yaml`	CLI configuration file path

depictio-cli backup check-coverage

🔄 Migrate Commands¶

Admin Access Required

All migrate operations require administrator privileges on both the source and target instances.

`migrate`¶

Migrate a project from one Depictio instance to another — non-destructive upsert, never wipes existing data on the target.

depictio-cli migrate --project <name> [OPTIONS]

Parameter	Type	Default	Description
`--project`	`string`	required	Project name to migrate
`--CLI-config-path`	`string`	`~/.depictio/CLI.yaml`	Source instance CLI config (credentials + API URL)
`--target-config`	`string`	`~/.depictio/CLI_remote.yaml`	Target instance CLI config
`--mode`	`string`	`all`	Migration scope: `all`, `metadata`, `dashboard`, `files`
`--dry-run`	`flag`	`False`	Preview changes without writing anything

Migration modes:

Mode	What is migrated	When to use
`all`	MongoDB documents + S3 files	First-time full migration
`metadata`	MongoDB documents only	Both instances share the same S3 storage
`dashboard`	Dashboard documents only	Project already exists on remote, updating layouts
`files`	S3 files only	MongoDB already migrated, syncing data files

# Dry-run first (recommended before any migration)
depictio-cli migrate \
  --project "My Project" \
  --CLI-config-path ~/.depictio/CLI_local.yaml \
  --target-config ~/.depictio/CLI_remote.yaml \
  --dry-run

# Full migration (MongoDB + S3)
depictio-cli migrate \
  --project "My Project" \
  --CLI-config-path ~/.depictio/CLI_local.yaml \
  --target-config ~/.depictio/CLI_remote.yaml

# Metadata-only (shared S3)
depictio-cli migrate \
  --project "My Project" \
  --CLI-config-path ~/.depictio/CLI_local.yaml \
  --target-config ~/.depictio/CLI_remote.yaml \
  --mode metadata

# Dashboard-only update
depictio-cli migrate \
  --project "My Project" \
  --CLI-config-path ~/.depictio/CLI_local.yaml \
  --target-config ~/.depictio/CLI_remote.yaml \
  --mode dashboard

🛠️ Common Use Cases¶

🚀 Quick Start¶

Complete Setup

# 1. Validate your project configuration
depictio-cli config validate-project-config --project-config-path ./config.yaml

# 2. Run the complete workflow
depictio-cli run --project-config-path ./config.yaml

Step by Step

# 1. Check system status
depictio-cli config check-server-accessibility
depictio-cli config check-s3-storage

# 2. Validate and sync configuration
depictio-cli config validate-project-config --project-config-path ./config.yaml
depictio-cli config sync-project-config-to-server --project-config-path ./config.yaml

# 3. Scan and process data
depictio-cli data scan --project-config-path ./config.yaml
depictio-cli data process --project-config-path ./config.yaml

🔧 Development Workflow¶

Testing Changes

# Preview changes without execution
depictio-cli run --project-config-path ./config.yaml --dry-run

# Test with specific workflow
depictio-cli run --project-config-path ./config.yaml --workflow-name test-workflow

Debugging

# Enable verbose logging
depictio-cli run --project-config-path ./config.yaml --verbose --continue-on-error

# Skip problematic steps
depictio-cli run --project-config-path ./config.yaml --skip-server-check --skip-s3-check

💾 Backup Operations¶

Admin Access Required

All backup operations require administrator privileges.

Create Backup

# Create database backup
depictio-cli backup create

# Include S3 deltatable data
depictio-cli backup create --include-s3-data

Restore Backup

# List available backups
depictio-cli backup list

# Preview restore
depictio-cli backup restore 20240115_143052 --dry-run

# Restore specific backup
depictio-cli backup restore 20240115_143052 --force

📊 Data Management¶

Rescan Data

# Rescan all folders
depictio-cli data scan --project-config-path ./config.yaml --rescan-folders

# Sync file updates
depictio-cli data scan --project-config-path ./config.yaml --sync-files

Process Updates

# Overwrite existing workflow
depictio-cli data process --project-config-path ./config.yaml --overwrite

# Update and reprocess
depictio-cli run --project-config-path ./config.yaml --update-config --overwrite

🔄 Project Migration¶

Admin Access Required

Migration requires administrator privileges on both source and target instances.

First-time Migration

# 1. Dry-run to preview what will be migrated
depictio-cli migrate \
  --project "My Project" \
  --CLI-config-path ~/.depictio/CLI_local.yaml \
  --target-config ~/.depictio/CLI_remote.yaml \
  --dry-run

# 2. Run full migration (MongoDB docs + S3 files)
depictio-cli migrate \
  --project "My Project" \
  --CLI-config-path ~/.depictio/CLI_local.yaml \
  --target-config ~/.depictio/CLI_remote.yaml

Shared S3 Storage

# When source and target share the same S3, only migrate MongoDB docs
depictio-cli migrate \
  --project "My Project" \
  --CLI-config-path ~/.depictio/CLI_local.yaml \
  --target-config ~/.depictio/CLI_remote.yaml \
  --mode metadata

Dashboard Update

# Push updated dashboard layouts to a remote instance
# (project and data already exist on remote)
depictio-cli migrate \
  --project "My Project" \
  --CLI-config-path ~/.depictio/CLI_local.yaml \
  --target-config ~/.depictio/CLI_remote.yaml \
  --mode dashboard

Depictio CLI Usage¶

📚 Table of Contents¶

Installation¶

Quick Reference¶

Global Options¶

🚀 Commands¶

🏃 Run Command¶

🍳 Recipe Commands¶

recipe list¶

recipe info <name>¶

recipe run <name>¶

📋 Config Commands¶

config show-cli-config¶

config check-s3-storage¶

config check-server-accessibility¶

config show-depictio-project-metadata-on-server¶

config validate-project-config¶

config sync-project-config-to-server¶

📊 Data Commands¶

data scan¶

data process¶

📈 Dashboard Commands¶

dashboard validate¶

dashboard import¶

dashboard export¶

💾 Backup Commands¶

backup create¶

backup list¶

backup validate¶

backup restore¶

backup check-coverage¶

🔄 Migrate Commands¶

migrate¶

🛠️ Common Use Cases¶

🚀 Quick Start¶

🔧 Development Workflow¶

💾 Backup Operations¶

📊 Data Management¶

🔄 Project Migration¶

📖 Configuration References¶

`recipe list`¶

`recipe info <name>`¶

`recipe run <name>`¶

`config show-cli-config`¶

`config check-s3-storage`¶

`config check-server-accessibility`¶

`config show-depictio-project-metadata-on-server`¶

`config validate-project-config`¶

`config sync-project-config-to-server`¶

`data scan`¶

`data process`¶

`dashboard validate`¶

`dashboard import`¶

`dashboard export`¶

`backup create`¶

`backup list`¶

`backup validate`¶

`backup restore`¶

`backup check-coverage`¶

`migrate`¶