The proteinfamilies template follows an nf-core/proteinfamilies run from the input sequences to one family's profile model, funnel-wise:
- Overview: the run in four key figures, live findings and four figures, each linked to the tab that explains it
- Data & QC: the input sequences before and after preprocessing, and the MMseqs2 cluster size distribution per sample
- Families: the families a run built or extended, their size and conservation, and how close their members sit to the representative
- Alignment: each family's alignment, residue conservation and predicted structure, and its profile HMM as a sequence logo with the members that matched it
The persistent Sample filters (mode, sample) sit in the left panel and narrow
every tab through the samplesheet links.
Structure tiles need the structure resolver
proteinfamilies publishes no structure files. The 3D tiles of the Family
alignment and Profile HMM tabs fold the family representative from its
sequence through the structure resolver,
which is off by default: an administrator sets
DEPICTIO_STRUCTURE_RESOLVER_ENABLED=true on the server. The sequence is
sent to ESMFold on Meta's ESM Atlas, which folds at most 400 residues. With
the resolver off the tile shows an empty state.
Create and update runs
A run either creates families from its input sequences or, given the HMMs
and alignments of an existing library, first matches the sequences against
those families and extends them. The mode column (create or update) is
derived from the samplesheet and is the default grouping. Families an update
run extended carry the origin updated, the others new.
Quick start¶
The results directory is the only thing you have to pass, with one
exception: proteinfamilies reads its samplesheet from --input and does not
publish it. The template looks for it at {DATA_ROOT}/input/samplesheet.csv;
copy it there, or point to it:
depictio ingest /path/to/proteinfamilies_results \
--template nf-core/proteinfamilies/latest \
--var METADATA_FILE=/path/to/samplesheet.csv
| Variable | Default | Role |
|---|---|---|
METADATA_FILE |
{DATA_ROOT}/input/samplesheet.csv |
the samplesheet the run was started with |
GROUP_COL |
mode |
samplesheet column the dashboard groups and filters samples by; any design column added to the sheet can replace mode |
GROUP_COL_DISPLAY |
Mode |
reader-facing label of GROUP_COL in filters and cards |
depictio config nextflow --install # once per machine
nextflow run nf-core/proteinfamilies -r 2.5.0 -profile docker --outdir results
No depictio ingest, and no template named: the pipeline ingests its own
output directory when it finishes and resolves this template from its own
manifest. See Nextflow trigger. The
trigger passes no --var, so put a copy of the samplesheet at
results/input/samplesheet.csv before the run.
Reference¶
The template reads the SeqFu statistics of the input, the MMseqs2 cluster size
distribution, the family metadata and member rosters, the family alignments,
the family HMMs and the hmmalign alignments. Per-family files are laid out
<step>/<sample>/<family>.<ext>, so every family collection carries the sample
(the directory) and the family (the file name), and the families table carries a
family pick to the alignment, residue and member collections. Families an update
run extended live under update_families/. The alignment scan reads FAMSA
(.aln, the default) and MAFFT (.fas) full alignments alike, and the
HH-suite reformatted .fas.gz that a --skip_sequence_redundancy_removal run
writes.
Self-adapting layout
The dashboard adapts to whatever the run actually produced: components bound
to pruned or unparsed data collections are hidden, tabs left with no real
visualizations are dropped entirely, and the remaining components are
re-packed so there are no empty rows. A run that published neither the
family HMMs (hmm/) nor the hmmalign alignments (full_msa/raw/) loses the
logo, the hits lane and the table of members on the model. A run with no
update sample has no updated family.
Template variables¶
Variables you provide when running the template: DATA_ROOT is the results directory given to depictio ingest, the others are set with --var NAME=value:
| Variable | Required | Description |
|---|---|---|
DATA_ROOT |
✓ | proteinfamilies output directory (qc/, mmseqs/, family_reps/, full_msa/, hmm/, update_families/, pipeline_info/). |
METADATA_FILE |
— | The samplesheet the run was started with (--input: sample, fasta, optional existing_hmms_to_update and existing_msas_to_update, plus any design columns). Defaults to input/samplesheet.csv under DATA_ROOT. |
METADATA_ID_COL |
— | Samplesheet id column. proteinfamilies fixes it to sample; recorded for the metadata auto-detection. |
GROUP_COL |
— | Samplesheet column the dashboard groups and filters samples by. Defaults to mode (create or update), derived from the sheet; name any design column added to the sheet instead. |
GROUP_COL_DISPLAY |
— | Reader-facing label of GROUP_COL, used in filter and card titles. |
Data collections¶
16 data collections — 12 required 4 optional · 8 direct 8 derived.
Origin tells you whether a collection is real pipeline data or a reshape of it: direct = a pipeline output (scanned, or a recipe that reads raw files); derived = a recipe that reshapes one or more direct collections into the layout a visualization needs (no new measurement). Reads shows what produces it: a recipe .py transform, whose name links to its source on GitHub, or a raw file scanned off disk. (A direct collection can still have a recipe — one that merely parses/cleans the raw file; derived means the recipe reshapes another collection.)
| Tag | Origin | Type | Reads | Status |
|---|---|---|---|---|
samples |
direct | Table | recipe nf-core/proteinfamilies/samples.py |
required |
seqfu_stats_raw |
direct | Table | file (?:.*/)?qc/[/]+/[/]+_(?:before|after).tsv$ |
required |
sequence_stats |
derived | Table | recipe seqfu/stats.py |
required |
proteinfamilies_cluster_distribution_raw |
direct | Table | file .*_clustering_distribution_mqc.csv$ |
required |
cluster_sizes |
derived | Table | recipe proteinfamilies/cluster_size_distribution.py |
required |
proteinfamilies_family_meta_raw |
direct | Table | file (?:.*/)?family_reps/[/]+/[/]+_meta_mqc.csv$ |
required |
proteinfamilies_family_members_raw |
direct | Table | file (?:.*/)?family_reps/[/]+/[/]+.tsv$ |
required |
famsa_alignment_raw |
direct | Table | file (?:.*/)?(?:full_msa/filtered/(?:famsa_align|mafft_align|hhsuite_reformat)|update_families/full_msa/(?:famsa_align|mafft_align))/[/]+/[/]+.(?:aln|fas)(?:.gz)?$ |
required |
hmmer_hmm_raw |
direct | Table | file (?:.*/)?(?:hmm/(?:filtered|raw/[/]+)|update_families/hmm)/[/]+/[^/]+.hmm.gz$ |
optional |
hmmer_hmmalign_raw |
direct | Table | file (?:.*/)?full_msa/raw/hmmer_hmmalign/[/]+/[/]+.sto(?:.gz)?$ |
optional |
families |
derived | Table | recipe proteinfamilies/family_metadata.py |
required |
family_members |
derived | Table | recipe proteinfamilies/family_members.py |
required |
family_msa |
derived | Table | recipe famsa/alignment.py |
required |
family_residues |
derived | Table | recipe famsa/conservation.py |
required |
hmm_profile |
derived | Table | recipe hmmer/hmm_profile.py |
optional |
hmmalign_hits |
derived | Table | recipe hmmer/hmmalign_hits.py |
optional |
Cross-DC links¶
14 links — selecting a value in the source collection filters the target. The join column is shown after the source.
| Source · column | Target | Filters | |
|---|---|---|---|
samples · sample |
→ | sequence_stats |
Filter the input sequence statistics by samplesheet samples |
samples · sample |
→ | cluster_sizes |
Filter the cluster size distribution by samplesheet samples |
samples · sample |
→ | families |
Filter the families by samplesheet samples |
samples · sample |
→ | family_members |
Filter the family members by samplesheet samples |
samples · sample |
→ | family_msa |
Filter the family alignments by samplesheet samples |
samples · sample |
→ | family_residues |
Filter the residue conservation by samplesheet samples |
samples · sample |
→ | hmm_profile |
Filter the profile HMMs by samplesheet samples |
samples · sample |
→ | hmmalign_hits |
Filter the members' coverage of the models by samplesheet samples |
families · family |
→ | family_members |
Filter the members by family |
families · family |
→ | family_msa |
Filter the alignments by family |
families · family |
→ | family_residues |
Filter the residue conservation by family |
families · family |
→ | hmm_profile |
Filter the profile HMMs by family |
families · family |
→ | hmmalign_hits |
Filter the members' coverage of the models by family |
family_members · family |
→ | families |
Filter the families by the families of selected members |
Recipes¶
Each recipe reshapes raw pipeline output into a tidy table. The name links to its source; Output lists the validated OUTPUT_SCHEMA columns.
| Recipe | Transforms | Output |
|---|---|---|
famsa/alignment.py |
The protein module's MSA table from per-family aligned FASTA files. | msa_id, seq_id, rank, aligned_sequence, identity, coverage, sample |
famsa/conservation.py |
Per-residue conservation of every family, in its representative's numbering. | entity, sample, position, residue, value, category, consensus, consensus_share, occupancy, n_members, sequence |
hmmer/hmm_profile.py |
The profile HMM of every family as the table an HMM logo reads. | entity, position, hmm_name, hmm_acc, hmm_position, hmm_letter, emission, height_bits, info_bits, insert_prob, insert_length, occupancy, delete_prob, consensus, rf, cs, map_column, sample |
hmmer/hmmalign_hits.py |
How much of its family HMM each member covers, from the hmmalign alignment. | entity, sample, label, protein_length, hmm_length, hmm_from, hmm_to, ali_from, ali_to, hmm_coverage, i_evalue, dom_score |
nf-core/proteinfamilies/samples.py |
Sample hub for nf-core/proteinfamilies: the run's samplesheet, one row per sample. | sample, mode, fasta, existing_hmms_to_update, existing_msas_to_update |
proteinfamilies/cluster_size_distribution.py |
Initial MMseqs2 cluster size distribution per sample. | sample, cluster_size, n_clusters, n_sequences, sequence_share |
proteinfamilies/family_members.py |
One row per family member, with its identity to the family representative. | family, member_id, sample, origin, is_representative, in_alignment, identity, coverage, length |
proteinfamilies/family_metadata.py |
One row per protein family: representative, size, seed, model and alignment summary. | family, sample, origin, size, representative_id, representative_length, sequence, seed_size, recruited, effective_sequences, model_length, aligned_members, alignment_columns, mean_identity, mean_coverage, mean_conservation, conserved_share |
seqfu/stats.py |
One row per file SeqFu stats summarised, with its sample and stage. |
sample, stage, file, sequences, total_length, mean_length, n50, n75, n90, aun, min_length, max_length |
Dashboard tabs¶
One dashboard: the Overview, then five child tabs in three groups, read as a funnel from the input sequences to one family's profile model. Each tab below carries the same icon and colour the dashboard gives it, so the page and the app read alike.
| Group | Tabs |
|---|---|
| Data & QC | Input & Clustering |
| Families | Families, Members |
| Alignment | Family alignment, Profile HMM |
Each child tab opens with a short intro and a strip of four cards, then at most three open sections; tables and record cards follow, collapsed. The Sample sheet and the family table (Reference tables) are pinned, collapsed, to the bottom of every child tab. The family origin and the conservation class (conserved, intermediate, variable) have fixed colours, so the landscape, the histograms, the track and the structure agree.
Protein families, from the input sequences to each family's profile HMM.
A short hero links the run parameters. About this dashboard says how to move through the tabs, The run lists the samples, the clustering tool and identity, the aligner, the seed cluster threshold and the recruiting E-value, and Pipeline walks five steps (cluster, align, model, recruit, prune), each linked to its parameters and its tab. The findings are live values that follow the filters.
Filters and components
Filters · the left panel starts collapsed. Key figures has its own filter bar (mode and sample), Findings another (mode and family origin): each narrows its own section only.
| Section | What it holds |
|---|---|
| Top | Hero, About this dashboard, The run, Pipeline |
| Key figures | 4 headline cards: samples, input sequences, families, family members |
| Findings | Live result rows, then 4 figures: the initial cluster size distribution, the family landscape, the member scatter and the residue conservation histogram |
| How to read this dashboard | The tabs by group, each with its question |
Data & QC · How many sequences went in, and how did they cluster?
The SeqFu statistics of the input and the MMseqs2 cluster sizes. The input sequences, the sequences kept after the length filter, gap removal and deduplication, the mean sequence length and the share left as singleton clusters. Then the initial cluster size distribution, one curve per sample on log axes (only clusters above the size threshold seed a family), and the sequences before and after preprocessing. A large drop there points at a length range or duplicate content worth checking.
Filters and components
Filters · a Members per cluster band on cluster_sizes.
| Section | What it holds |
|---|---|
| Input at a glance | 4 cards |
| Initial clusters | 1 advanced visualization |
| Preprocessing | Input sequences before and after preprocessing |
| Input tables (collapsed) | Input sequence statistics, Initial cluster sizes |
Families · Which families were built, and how large and conserved?
The families, split into new ones and existing ones the run extended, the members per family, the members recruited beyond the seed and the mean residue conservation. Then the family landscape, one point per family by size against mean conservation and sized by representative length, with the record of the picked family beside it, and the family size per sample. The predicted structure of the picked family's representative floats over the dashboard: it opens compact, stays on screen on every tab and follows the filters of the tab you read.
Filters and components
Filters · Family origin, Members per family and Mean residue
conservation on families.
| Section | What it holds |
|---|---|
| Families at a glance | 4 cards |
| Family landscape | 2 advanced visualizations: the landscape and the Picked family record |
| Family size | Family size per sample |
| Floating panel | Predicted structure of the representative |
Families · How close are the members to their family's representative?
The members, their identity to the representative, the share of it they cover and their length. Then the identity per family, where a wide box is a family ranging from near copies to distant homologues, and the member scatter, coverage against identity, with the record of the picked member beside it. The member roster is collapsed.
Filters and components
Filters · Identity to the representative, Representative covered
and Family origin on family_members.
| Section | What it holds |
|---|---|
| Members at a glance | 4 cards |
| Identity per family | Identity to the representative per family |
| Member detail | 2 advanced visualizations: the scatter and the Picked member record |
| Member table (collapsed) | Family members |
Alignment · How conserved is each family's representative, residue by residue?
The aligned sequences, their identity to the representative, the residue conservation and the column occupancy. Then two histograms of the representative's residues, by conservation score and by column occupancy, coloured by class. A linked view then holds the predicted structure of the picked family's representative, coloured by conservation, on the left, and the family scatter, the sequence track and the family alignment stacked on the right. Residues picked in the structure, on the track or by brushing alignment columns are selected in all three. The alignment shows at most 500 rows per family.
Filters and components
Filters · an Identity to the representative range on family_msa.
| Section | What it holds |
|---|---|
| Alignment at a glance | 4 cards |
| Conservation | 2 figures |
| Fold, track and alignment (linked view) | 4 advanced visualizations: the family scatter, the predicted structure, the sequence track, the family alignment |
| Residue table (collapsed) | Residue conservation |
Alignment · What does a family's profile model look like?
One family at a time: the Family filter always holds a value, opens on the
family with most members and cannot be cleared, so the logo, the structure and
the track agree. Its scope banner
prints the family's members, its representative's length and the members'
mean identity to it. The strip reads the representative's residues and the
members' identity and length. Then a
linked view holds the predicted
structure of the representative, coloured by conservation class, on the left
and, stacked on the right, the profile logo of the family HMM (one stack of
letters per match state with the insert, occupancy and consensus lanes and,
under it, the members the hmmalign alignment placed on the model), the
sequence track and the family alignment. The states are mapped onto the
representative's residues through the family's hmmalign alignment, so a brush
on the logo selects those residues in the structure, on the track and in the
alignment.
Filters and components
Filters · a Family picker on families.
| Section | What it holds |
|---|---|
| Profile at a glance | 4 cards |
| Profile, fold and alignment (linked view) | 4 advanced visualizations: the logo, the predicted structure, the sequence track, the family alignment |
| Hits table (collapsed) | Members on the model |
Families an update run extended
They have no hmmalign alignment: their logo is drawn, but a pick on it selects no residue. The family scatter is not repeated here, since the picker holds one family.
Tables and point views select on their entity column. The samplesheet links carry a sample pick to every sample-level collection; a family picked on a scatter narrows the family-level tables and moves the structure, the track and the alignment to that family, and each record card follows the scatter beside it.
Running the pipeline¶
Depictio reads the output of nf-core/proteinfamilies, it does not run the pipeline. Run the pipeline first:
nextflow run nf-core/proteinfamilies -r 2.5.0 \
--input samplesheet.csv \
--outdir results \
-profile docker
The samplesheet is sample,fasta, plus existing_hmms_to_update and
existing_msas_to_update for a sample that extends an existing family library.
Then point Depictio at the results, naming the samplesheet the pipeline did not
publish:
depictio ingest results/ \
--template nf-core/proteinfamilies/latest \
--var METADATA_FILE=samplesheet.csv
The run can also write samplesheets for proteinfold and
proteinannotator; --skip_proteinfold_samplesheet and
--skip_proteinannotator_samplesheet turn them off.
See nf-co.re/proteinfamilies/usage for full pipeline documentation.
Required data structure¶
Point depictio ingest at the directory holding the pipeline output. Depictio
scans recursively and matches on file name; the sample and the family are read
off the path.
<DATA_ROOT>/
├── input/
│ └── samplesheet.csv # --var METADATA_FILE (not published by the pipeline)
├── pipeline_info/
│ ├── params_*.json
│ └── *software*versions*.yml
├── qc/<sample>/<sample>_{before,after}.tsv # SeqFu statistics of the input
├── mmseqs/initial_clustering/
│ └── *_clustering_distribution_mqc.csv # cluster size distribution
├── family_reps/<sample>/
│ ├── *_meta_mqc.csv # family metadata
│ └── <sample>.tsv # member roster
├── full_msa/
│ ├── filtered/{famsa_align,mafft_align,hhsuite_reformat}/<sample>/<family>.{aln,fas}
│ └── raw/hmmer_hmmalign/<sample>/<family>.sto.gz # optional: logo mapping, hits lane
├── hmm/filtered/<sample>/<family>.hmm.gz # optional: the profile logo
└── update_families/ # update samples only, same layout
Only the HMMs and the hmmalign alignments are optional.
Validation runs¶
The repository ships
download_test_data.sh,
which fetches the subset of nf-core's AWS megatest run that the template needs
(files up to 5 MB) and copies the vendored samplesheet into input/:
bash depictio/projects/nf-core/proteinfamilies/2.5.0/download_test_data.sh \
/tmp/proteinfamilies_test
depictio ingest /tmp/proteinfamilies_test \
--template nf-core/proteinfamilies/latest
The run is
s3://nf-core-awsmegatests/proteinfamilies/results-f8c0b183e59df3d87c38d0f7c4acc6918593f4f5/,
the test_full profile: two samples of 50,000 metagenomic protein sequences
each, one creating families from scratch and one updating an existing family
library before creating new families from its unmatched sequences. The
screenshots above come from that run.
Additional resources¶
- nf-co.re/proteinfamilies: official pipeline documentation
- nf-co.re/proteinfamilies/2.5.0/results: AWS test results
- Template System Reference: YAML format, variables, conditionals
- Recipes: how to read, test, and write recipes
Authorship¶
weber8thomas
Reviewing a template on your own data, or taking over a role here, is a contribution in itself: the contributing guide says what each one involves.