immuneML.IO.dataset_import package

Submodules

immuneML.IO.dataset_import.AIRRImport module

class immuneML.IO.dataset_import.AIRRImport.AIRRImport[source]

Bases: immuneML.IO.dataset_import.DataImport.DataImport

Imports data in AIRR format into a Repertoire-, Sequence- or ReceptorDataset. RepertoireDatasets should be used when making predictions per repertoire, such as predicting a disease state. SequenceDatasets or ReceptorDatasets should be used when predicting values for unpaired (single-chain) and paired immune receptors respectively, like antigen specificity.

The AIRR .tsv format is explained here: https://docs.airr-community.org/en/stable/datarep/format.html And the AIRR rearrangement schema can be found here: https://docs.airr-community.org/en/stable/datarep/rearrangements.html

When importing a ReceptorDataset, the AIRR field cell_id is used to determine the chain pairs.

Parameters

  • path (str) – This is the path to a directory with AIRR files to import. By default path is set to the current working directory.

  • is_repertoire (bool) – If True, this imports a RepertoireDataset. If False, it imports a SequenceDataset or

  • default (ReceptorDataset. By) –

  • True. (is_repertoire is set to) –

  • metadata_file (str) – Required for RepertoireDatasets. This parameter specifies the path to the metadata file.

  • filename (This is a csv file with columns) –

  • instructions. (subject_id and arbitrary other columns which can be used as labels in) –

  • RepertoireDataset. (Only the AIRR files included under the column 'filename' are imported into the) –

  • metadata (For setting SequenceDataset) –

  • ignored (metadata_file is) –

  • instead. (see metadata_column_mapping) –

  • paired (str) – Required for Sequence- or ReceptorDatasets. This parameter determines whether to import a

  • SequenceDataset (paired = False) or a ReceptorDataset (paired = True) –

  • ReceptorDataset (In a) –

  • together (two sequences with chain types specified by receptor_chains are paired) –

  • 'cell_id'. (based on the identifier given in the AIRR column named) –

  • receptor_chains (str) – Required for ReceptorDatasets. Determines which pair of chains to import for each Receptor.

:param Valid values for receptor_chains are the names of the ChainPair enum.: :param If receptor_chains is not provided: :type If receptor_chains is not provided: only one chain pair type allowed per repertoire :param the chain pair is automatically detected: :type the chain pair is automatically detected: only one chain pair type allowed per repertoire :param import_productive: Whether productive sequences (with value ‘T’ in column productive) should be included :type import_productive: bool :param in the imported sequences. By default: :param import_productive is True.: :param import_with_stop_codon: Whether sequences with stop codons (with value ‘T’ in column stop_codon) should :type import_with_stop_codon: bool :param be included in the imported sequences. This only applies if column stop_codon is present. By default: :param : :param import_with_stop_codon is False.: :param import_out_of_frame: Whether out of frame sequences (with value ‘F’ in column vj_in_frame) should :type import_out_of_frame: bool :param be included in the imported sequences. This only applies if column vj_in_frame is present. By default: :param : :param import_out_of_frame is False.: :param import_illegal_characters: Whether to import sequences that contain illegal characters, i.e., characters :type import_illegal_characters: bool :param that do not appear in the sequence alphabet: :type that do not appear in the sequence alphabet: amino acids including stop codon ‘*’, or nucleotides :param applied to the sequence type of interest (when running immuneML in amino acid mode: :param only entries with illegal: :param characters in the amino acid sequence are removed). By default import_illegal_characters is False.: :param import_empty_nt_sequences: imports sequences which have an empty nucleotide sequence field; can be True or False. :type import_empty_nt_sequences: bool :param By default: :param import_empty_nt_sequences is set to True.: :param import_empty_aa_sequences: imports sequences which have an empty amino acid sequence field; can be True or False; for analysis on :type import_empty_aa_sequences: bool :param amino acid sequences: :type amino acid sequences: import only non-empty amino acid sequences :param this parameter should be False: :type this parameter should be False: import only non-empty amino acid sequences :param region_type: Which part of the sequence to import. By default, this value is set to IMGT_CDR3. This means the :type region_type: str :param first and last amino acids are removed from the CDR3 sequence: :param as AIRR uses the IMGT junction. Specifying: :param any other value will result in importing the sequences as they are.: :param Valid values for region_type are the names of the RegionType enum.: :param column_mapping: A mapping from AIRR column names to immuneML’s internal data representation. :type column_mapping: dict :param For AIRR: :param this is by default set to: :param .. indent with spaces: :param .. code-block:: yaml: junction: sequences

junction_aa: sequence_aas v_call: v_alleles j_call: j_alleles locus: chains duplicate_count: counts sequence_id: sequence_identifiers

Parameters

  • if (A custom column mapping can be specified here if necessary (for example; adding additional data fields) –

  • file (The formatting is the same as column_mapping. If some columns specified in column_mapping are not found in the) –

  • names) (or using alternative column) –

  • Repertoire.FIELDS (Valid immuneML fields that can be specified here are defined by) –

  • column_mapping_synonyms (dict) – This is a column mapping that can be used if a column could have alternative names.

  • file –

:param : :param the columns specified in column_mapping_synonyms are instead attempted to be loaded.: :param For AIRR format: :param there is no default column_mapping_synonyms.: :param metadata_column_mapping: Specifies metadata for Sequence- and ReceptorDatasets. This should specify a mapping similar :type metadata_column_mapping: dict :param to column_mapping where keys are AIRR column names and values are the names that are internally used in immuneML: :param as metadata fields. These metadata fields can be used as prediction labels for Sequence- and ReceptorDatasets.: :param For AIRR format: :param there is no default metadata_column_mapping.: :param For setting RepertoireDataset metadata: :param metadata_column_mapping is ignored: :param see metadata_file instead.: :param separator: Column separator, for AIRR this is by default “t”. :type separator: str

YAML specification:

my_airr_dataset:
    format: AIRR
    params:
        path: path/to/files/
        is_repertoire: True # whether to import a RepertoireDataset
        metadata_file: path/to/metadata.csv # metadata file for RepertoireDataset
        metadata_column_mapping: # metadata column mapping AIRR: immuneML for Sequence- or ReceptorDatasetDataset
            airr_column_name1: metadata_label1
            airr_column_name2: metadata_label2
        import_productive: True # whether to include productive sequences in the dataset
        import_with_stop_codon: False # whether to include sequences with stop codon in the dataset
        import_out_of_frame: False # whether to include out of frame sequences in the dataset
        import_illegal_characters: False # remove sequences with illegal characters for the sequence_type being used
        import_empty_nt_sequences: True # keep sequences even if the `sequences` column is empty (provided that other fields are as specified here)
        import_empty_aa_sequences: False # remove all sequences with empty `sequence_aas` column
        # Optional fields with AIRR-specific defaults, only change when different behavior is required:
        separator: "\t" # column separator
        region_type: IMGT_CDR3 # what part of the sequence to import
        column_mapping: # column mapping AIRR: immuneML
            junction: sequences
            junction_aa: sequence_aas
            v_call: v_alleles
            j_call: j_alleles
            locus: chains
            duplicate_count: counts
            sequence_id: sequence_identifiers
static alternative_load_func(filename, params)[source]
static get_documentation()[source]
static import_dataset(params: dict, dataset_name: str) → immuneML.data_model.dataset.Dataset.Dataset[source]
static import_receptors(df, params)[source]
static preprocess_dataframe(df: pandas.DataFrame, params: immuneML.IO.dataset_import.DatasetImportParams.DatasetImportParams)[source]
Function for preprocessing data from a dataframe containing AIRR data, such that:

  • productive sequences, sequences with stop codons or out of frame sequences are filtered according to specification

  • if RegionType is CDR3, the leading C and trailing W are removed from the sequence to match the CDR3 definition

  • if no chain column was specified, the chain is extracted from the v gene name

  • the allele information is removed from the V and J genes

immuneML.IO.dataset_import.DataImport module

class immuneML.IO.dataset_import.DataImport.DataImport[source]

Bases: object

abstract static import_dataset(params, dataset_name: str) → immuneML.data_model.dataset.Dataset.Dataset[source]

immuneML.IO.dataset_import.DatasetImportParams module

class immuneML.IO.dataset_import.DatasetImportParams.DatasetImportParams(path: pathlib.Path = None, is_repertoire: bool = None, metadata_file: pathlib.Path = None, paired: bool = None, receptor_chains: immuneML.data_model.receptor.ChainPair.ChainPair = None, result_path: pathlib.Path = None, columns_to_load: list = None, separator: str = None, column_mapping: dict = None, column_mapping_synonyms: dict = None, region_type: immuneML.data_model.receptor.RegionType.RegionType = None, import_productive: bool = None, import_unproductive: bool = None, import_with_stop_codon: bool = None, import_out_of_frame: bool = None, import_illegal_characters: bool = None, metadata_column_mapping: dict = None, number_of_processes: int = 1, sequence_file_size: int = 50000, organism: str = None, import_empty_nt_sequences: bool = None, import_empty_aa_sequences: bool = None)[source]

Bases: object

classmethod build_object(path: Optional[pathlib.Path] = None, metadata_file: Optional[pathlib.Path] = None, result_path: Optional[pathlib.Path] = None, region_type: Optional[str] = None, receptor_chains: Optional[str] = None, **kwargs)[source]
column_mapping: dict = None
column_mapping_synonyms: dict = None
columns_to_load: list = None
import_empty_aa_sequences: bool = None
import_empty_nt_sequences: bool = None
import_illegal_characters: bool = None
import_out_of_frame: bool = None
import_productive: bool = None
import_unproductive: bool = None
import_with_stop_codon: bool = None
is_repertoire: bool = None
metadata_column_mapping: dict = None
metadata_file: pathlib.Path = None
number_of_processes: int = 1
organism: str = None
paired: bool = None
path: pathlib.Path = None
receptor_chains: immuneML.data_model.receptor.ChainPair.ChainPair = None
region_type: immuneML.data_model.receptor.RegionType.RegionType = None
result_path: pathlib.Path = None
separator: str = None
sequence_file_size: int = 50000

immuneML.IO.dataset_import.GenericImport module

class immuneML.IO.dataset_import.GenericImport.GenericImport[source]

Bases: immuneML.IO.dataset_import.DataImport.DataImport

Imports data from any tabular file into a Repertoire-, Sequence- or ReceptorDataset. RepertoireDatasets should be used when making predictions per repertoire, such as predicting a disease state. SequenceDatasets or ReceptorDatasets should be used when predicting values for unpaired (single-chain) and paired immune receptors respectively, like antigen specificity.

This importer works similarly to other importers, but has no predefined default values for which fields are imported, and can therefore be tailored to import data from various different tabular files with headers.

For ReceptorDatasets: this importer assumes the two receptor sequences appear on different lines in the file, and can be paired together by a common sequence identifier. If you instead want to import a ReceptorDataset from a tabular file that contains both receptor chains on one line, see SingleLineReceptor import

Parameters

  • path (str) – Required parameter. This is the path to a directory with files to import.

  • is_repertoire (bool) – If True, this imports a RepertoireDataset. If False, it imports a SequenceDataset or

  • default (By) –

  • True. (import_empty_nt_sequences is set to) –

  • metadata_file (str) – Required for RepertoireDatasets. This parameter specifies the path to the metadata file.

  • filename (This is a csv file with columns) –

  • instructions. (subject_id and arbitrary other columns which can be used as labels in) –

  • metadata (For setting Sequence- or ReceptorDataset) –

  • ignored (metadata_file is) –

  • instead. (see metadata_column_mapping) –

  • paired (str) – Required for Sequence- or ReceptorDatasets. This parameter determines whether to import a

  • SequenceDataset (paired = False) or a ReceptorDataset (paired = True) –

  • ReceptorDataset (In a) –

  • based (two sequences with chain types specified by receptor_chains are paired together) –

  • using (on a common identifier. This identifier should be mapped to the immuneML field 'sequence_identifiers') –

  • column_mapping. (the) –

  • receptor_chains (str) – Required for ReceptorDatasets. Determines which pair of chains to import for each Receptor.

  • enum. (Valid values for receptor_chains are the names of the ChainPair) –

  • import_illegal_characters (bool) – Whether to import sequences that contain illegal characters, i.e., characters

  • alphabet (that do not appear in the sequence) –

  • mode (applied to the sequence type of interest (when running immuneML in amino acid) –

  • illegal (only entries with) –

  • removed) (characters in the amino acid sequence are) –

  • import_empty_nt_sequences (bool) – imports sequences which have an empty nucleotide sequence field; can be True or False.

  • default –

  • True. –

  • import_empty_aa_sequences (bool) – imports sequences which have an empty amino acid sequence field; can be True or False; for analysis on

  • sequences (amino acid) –

  • False (this parameter should be) –

  • region_type (str) – Which part of the sequence to import. When IMGT_CDR3 is specified, immuneML assumes the IMGT

  • junction (including leading C and trailing Y/F amino acids) –

  • value (amino acids will be removed from the sequences to retrieve the IMGT CDR3 sequence. Specifying any other) –

  • are. (will result in importing the sequences as they) –

:param Valid values for region_type are the names of the RegionType enum.: :param column_mapping: Required for all datasets. A mapping where the keys are the column names in the input file, :type column_mapping: dict :param and the values correspond to the names used in immuneML’s internal data representation.: :param Valid immuneML fields that can be specified here are defined by Repertoire.FIELDS: :param At least sequences: :type At least sequences: nucleotide) or sequence_aas (amino acids :param are optional. A column mapping can look for example like this: :param .. indent with spaces: :param .. code-block:: yaml: file_column_amino_acids: sequence_aas

file_column_v_genes: v_genes file_column_j_genes: j_genes file_column_frequencies: counts

Parameters

  • column_mapping_synonyms (dict) – This is a column mapping that can be used if a column could have alternative names.

  • file (The formatting is the same as column_mapping. If some columns specified in column_mapping are not found in the) –

:param : :param the columns specified in column_mapping_synonyms are instead attempted to be loaded.: :param For Generic import: :param there is no default column_mapping_synonyms.: :param metadata_column_mapping: Optional; specifies metadata for Sequence- and ReceptorDatasets. This is a column :type metadata_column_mapping: dict :param mapping that is formatted similarly to column_mapping: :param but here the values are the names that immuneML internally: :param uses as metadata fields. These fields can subsequently be used as labels in instructions (for example labels: :param that are used for prediction by ML methods). This column mapping could for example look like this: :param .. indent with spaces: :param .. code-block:: yaml: file_column_antigen_specificity: antigen_specificity :param The label antigen_specificity can now be used throughout immuneML.: :param For setting RepertoireDataset metadata: :param metadata_column_mapping is ignored: :param see metadata_file instead.: :param columns_to_load: Optional; specifies which columns to load from the input file. This may be useful if :type columns_to_load: list :param the input files contain many unused columns. If no value is specified: :param all columns are loaded.: :param separator: Required parameter. Column separator, for example “t” or “,”. :type separator: str

YAML specification:

my_generic_dataset:
    format: Generic
    params:
        path: path/to/files/
        is_repertoire: True # whether to import a RepertoireDataset
        metadata_file: path/to/metadata.csv # metadata file for RepertoireDataset
        paired: False # whether to import SequenceDataset (False) or ReceptorDataset (True) when is_repertoire = False
        receptor_chains: TRA_TRB # what chain pair to import for a ReceptorDataset
        separator: "\t" # column separator
        import_illegal_characters: False # remove sequences with illegal characters for the sequence_type being used
        import_empty_nt_sequences: True # keep sequences even though the nucleotide sequence might be empty
        import_empty_aa_sequences: False # filter out sequences if they don't have sequence_aa set
        region_type: IMGT_CDR3 # what part of the sequence to import
        column_mapping: # column mapping file: immuneML
            file_column_amino_acids: sequence_aas
            file_column_v_genes: v_genes
            file_column_j_genes: j_genes
            file_column_frequencies: counts
        metadata_column_mapping: # metadata column mapping file: immuneML
            file_column_antigen_specificity: antigen_specificity
        columns_to_load:  # which subset of columns to load from the file
            - file_column_amino_acids
            - file_column_v_genes
            - file_column_j_genes
            - file_column_frequencies
            - file_column_antigen_specificity
static get_documentation()[source]
static import_dataset(params: dict, dataset_name: str) → <module 'immuneML.data_model.dataset.Dataset' from '/Users/milenpa/PycharmProjects/BMIImmuneML/immuneML/data_model/dataset/Dataset.py'>[source]
static import_receptors(df, params)[source]
static preprocess_dataframe(df: pandas.DataFrame, params: immuneML.IO.dataset_import.DatasetImportParams.DatasetImportParams)[source]

immuneML.IO.dataset_import.IGoRImport module

class immuneML.IO.dataset_import.IGoRImport.IGoRImport[source]

Bases: immuneML.IO.dataset_import.DataImport.DataImport

Imports data generated by IGoR simulations into a Repertoire-, or SequenceDataset. RepertoireDatasets should be used when making predictions per repertoire, such as predicting a disease state. SequenceDatasets should be used when predicting values for unpaired (single-chain) immune receptors, like antigen specificity.

Note that you should run IGoR with the –CDR3 option specified, this tool imports the generated CDR3 files. Sequences with missing anchors are not imported, meaning only sequences with value ‘1’ in the anchors_found column are imported. Nucleotide sequences are automatically translated to amino acid sequences.

Reference: Quentin Marcou, Thierry Mora, Aleksandra M. Walczak ‘High-throughput immune repertoire analysis with IGoR’. Nature Communications, (2018) doi.org/10.1038/s41467-018-02832-w.

Parameters

  • path (str) – This is the path to a directory with IGoR files to import. By default path is set to the current working directory.

  • is_repertoire (bool) – If True, this imports a RepertoireDataset. If False, it imports a SequenceDataset.

  • default (By) –

  • True. (import_empty_nt_sequences is set to) –

  • metadata_file (str) – Required for RepertoireDatasets. This parameter specifies the path to the metadata file.

  • filename (This is a csv file with columns) –

  • instructions. (subject_id and arbitrary other columns which can be used as labels in) –

  • RepertoireDataset. (Only the IGoR files included under the column 'filename' are imported into the) –

  • metadata (For setting SequenceDataset) –

  • ignored (metadata_file is) –

  • instead. (see metadata_column_mapping) –

  • import_with_stop_codon (bool) – Whether sequences with stop codons should be included in the imported sequences.

  • default –

  • False. (characters in the amino acid sequence are removed) By default import_illegal_characters is) –

  • import_out_of_frame (bool) – Whether out of frame sequences (with value ‘0’ in column is_inframe) should

  • default –

  • False. –

  • import_illegal_characters (bool) – Whether to import sequences that contain illegal characters, i.e., characters

  • alphabet (that do not appear in the sequence) –

  • mode (applied to the sequence type of interest (when running immuneML in amino acid) –

  • illegal (only entries with) –

  • False. –

  • import_empty_nt_sequences (bool) – imports sequences which have an empty nucleotide sequence field; can be True or False.

  • default –

  • True. –

  • region_type (str) – Which part of the sequence to import. By default, this value is set to IMGT_CDR3. This means the

  • sequence (first and last amino acids are removed from the CDR3) –

  • Specifying (as IGoR uses the IMGT junction.) –

  • are. (any other value will result in importing the sequences as they) –

:param Valid values for region_type are the names of the RegionType enum.: :param column_mapping: A mapping from IGoR column names to immuneML’s internal data representation. :type column_mapping: dict :param For IGoR: :param this is by default set to: :param .. indent with spaces: :param .. code-block:: yaml: nt_CDR3: sequences

seq_index: sequence_identifiers

Parameters

  • if (A custom column mapping can be specified here if necessary (for example; adding additional data fields) –

  • file (The formatting is the same as column_mapping. If some columns specified in column_mapping are not found in the) –

  • names) (or using alternative column) –

  • Repertoire.FIELDS (Valid immuneML fields that can be specified here are defined by) –

  • column_mapping_synonyms (dict) – This is a column mapping that can be used if a column could have alternative names.

  • file –

:param : :param the columns specified in column_mapping_synonyms are instead attempted to be loaded.: :param For IGoR format: :param there is no default column_mapping_synonyms.: :param metadata_column_mapping: Specifies metadata for SequenceDatasets. This should specify a mapping similar :type metadata_column_mapping: dict :param to column_mapping where keys are IGoR column names and values are the names that are internally used in immuneML: :param as metadata fields. These metadata fields can be used as prediction labels for SequenceDatasets.: :param For IGoR format: :param there is no default metadata_column_mapping.: :param For setting RepertoireDataset metadata: :param metadata_column_mapping is ignored: :param see metadata_file instead.: :param separator: Column separator, for IGoR this is by default “,”. :type separator: str

YAML specification:

my_igor_dataset:
    format: IGoR
    params:
        path: path/to/files/
        is_repertoire: True # whether to import a RepertoireDataset (True) or a SequenceDataset (False)
        metadata_file: path/to/metadata.csv # metadata file for RepertoireDataset
        metadata_column_mapping: # metadata column mapping IGoR: immuneML for SequenceDataset
            igor_column_name1: metadata_label1
            igor_column_name2: metadata_label2
        import_with_stop_codon: False # whether to include sequences with stop codon in the dataset
        import_out_of_frame: False # whether to include out of frame sequences in the dataset
        import_illegal_characters: False # remove sequences with illegal characters for the sequence_type being used
        import_empty_nt_sequences: True # keep sequences even though the nucleotide sequence might be empty
        # Optional fields with IGoR-specific defaults, only change when different behavior is required:
        separator: "," # column separator
        region_type: IMGT_CDR3 # what part of the sequence to import
        column_mapping: # column mapping IGoR: immuneML
            nt_CDR3: sequences
            seq_index: sequence_identifiers
CODON_TABLE = {'AAA': 'K', 'AAC': 'N', 'AAG': 'K', 'AAT': 'N', 'ACA': 'T', 'ACC': 'T', 'ACG': 'T', 'ACT': 'T', 'AGA': 'R', 'AGC': 'S', 'AGG': 'R', 'AGT': 'S', 'ATA': 'I', 'ATC': 'I', 'ATG': 'M', 'ATT': 'I', 'CAA': 'Q', 'CAC': 'H', 'CAG': 'Q', 'CAT': 'H', 'CCA': 'P', 'CCC': 'P', 'CCG': 'P', 'CCT': 'P', 'CGA': 'R', 'CGC': 'R', 'CGG': 'R', 'CGT': 'R', 'CTA': 'L', 'CTC': 'L', 'CTG': 'L', 'CTT': 'L', 'GAA': 'E', 'GAC': 'D', 'GAG': 'E', 'GAT': 'D', 'GCA': 'A', 'GCC': 'A', 'GCG': 'A', 'GCT': 'A', 'GGA': 'G', 'GGC': 'G', 'GGG': 'G', 'GGT': 'G', 'GTA': 'V', 'GTC': 'V', 'GTG': 'V', 'GTT': 'V', 'TAA': '*', 'TAC': 'Y', 'TAG': '*', 'TAT': 'Y', 'TCA': 'S', 'TCC': 'S', 'TCG': 'S', 'TCT': 'S', 'TGA': '*', 'TGC': 'C', 'TGG': 'W', 'TGT': 'C', 'TTA': 'L', 'TTC': 'F', 'TTG': 'L', 'TTT': 'F'}
static get_documentation()[source]
static import_dataset(params: dict, dataset_name: str) → <module 'immuneML.data_model.dataset.Dataset' from '/Users/milenpa/PycharmProjects/BMIImmuneML/immuneML/data_model/dataset/Dataset.py'>[source]
static preprocess_dataframe(df: pandas.DataFrame, params: immuneML.IO.dataset_import.DatasetImportParams.DatasetImportParams)[source]
static translate_sequence(nt_seq)[source]

Code inspired by: https://github.com/prestevez/dna2proteins/blob/master/dna2proteins.py

immuneML.IO.dataset_import.IReceptorImport module

class immuneML.IO.dataset_import.IReceptorImport.IReceptorImport[source]

Bases: immuneML.IO.dataset_import.DataImport.DataImport

Imports AIRR datasets retrieved through the iReceptor Gateway into a Repertoire-, Sequence- or ReceptorDataset. The differences between this importer and the AIRR importer are:

  • This importer takes in a list of .zip files, which must contain one or more AIRR tsv files, and for each AIRR file, a corresponding metadata json file must be present.

  • This importer does not require a metadata csv file for RepertoireDataset import, it is generated automatically from the metadata json files.

RepertoireDatasets should be used when making predictions per repertoire, such as predicting a disease state. SequenceDatasets or ReceptorDatasets should be used when predicting values for unpaired (single-chain) and paired immune receptors respectively, like antigen specificity.

AIRR rearrangement schema can be found here: https://docs.airr-community.org/en/stable/datarep/rearrangements.html

When importing a ReceptorDataset, the AIRR field cell_id is used to determine the chain pairs.

Parameters

  • path (str) – This is the path to a directory with .zip files retrieved from the iReceptor Gateway. These .zip

  • files (files should include AIRR) –

  • extension. (the corresponding metadata file is called my_dataset-metadata.json) The zip files must use the .zip) –

  • is_repertoire (bool) – If True, this imports a RepertoireDataset. If False, it imports a SequenceDataset or

  • default (ReceptorDataset. By) –

  • True. (is_repertoire is set to) –

  • paired (str) – Required for Sequence- or ReceptorDatasets. This parameter determines whether to import a

  • SequenceDataset (paired = False) or a ReceptorDataset (paired = True) –

  • ReceptorDataset (In a) –

  • together (two sequences with chain types specified by receptor_chains are paired) –

  • 'cell_id'. (based on the identifier given in the AIRR column named) –

  • receptor_chains (str) – Required for ReceptorDatasets. Determines which pair of chains to import for each Receptor.

:param Valid values for receptor_chains are the names of the ChainPair enum.: :param If receptor_chains is not provided: :type If receptor_chains is not provided: only one chain pair type allowed per repertoire :param the chain pair is automatically detected: :type the chain pair is automatically detected: only one chain pair type allowed per repertoire :param import_productive: Whether productive sequences (with value ‘T’ in column productive) should be included :type import_productive: bool :param in the imported sequences. By default: :param import_productive is True.: :param import_with_stop_codon: Whether sequences with stop codons (with value ‘T’ in column stop_codon) should :type import_with_stop_codon: bool :param be included in the imported sequences. This only applies if column stop_codon is present. By default: :param : :param import_with_stop_codon is False.: :param import_out_of_frame: Whether out of frame sequences (with value ‘F’ in column vj_in_frame) should :type import_out_of_frame: bool :param be included in the imported sequences. This only applies if column vj_in_frame is present. By default: :param : :param import_out_of_frame is False.: :param import_illegal_characters: Whether to import sequences that contain illegal characters, i.e., characters :type import_illegal_characters: bool :param that do not appear in the sequence alphabet: :type that do not appear in the sequence alphabet: amino acids including stop codon ‘*’, or nucleotides :param applied to the sequence type of interest (when running immuneML in amino acid mode: :param only entries with illegal: :param characters in the amino acid sequence are removed). By default import_illegal_characters is False.: :param import_empty_nt_sequences: imports sequences which have an empty nucleotide sequence field; can be True or False. :type import_empty_nt_sequences: bool :param By default: :param import_empty_nt_sequences is set to True.: :param import_empty_aa_sequences: imports sequences which have an empty amino acid sequence field; can be True or False; for analysis on :type import_empty_aa_sequences: bool :param amino acid sequences: :type amino acid sequences: import only non-empty amino acid sequences :param this parameter should be False: :type this parameter should be False: import only non-empty amino acid sequences :param region_type: Which part of the sequence to import. By default, this value is set to IMGT_CDR3. This means the :type region_type: str :param first and last amino acids are removed from the CDR3 sequence: :param as AIRR uses the IMGT junction. Specifying: :param any other value will result in importing the sequences as they are.: :param Valid values for region_type are the names of the RegionType enum.: :param column_mapping: A mapping from AIRR column names to immuneML’s internal data representation. :type column_mapping: dict :param For AIRR: :param this is by default set to: :param .. indent with spaces: :param .. code-block:: yaml: junction: sequences

junction_aa: sequence_aas v_call: v_alleles j_call: j_alleles locus: chains duplicate_count: counts sequence_id: sequence_identifiers

Parameters

  • if (A custom column mapping can be specified here if necessary (for example; adding additional data fields) –

  • file (The formatting is the same as column_mapping. If some columns specified in column_mapping are not found in the) –

  • names) (or using alternative column) –

  • Repertoire.FIELDS (Valid immuneML fields that can be specified here are defined by) –

  • column_mapping_synonyms (dict) – This is a column mapping that can be used if a column could have alternative names.

  • file –

:param : :param the columns specified in column_mapping_synonyms are instead attempted to be loaded.: :param For AIRR format: :param there is no default column_mapping_synonyms.: :param metadata_column_mapping: Specifies metadata for Sequence- and ReceptorDatasets. This should specify a mapping similar :type metadata_column_mapping: dict :param to column_mapping where keys are AIRR column names and values are the names that are internally used in immuneML: :param as metadata fields. These metadata fields can be used as prediction labels for Sequence- and ReceptorDatasets.: :param For AIRR format: :param there is no default metadata_column_mapping.: :param When importing a RepertoireDataset: :param the metadata is automatically extracted from the metadata json files.: :param separator: Column separator, for AIRR this is by default “t”. :type separator: str

YAML specification:

my_airr_dataset:
    format: IReceptor
    params:
        path: path/to/zipfiles/
        is_repertoire: True # whether to import a RepertoireDataset
        metadata_column_mapping: # metadata column mapping AIRR: immuneML for Sequence- or ReceptorDatasetDataset
            airr_column_name1: metadata_label1
            airr_column_name2: metadata_label2
        import_productive: True # whether to include productive sequences in the dataset
        import_with_stop_codon: False # whether to include sequences with stop codon in the dataset
        import_out_of_frame: False # whether to include out of frame sequences in the dataset
        import_illegal_characters: False # remove sequences with illegal characters for the sequence_type being used
        import_empty_nt_sequences: True # keep sequences even if the `sequences` column is empty (provided that other fields are as specified here)
        import_empty_aa_sequences: False # remove all sequences with empty `sequence_aas` column
        # Optional fields with AIRR-specific defaults, only change when different behavior is required:
        separator: "\t" # column separator
        region_type: IMGT_CDR3 # what part of the sequence to import
        column_mapping: # column mapping AIRR: immuneML
            junction: sequences
            junction_aa: sequence_aas
            v_call: v_alleles
            j_call: j_alleles
            locus: chains
            duplicate_count: counts
            sequence_id: sequence_identifiers
REPERTOIRES_FOLDER = 'repertoires/'
static get_documentation()[source]
static import_dataset(params: dict, dataset_name: str) → immuneML.data_model.dataset.Dataset.Dataset[source]
static import_repertoire_dataset(params: dict, dataset_name: str) → immuneML.data_model.dataset.RepertoireDataset.RepertoireDataset[source]
static import_sequence_dataset(params: dict, dataset_name: str) → immuneML.data_model.dataset.RepertoireDataset.RepertoireDataset[source]

immuneML.IO.dataset_import.ImmuneMLImport module

class immuneML.IO.dataset_import.ImmuneMLImport.ImmuneMLImport[source]

Bases: immuneML.IO.dataset_import.DataImport.DataImport

Imports the dataset from the files previously exported by immuneML. It closely resembles AIRR format but relies on binary representations and is optimized for faster read-in at runtime.

ImmuneMLImport can import any kind of dataset (RepertoireDataset, SequenceDataset, ReceptorDataset).

This format includes:

  1. a dataset file in yaml format with iml_dataset extension with parameters:

    • name,

    • identifier,

    • metadata_file (for repertoire datasets),

    • metadata_fields (for repertoire datasets),

    • repertoire_ids (for repertoire datasets)

    • element_ids (for receptor and sequence datasets),

    • labels,

  2. a csv metadata file (only for repertoire datasets, should be in the same folder as the iml_dataset file),

3. data files for different types of data. For repertoire datasets, data files include one binary numpy file per repertoire with sequences and associated information and one metadata yaml file per repertoire with details such as repertoire identifier, disease status, subject id and other similar available information. For sequence and receptor datasets, sequences or receptors respectively, are stored in batches in binary numpy files.

Parameters

  • path (str) – The path to the previously created dataset file. This file should have an ‘.iml_dataset’ extension.

  • specified (If the path has not been) –

  • file (immuneML attempts to load the dataset from a specified metadata) –

  • RepertoireDatasets) ((only for) –

  • metadata_file (str) – An optional metadata file for a RepertoireDataset. If specified, the RepertoireDataset

  • objects (metadata will be updated to the newly specified metadata without otherwise changing the Repertoire) –

YAML specification:

my_dataset:
    format: ImmuneML
    params:
        path: path/to/dataset.iml_dataset
        metadata_file: path/to/metadata.csv
static import_dataset(params: dict, dataset_name: str) → immuneML.data_model.dataset.Dataset.Dataset[source]

immuneML.IO.dataset_import.ImmunoSEQRearrangementImport module

class immuneML.IO.dataset_import.ImmunoSEQRearrangementImport.ImmunoSEQRearrangementImport[source]

Bases: immuneML.IO.dataset_import.DataImport.DataImport

Imports data from Adaptive Biotechnologies immunoSEQ Analyzer rearrangement-level .tsv files into a Repertoire-, or SequenceDataset. RepertoireDatasets should be used when making predictions per repertoire, such as predicting a disease state. SequenceDatasets should be used when predicting values for unpaired (single-chain) immune receptors, like antigen specificity.

The format of the files imported by this importer is described here: https://www.adaptivebiotech.com/wp-content/uploads/2019/07/MRK-00342_immunoSEQ_TechNote_DataExport_WEB_REV.pdf Alternatively, to import sample-level .tsv files, see ImmunoSEQSample import

The only difference between these two importers is which columns they load from the .tsv files.

Parameters

  • path (str) – This is the path to a directory with files to import. By default path is set to the current working directory.

  • is_repertoire (bool) – If True, this imports a RepertoireDataset. If False, it imports a SequenceDataset.

  • default (By) –

  • True. (import_empty_nt_sequences is set to) –

  • metadata_file (str) – Required for RepertoireDatasets. This parameter specifies the path to the metadata file.

  • filename (This is a csv file with columns) –

  • instructions. (subject_id and arbitrary other columns which can be used as labels in) –

  • RepertoireDataset. (Only the files included under the column 'filename' are imported into the) –

  • metadata (For setting SequenceDataset) –

  • ignored (metadata_file is) –

  • instead. (see metadata_column_mapping) –

  • import_productive (bool) – Whether productive sequences (with value ‘In’ in column frame_type) should be included

  • default –

  • True. –

  • import_with_stop_codon (bool) – Whether sequences with stop codons (with value ‘Stop’ in column frame_type) should

  • default –

  • False. (characters in the amino acid sequence are removed) By default import_illegal_characters is) –

  • import_out_of_frame (bool) – Whether out of frame sequences (with value ‘Out’ in column frame_type) should

  • default –

  • False. –

  • import_illegal_characters (bool) – Whether to import sequences that contain illegal characters, i.e., characters

  • alphabet (that do not appear in the sequence) –

  • mode (applied to the sequence type of interest (when running immuneML in amino acid) –

  • illegal (only entries with) –

  • False. –

  • import_empty_nt_sequences (bool) – imports sequences which have an empty nucleotide sequence field; can be True or False.

  • default –

  • True. –

  • import_empty_aa_sequences (bool) – imports sequences which have an empty amino acid sequence field; can be True or False; for analysis on

  • sequences (amino acid) –

  • False (this parameter should be) –

  • region_type (str) – Which part of the sequence to import. By default, this value is set to IMGT_CDR3. This means the

  • sequence (first and last amino acids are removed from the CDR3) –

  • junction. (as immunoSEQ files use the IMGT) –

  • are. (Specifying any other value will result in importing the sequences as they) –

:param Valid values for region_type are the names of the RegionType enum.: :param column_mapping: A mapping from immunoSEQ column names to immuneML’s internal data representation. :type column_mapping: dict :param For immunoSEQ rearrangement-level files: :param this is by default set to: :param .. indent with spaces: :param .. code-block:: yaml: rearrangement: sequences

amino_acid: sequence_aas v_gene: v_genes j_gene: j_genes frame_type: frame_types v_family: v_subgroups j_family: j_subgroups v_allele: v_alleles j_allele: j_alleles templates: counts locus: chains

Parameters

  • if (A custom column mapping can be specified here if necessary (for example; adding additional data fields) –

  • file (The formatting is the same as column_mapping. If some columns specified in column_mapping are not found in the) –

  • names) (or using alternative column) –

  • Repertoire.FIELDS (Valid immuneML fields that can be specified here are defined by) –

  • column_mapping_synonyms (dict) – This is a column mapping that can be used if a column could have alternative names.

  • file –

:param : :param the columns specified in column_mapping_synonyms are instead attempted to be loaded.: :param For immunoSEQ rearrangement-level files: :param this is by default set to: :param .. indent with spaces: :param .. code-block:: yaml: v_resolved: v_alleles

j_resolved: j_alleles

Parameters

  • columns_to_load (list) – Specifies which subset of columns must be loaded from the file. By default, this is:

  • [rearrangement –

  • v_family –

  • v_gene –

  • v_allele –

  • j_family –

  • j_gene –

  • j_allele –

  • amino_acid –

  • templates –

  • frame_type –

  • locus] –

  • metadata_column_mapping (dict) – Specifies metadata for SequenceDatasets. This should specify a mapping similar

  • immuneML (to column_mapping where keys are immunoSEQ column names and values are the names that are internally used in) –

  • SequenceDatasets. (as metadata fields. These metadata fields can be used as prediction labels for) –

  • files (For immunoSEQ rearrangement .tsv) –

  • metadata_column_mapping. (there is no default) –

  • metadata (For setting RepertoireDataset) –

  • ignored (metadata_column_mapping is) –

  • instead. (see metadata_file) –

  • separator (str) – Column separator, for ImmunoSEQ files this is by default “t”.

  • import_empty_nt_sequences (bool) – imports sequences which have an empty nucleotide sequence field; can be True or False

  • import_empty_aa_sequences (bool) – imports sequences which have an empty amino acid sequence field; can be True or False; for analysis on

  • sequences (amino acid) –

  • False (this parameter will typically be) –

YAML specification:

my_immunoseq_dataset:
    format: ImmunoSEQRearrangement
    params:
        path: path/to/files/
        is_repertoire: True # whether to import a RepertoireDataset (True) or a SequenceDataset (False)
        metadata_file: path/to/metadata.csv # metadata file for RepertoireDataset
        metadata_column_mapping: # metadata column mapping ImmunoSEQ: immuneML for SequenceDataset
            immunoseq_column_name1: metadata_label1
            immunoseq_column_name2: metadata_label2
        import_productive: True # whether to include productive sequences in the dataset
        import_with_stop_codon: False # whether to include sequences with stop codon in the dataset
        import_out_of_frame: False # whether to include out of frame sequences in the dataset
        import_illegal_characters: False # remove sequences with illegal characters for the sequence_type being used
        import_empty_nt_sequences: True # keep sequences even though the nucleotide sequence might be empty
        import_empty_aa_sequences: False # filter out sequences if they don't have sequence_aa set
        # Optional fields with ImmunoSEQ rearrangement-specific defaults, only change when different behavior is required:
        separator: "\t" # column separator
        columns_to_load: # subset of columns to load
        - rearrangement
        - v_family
        - v_gene
        - v_allele
        - j_family
        - j_gene
        - j_allele
        - amino_acid
        - templates
        - frame_type
        - locus
        region_type: IMGT_CDR3 # what part of the sequence to import
        column_mapping: # column mapping immunoSEQ: immuneML
            rearrangement: sequences
            amino_acid: sequence_aas
            v_gene: v_genes
            j_gene: j_genes
            frame_type: frame_types
            v_family: v_subgroups
            j_family: j_subgroups
            v_allele: v_alleles
            j_allele: j_alleles
            templates: counts
            locus: chains
static get_documentation()[source]
static import_dataset(params: dict, dataset_name: str) → <module 'immuneML.data_model.dataset.Dataset' from '/Users/milenpa/PycharmProjects/BMIImmuneML/immuneML/data_model/dataset/Dataset.py'>[source]
static preprocess_dataframe(df: pandas.DataFrame, params: immuneML.IO.dataset_import.DatasetImportParams.DatasetImportParams)[source]

immuneML.IO.dataset_import.ImmunoSEQSampleImport module

class immuneML.IO.dataset_import.ImmunoSEQSampleImport.ImmunoSEQSampleImport[source]

Bases: immuneML.IO.dataset_import.DataImport.DataImport

Imports data from Adaptive Biotechnologies immunoSEQ Analyzer sample-level .tsv files into a Repertoire-, or SequenceDataset. RepertoireDatasets should be used when making predictions per repertoire, such as predicting a disease state. SequenceDatasets should be used when predicting values for unpaired (single-chain) immune receptors, like antigen specificity.

The format of the files imported by this importer is described here in section 3.4.13 https://clients.adaptivebiotech.com/assets/downloads/immunoSEQ_AnalyzerManual.pdf Alternatively, to import rearrangement-level .tsv files, see ImmunoSEQRearrangement import. The only difference between these two importers is which columns they load from the .tsv files.

Parameters

  • path (str) – This is the path to a directory with files to import. By default path is set to the current working directory.

  • is_repertoire (bool) – If True, this imports a RepertoireDataset. If False, it imports a SequenceDataset.

  • default (By) –

  • True. (import_empty_nt_sequences is set to) –

  • metadata_file (str) – Required for RepertoireDatasets. This parameter specifies the path to the metadata file.

  • filename (This is a csv file with columns) –

  • instructions. (subject_id and arbitrary other columns which can be used as labels in) –

  • RepertoireDataset. (Only the files included under the column 'filename' are imported into the) –

  • metadata (For setting SequenceDataset) –

  • ignored (metadata_file is) –

  • instead. (see metadata_column_mapping) –

  • import_productive (bool) – Whether productive sequences (with value ‘In’ in column frame_type) should be included

  • default –

  • True. –

  • import_with_stop_codon (bool) – Whether sequences with stop codons (with value ‘Stop’ in column frame_type) should

  • default –

  • False. (characters in the amino acid sequence are removed) By default import_illegal_characters is) –

  • import_out_of_frame (bool) – Whether out of frame sequences (with value ‘Out’ in column frame_type) should

  • default –

  • False. –

  • import_illegal_characters (bool) – Whether to import sequences that contain illegal characters, i.e., characters

  • alphabet (that do not appear in the sequence) –

  • mode (applied to the sequence type of interest (when running immuneML in amino acid) –

  • illegal (only entries with) –

  • False. –

  • import_empty_nt_sequences (bool) – imports sequences which have an empty nucleotide sequence field; can be True or False.

  • default –

  • True. –

  • import_empty_aa_sequences (bool) – imports sequences which have an empty amino acid sequence field; can be True or False; for analysis on

  • sequences (amino acid) –

  • False (this parameter should be) –

  • region_type (str) – Which part of the sequence to import. By default, this value is set to IMGT_CDR3. This means the

  • sequence (first and last amino acids are removed from the CDR3) –

  • junction. (as immunoSEQ files use the IMGT) –

  • are. (Specifying any other value will result in importing the sequences as they) –

:param Valid values for region_type are the names of the RegionType enum.: :param column_mapping: A mapping from immunoSEQ column names to immuneML’s internal data representation. :type column_mapping: dict :param For immunoSEQ sample-level files: :param this is by default set to: :param .. indent with spaces: :param .. code-block:: yaml: nucleotide: sequences

aminoAcid: sequence_aas vGeneName: v_genes jGeneName: j_genes sequenceStatus: frame_types vFamilyName: v_subgroups jFamilyName: j_subgroups vGeneAllele: v_alleles jGeneAllele: j_alleles count (templates/reads): counts

Parameters

  • if (A custom column mapping can be specified here if necessary (for example; adding additional data fields) –

  • file (The formatting is the same as column_mapping. If some columns specified in column_mapping are not found in the) –

  • names) (or using alternative column) –

  • Repertoire.FIELDS (Valid immuneML fields that can be specified here are defined by) –

  • column_mapping_synonyms (dict) – This is a column mapping that can be used if a column could have alternative names.

  • file –

:param : :param the columns specified in column_mapping_synonyms are instead attempted to be loaded.: :param For immunoSEQ sample .tsv files: :param there is no default column_mapping_synonyms.: :param columns_to_load: Specifies which subset of columns must be loaded from the file. By default, this is: :type columns_to_load: list :param [nucleotide: :type [nucleotide: templates/reads :param aminoAcid: :type aminoAcid: templates/reads :param count: :type count: templates/reads :param metadata_column_mapping: Specifies metadata for SequenceDatasets. This should specify a mapping similar :type metadata_column_mapping: dict :param to column_mapping where keys are immunoSEQ column names and values are the names that are internally used in immuneML: :param as metadata fields. These metadata fields can be used as prediction labels for SequenceDatasets.: :param For immunoSEQ sample .tsv files: :param there is no default metadata_column_mapping.: :param For setting RepertoireDataset metadata: :param metadata_column_mapping is ignored: :param see metadata_file instead.: :param separator: Column separator, for ImmunoSEQ files this is by default “t”. :type separator: str

YAML specification:

my_immunoseq_dataset:
    format: ImmunoSEQSample
    params:
        path: path/to/files/
        is_repertoire: True # whether to import a RepertoireDataset (True) or a SequenceDataset (False)
        metadata_file: path/to/metadata.csv # metadata file for RepertoireDataset
        metadata_column_mapping: # metadata column mapping ImmunoSEQ: immuneML for SequenceDataset
            immunoseq_column_name1: metadata_label1
            immunoseq_column_name2: metadata_label2
        import_productive: True # whether to include productive sequences in the dataset
        import_with_stop_codon: False # whether to include sequences with stop codon in the dataset
        import_out_of_frame: False # whether to include out of frame sequences in the dataset
        import_illegal_characters: False # remove sequences with illegal characters for the sequence_type being used
        import_empty_nt_sequences: True # keep sequences even though the nucleotide sequence might be empty
        import_empty_aa_sequences: False # filter out sequences if they don't have sequence_aa set
        # Optional fields with ImmunoSEQ sample-specific defaults, only change when different behavior is required:
        separator: "\t" # column separator
        columns_to_load: # subset of columns to load
        - nucleotide
        - aminoAcid
        - count (templates/reads)
        - vFamilyName
        - vGeneName
        - vGeneAllele
        - jFamilyName
        - jGeneName
        - jGeneAllele
        - sequenceStatus
        region_type: IMGT_CDR3 # what part of the sequence to import
        column_mapping: # column mapping immunoSEQ: immuneML
            nucleotide: sequences
            aminoAcid: sequence_aas
            vGeneName: v_genes
            jGeneName: j_genes
            sequenceStatus: frame_types
            vFamilyName: v_subgroups
            jFamilyName: j_subgroups
            vGeneAllele: v_alleles
            jGeneAllele: j_alleles
            count (templates/reads): counts
static get_documentation()[source]
static import_dataset(params: dict, dataset_name: str) → immuneML.data_model.dataset.Dataset.Dataset[source]
static preprocess_dataframe(df: pandas.DataFrame, params: immuneML.IO.dataset_import.DatasetImportParams.DatasetImportParams)[source]

immuneML.IO.dataset_import.MiXCRImport module

class immuneML.IO.dataset_import.MiXCRImport.MiXCRImport[source]

Bases: immuneML.IO.dataset_import.DataImport.DataImport

Imports data in MiXCR format into a Repertoire-, or SequenceDataset. RepertoireDatasets should be used when making predictions per repertoire, such as predicting a disease state. SequenceDatasets should be used when predicting values for unpaired (single-chain) immune receptors, like antigen specificity.

Parameters

  • path (str) – This is the path to a directory with MiXCR files to import. By default path is set to the current working directory.

  • is_repertoire (bool) – If True, this imports a RepertoireDataset. If False, it imports a SequenceDataset.

  • default (By) –

  • True. (import_empty_nt_sequences is set to) –

  • metadata_file (str) – Required for RepertoireDatasets. This parameter specifies the path to the metadata file.

  • filename (This is a csv file with columns) –

  • instructions. (subject_id and arbitrary other columns which can be used as labels in) –

  • RepertoireDataset. (Only the MiXCR files included under the column 'filename' are imported into the) –

  • metadata (For setting SequenceDataset) –

  • ignored (metadata_file is) –

  • instead. (see metadata_column_mapping) –

  • import_illegal_characters (bool) – Whether to import sequences that contain illegal characters, i.e., characters

  • alphabet (that do not appear in the sequence) –

  • mode (applied to the sequence type of interest (when running immuneML in amino acid) –

  • illegal (only entries with) –

  • sequence (Alternatively to importing the CDR3) –

  • '_' (such as) –

  • False. (are removed) By default import_illegal_characters is) –

  • import_empty_nt_sequences (bool) – imports sequences which have an empty nucleotide sequence field; can be True or False.

  • default –

  • True. –

  • import_empty_aa_sequences (bool) – imports sequences which have an empty amino acid sequence field; can be True or False; for analysis on

  • sequences (amino acid) –

  • False (this parameter should be) –

  • region_type (str) – Which part of the sequence to import. By default, this value is set to IMGT_CDR3. This means the

  • sequence –

  • CDR3. (as MiXCR uses IMGT junction as) –

  • sequence –

  • well. (other region types can be specified here as) –

  • MiXCRImport.SEQUENCE_NAME_MAP. (Valid values for region_type are defined in) –

  • column_mapping (dict) – A mapping from MiXCR column names to immuneML’s internal data representation.

  • MiXCR (For) –

  • to (this is by default set) –

  • spaces (. indent with) –

  • code-block: (.) – yaml: cloneCount: counts allVHitsWithScore: v_alleles allJHitsWithScore: j_alleles

  • parameter. (The columns that specify the sequences to import are handled by the region_type) –

  • if (A custom column mapping can be specified here if necessary (for example; adding additional data fields) –

  • file (The formatting is the same as column_mapping. If some columns specified in column_mapping are not found in the) –

  • names) (or using alternative column) –

  • Repertoire.FIELDS (Valid immuneML fields that can be specified here are defined by) –

  • column_mapping_synonyms (dict) – This is a column mapping that can be used if a column could have alternative names.

  • file –

:param : :param the columns specified in column_mapping_synonyms are instead attempted to be loaded.: :param For MiXCR format: :param there is no default column_mapping_synonyms.: :param columns_to_load: Specifies which subset of columns must be loaded from the MiXCR file. By default, this is: :type columns_to_load: list :param [cloneCount: :param allVHitsWithScore: :param allJHitsWithScore: :param aaSeqCDR3: :param nSeqCDR3]: :param metadata_column_mapping: Specifies metadata for SequenceDatasets. This should specify a mapping similar :type metadata_column_mapping: dict :param to column_mapping where keys are MiXCR column names and values are the names that are internally used in immuneML: :param as metadata fields. These metadata fields can be used as prediction labels for SequenceDatasets.: :param For MiXCR format: :param there is no default metadata_column_mapping.: :param For setting RepertoireDataset metadata: :param metadata_column_mapping is ignored: :param see metadata_file instead.: :param separator: Column separator, for MiXCR this is by default “t”. :type separator: str

YAML specification:

my_mixcr_dataset:
    format: MiXCR
    params:
        path: path/to/files/
        is_repertoire: True # whether to import a RepertoireDataset (True) or a SequenceDataset (False)
        metadata_file: path/to/metadata.csv # metadata file for RepertoireDataset
        metadata_column_mapping: # metadata column mapping MiXCR: immuneML for SequenceDataset
            mixcrColumnName1: metadata_label1
            mixcrColumnName2: metadata_label2
        region_type: IMGT_CDR3 # what part of the sequence to import
        import_illegal_characters: False # remove sequences with illegal characters for the sequence_type being used
        import_empty_nt_sequences: True # keep sequences even though the nucleotide sequence might be empty
        import_empty_aa_sequences: False # filter out sequences if they don't have sequence_aa set
        # Optional fields with MiXCR-specific defaults, only change when different behavior is required:
        separator: "\t" # column separator
        columns_to_load: # subset of columns to load, sequence columns are handled by region_type parameter
        - cloneCount
        - allVHitsWithScore
        - allJHitsWithScore
        - aaSeqCDR3
        - nSeqCDR3
        column_mapping: # column mapping MiXCR: immuneML
            cloneCount: counts
            allVHitsWithScore: v_genes
            allJHitsWithScore: j_genes
SEQUENCE_NAME_MAP = {<RegionType.IMGT_CDR3: 'IMGT_CDR3'>: {'AA': 'aaSeqCDR3', 'NT': 'nSeqCDR3'}, <RegionType.IMGT_CDR1: 'IMGT_CDR1'>: {'AA': 'aaSeqCDR1', 'NT': 'nSeqCDR1'}, <RegionType.IMGT_CDR2: 'IMGT_CDR2'>: {'AA': 'aaSeqCDR2', 'NT': 'nSeqCDR2'}, <RegionType.IMGT_FR1: 'IMGT_FR1'>: {'AA': 'aaSeqFR1', 'NT': 'nSeqFR1'}, <RegionType.IMGT_FR2: 'IMGT_FR2'>: {'AA': 'aaSeqFR2', 'NT': 'nSeqFR2'}, <RegionType.IMGT_FR3: 'IMGT_FR3'>: {'AA': 'aaSeqFR3', 'NT': 'nSeqFR3'}, <RegionType.IMGT_FR4: 'IMGT_FR4'>: {'AA': 'aaSeqFR4', 'NT': 'nSeqFR4'}}
static get_documentation()[source]
static import_dataset(params: dict, dataset_name: str) → <module 'immuneML.data_model.dataset.Dataset' from '/Users/milenpa/PycharmProjects/BMIImmuneML/immuneML/data_model/dataset/Dataset.py'>[source]
static preprocess_dataframe(df: pandas.DataFrame, params: immuneML.IO.dataset_import.DatasetImportParams.DatasetImportParams)[source]
Function for loading the data from one MiXCR file, such that:

  • for the given region (CDR3/full sequence), both nucleotide and amino acid sequence are loaded

  • if the region is CDR3, it adapts the sequence to the definition of the CDR3 (IMGT junction vs IMGT CDR3)

  • the chain for each sequence is extracted from the v gene name

  • the genes are loaded from the top score for gene without allele info

Parameters

  • metadata – the corresponding row from the metadata file with metadata such as subject_id, age, HLA or other info given there

  • params – DatasetImportParams object defining what to import and how to do it

Returns

data frame corresponding to Repertoire.FIELDS and custom lists which can be used to create a Repertoire object

immuneML.IO.dataset_import.OLGAImport module

class immuneML.IO.dataset_import.OLGAImport.OLGAImport[source]

Bases: immuneML.IO.dataset_import.DataImport.DataImport

Imports data generated by OLGA simulations into a Repertoire-, or SequenceDataset. Assumes that the columns in each file correspond to: nucleotide sequences, amino acid sequences, v genes, j genes

Reference: Sethna, Zachary et al. ‘High-throughput immune repertoire analysis with IGoR’. Bioinformatics, (2019) doi.org/10.1093/bioinformatics/btz035.

Parameters

  • path (str) – This is the path to a directory with OLGA files to import. By default path is set to the current working directory.

  • is_repertoire (bool) – If True, this imports a RepertoireDataset. If False, it imports a SequenceDataset.

  • default (By) –

  • True. (import_empty_nt_sequences is set to) –

  • metadata_file (str) – Required for RepertoireDatasets. This parameter specifies the path to the metadata file.

  • filename (This is a csv file with columns) –

  • instructions. (subject_id and arbitrary other columns which can be used as labels in) –

  • RepertoireDataset. (Only the OLGA files included under the column 'filename' are imported into the) –

  • supported. (SequenceDataset metadata is currently not) –

  • import_illegal_characters (bool) – Whether to import sequences that contain illegal characters, i.e., characters

  • alphabet (that do not appear in the sequence) –

  • mode (applied to the sequence type of interest (when running immuneML in amino acid) –

  • illegal (only entries with) –

  • False. (characters in the amino acid sequence are removed) By default import_illegal_characters is) –

  • import_empty_nt_sequences (bool) – imports sequences which have an empty nucleotide sequence field; can be True or False.

  • default –

  • True. –

  • import_empty_aa_sequences (bool) – imports sequences which have an empty amino acid sequence field; can be True or False; for analysis on

  • sequences (amino acid) –

  • False (this parameter should be) –

  • region_type (str) – Which part of the sequence to import. By default, this value is set to IMGT_CDR3. This means the

  • sequence (first and last amino acids are removed from the CDR3) –

  • Specifying (as OLGA uses the IMGT junction.) –

  • are. (any other value will result in importing the sequences as they) –

:param Valid values for region_type are the names of the RegionType enum.: :param separator: Column separator, for OLGA this is by default “t”. :type separator: str

YAML specification:

my_olga_dataset:
    format: OLGA
    params:
        path: path/to/files/
        is_repertoire: True # whether to import a RepertoireDataset (True) or a SequenceDataset (False)
        metadata_file: path/to/metadata.csv # metadata file for RepertoireDataset
        import_illegal_characters: False # remove sequences with illegal characters for the sequence_type being used
        import_empty_nt_sequences: True # keep sequences even though the nucleotide sequence might be empty
        import_empty_aa_sequences: False # filter out sequences if they don't have sequence_aa set
        # Optional fields with OLGA-specific defaults, only change when different behavior is required:
        separator: "\t" # column separator
        region_type: IMGT_CDR3 # what part of the sequence to import
static alternative_load_func(filepath, params)[source]
static get_documentation()[source]
static import_dataset(params: dict, dataset_name: str) → <module 'immuneML.data_model.dataset.Dataset' from '/Users/milenpa/PycharmProjects/BMIImmuneML/immuneML/data_model/dataset/Dataset.py'>[source]
static preprocess_dataframe(df: pandas.DataFrame, params: immuneML.IO.dataset_import.DatasetImportParams.DatasetImportParams)[source]

immuneML.IO.dataset_import.RandomReceptorDatasetImport module

class immuneML.IO.dataset_import.RandomReceptorDatasetImport.RandomReceptorDatasetImport[source]

Bases: immuneML.IO.dataset_import.DataImport.DataImport

Returns a ReceptorDataset consisting of randomly generated sequences, which can be used for benchmarking purposes. The sequences consist of uniformly chosen amino acids or nucleotides.

Parameters

  • receptor_count (int) – The number of receptors the ReceptorDataset should contain.

  • chain_1_length_probabilities (dict) – A mapping where the keys correspond to different sequence lengths for chain 1,

  • length. (and the values are the probabilities for choosing each sequence) –

  • example (For) –

  • 10 (to create a random ReceptorDataset where 40% of the sequences for chain 1 would be of length) –

  • and –

  • 12 (60% of the sequences would have length) –

  • specified (the following mapping should be) –

  • spaces (. indent with) –

  • code-block: (.) – yaml: 10: 0.4 12: 0.6

  • chain_2_length_probabilities (dict) – Same as chain_1_length_probabilities, but for chain 2.

  • labels (labels can be specified here. The keys of this mapping are the) – A mapping that specifies randomly chosen labels to be assigned to the receptors. One or multiple

  • labels –

  • mapping (and the values consist of another) –

  • probabilities. (between label classes and their) –

  • example –

  • class (to create a random ReceptorDataset with the label cmv_epitope where 70% of the receptors has) –

  • not_binding (binding and the remaining 30% has class) –

  • specified –

  • spaces –

  • code-block: –

    yaml: cmv_epitope:

    binding: 0.7 not_binding: 0.3

YAML specification:

my_random_dataset:
    format: RandomReceptorDataset
    params:
        receptor_count: 100 # number of random receptors to generate
        chain_1_length_probabilities:
            14: 0.8 # 80% of all generated sequences for all receptors (for chain 1) will have length 14
            15: 0.2 # 20% of all generated sequences across all receptors (for chain 1) will have length 15
        chain_2_length_probabilities:
            14: 0.8 # 80% of all generated sequences for all receptors (for chain 2) will have length 14
            15: 0.2 # 20% of all generated sequences across all receptors (for chain 2) will have length 15
        labels:
            epitope1: # label name
                True: 0.5 # 50% of the receptors will have class True
                False: 0.5 # 50% of the receptors will have class False
            epitope2: # next label with classes that will be assigned to receptors independently of the previous label or other parameters
                1: 0.3 # 30% of the generated receptors will have class 1
                0: 0.7 # 70% of the generated receptors will have class 0
static import_dataset(params, name: str) → immuneML.data_model.dataset.ReceptorDataset.ReceptorDataset[source]

Returns randomly generated receptor dataset according to the parameters;

YAML specification:

result_path: path/where/to/store/results/ receptor_count: 100 # number of random receptors to generate chain_1_length_probabilities:

14: 0.8 # 80% of all generated sequences for all receptors (for chain 1) will have length 14 15: 0.2 # 20% of all generated sequences across all receptors (for chain 1) will have length 15

chain_2_length_probabilities:

14: 0.8 # 80% of all generated sequences for all receptors (for chain 2) will have length 14 15: 0.2 # 20% of all generated sequences across all receptors (for chain 2) will have length 15

labels:
epitope1: # label name

True: 0.5 # 50% of the receptors will have class True False: 0.5 # 50% of the receptors will have class False

epitope2: # next label with classes that will be assigned to receptors independently of the previous label or other parameters

1: 0.3 # 30% of the generated receptors will have class 1 0: 0.7 # 70% of the generated receptors will have class 0

immuneML.IO.dataset_import.RandomRepertoireDatasetImport module

class immuneML.IO.dataset_import.RandomRepertoireDatasetImport.RandomRepertoireDatasetImport[source]

Bases: immuneML.IO.dataset_import.DataImport.DataImport

Returns a RepertoireDataset consisting of randomly generated sequences, which can be used for benchmarking purposes. The sequences consist of uniformly chosen amino acids or nucleotides.

Parameters

  • repertoire_count (int) – The number of repertoires the RepertoireDataset should contain.

  • sequence_count_probabilities (dict) – A mapping where the keys are the number of sequences per repertoire, and

  • sequences. (the values are the probabilities that any of the repertoires would have that number of) –

  • example (For) –

  • sequences (to create a random RepertoireDataset where 40% of the repertoires would have 1000) –

:param : :param and the other 60% would have 1100 sequences: :param this mapping would need to be specified: :param .. indent with spaces: :param .. code-block:: yaml: 1000: 0.4

1100: 0.6

Parameters

  • sequence_length_probabilities (dict) – A mapping where the keys correspond to different sequence lengths, and

  • length. (the values are the probabilities for choosing each sequence) –

  • example (For) –

  • 10 (to create a random RepertoireDataset where 40% of the sequences would be of length) –

  • and –

  • 12 (60% of the sequences would have length) –

  • specified (the following mapping should be) –

  • spaces (. indent with) –

  • code-block: (.) – yaml: 10: 0.4 12: 0.6

  • labels (labels can be specified here. The keys of this mapping are the) – A mapping that specifies randomly chosen labels to be assigned to the Repertoires. One or multiple

  • labels –

  • mapping (and the values consist of another) –

  • probabilities. (between label classes and their) –

  • example –

  • class (to create a random RepertoireDataset with the label CMV where 70% of the Repertoires has) –

  • cmv_negative (cmv_positive and the remaining 30% has class) –

  • specified –

  • spaces –

  • code-block: –

    yaml: CMV:

    cmv_positive: 0.7 cmv_negative: 0.3

YAML specification:

my_random_dataset:
    format: RandomRepertoireDataset
    params:
        repertoire_count: 100 # number of random repertoires to generate
        sequence_count_probabilities:
            10: 0.5 # probability that any of the repertoires would have 10 receptor sequences
            20: 0.5
        sequence_length_probabilities:
            10: 0.5 # probability that any of the receptor sequences would be 10 amino acids in length
            12: 0.5
        labels: # randomly assigned labels (only useful for simple benchmarking)
            cmv:
                True: 0.5 # probability of value True for label cmv to be assigned to any repertoire
                False: 0.5
static import_dataset(params: dict, dataset_name: str) → immuneML.data_model.dataset.RepertoireDataset.RepertoireDataset[source]

immuneML.IO.dataset_import.RandomSequenceDatasetImport module

class immuneML.IO.dataset_import.RandomSequenceDatasetImport.RandomSequenceDatasetImport[source]

Bases: immuneML.IO.dataset_import.DataImport.DataImport

Returns a SequenceDataset consisting of randomly generated sequences, which can be used for benchmarking purposes. The sequences consist of uniformly chosen amino acids or nucleotides.

Parameters

  • sequence_count (int) – The number of sequences the SequenceDataset should contain.

  • length_probabilities (dict) – A mapping where the keys correspond to different sequence lengths

  • length. (and the values are the probabilities for choosing each sequence) –

  • example (For) –

  • 10 (to create a random SequenceDataset where 40% of the sequences would be of length) –

  • and –

  • 12 (60% of the sequences would have length) –

  • specified (the following mapping should be) –

  • spaces (. indent with) –

  • code-block: (.) – yaml: 10: 0.4 12: 0.6

  • labels (labels can be specified here. The keys of this mapping are the) – A mapping that specifies randomly chosen labels to be assigned to the sequences. One or multiple

  • labels –

  • mapping (and the values consist of another) –

  • probabilities. (between label classes and their) –

  • example –

  • class (to create a random SequenceDataset with the label cmv_epitope where 70% of the sequences has) –

  • not_binding (binding and the remaining 30% has class) –

  • specified –

  • spaces –

  • code-block: –

    yaml: cmv_epitope:

    binding: 0.7 not_binding: 0.3

YAML specification:

my_random_dataset:
    format: RandomSequenceDataset
    params:
        sequence_count: 100 # number of random sequences to generate
        length_probabilities:
            14: 0.8 # 80% of all generated sequences for all sequences will have length 14
            15: 0.2 # 20% of all generated sequences across all sequences will have length 15
        labels:
            epitope1: # label name
                True: 0.5 # 50% of the sequences will have class True
                False: 0.5 # 50% of the sequences will have class False
            epitope2: # next label with classes that will be assigned to sequences independently of the previous label or other parameters
                1: 0.3 # 30% of the generated sequences will have class 1
                0: 0.7 # 70% of the generated sequences will have class 0
static import_dataset(params, name: str) → immuneML.data_model.dataset.SequenceDataset.SequenceDataset[source]

Returns randomly generated receptor dataset according to the parameters;

YAML specification:

result_path: path/where/to/store/results/ sequence_count: 100 # number of random sequences to generate chain_1_length_probabilities:

14: 0.8 # 80% of all generated sequences for all sequences will have length 14 15: 0.2 # 20% of all generated sequences across all sequences will have length 15

labels:
epitope1: # label name

True: 0.5 # 50% of the sequences will have class True False: 0.5 # 50% of the sequences will have class False

epitope2: # next label with classes that will be assigned to sequences independently of the previous label or other parameters

1: 0.3 # 30% of the generated sequences will have class 1 0: 0.7 # 70% of the generated sequences will have class 0

immuneML.IO.dataset_import.SingleLineReceptorImport module

class immuneML.IO.dataset_import.SingleLineReceptorImport.SingleLineReceptorImport[source]

Bases: immuneML.IO.dataset_import.DataImport.DataImport

Imports data from a tabular file (where each line contains a pair of immune receptor sequences) into a ReceptorDataset. If you instead want to import a ReceptorDataset from a tabular file that contains one receptor sequence per line, see Generic import.

Parameters

  • path (str) – Required parameter. This is the path to a directory with files to import.

  • receptor_chains (str) – Required parameter. Determines which pair of chains to import for each Receptor.

:param Valid values for receptor_chains are the names of the ChainPair enum.: :param import_empty_nt_sequences: imports sequences which have an empty nucleotide sequence field; can be True or False. :type import_empty_nt_sequences: bool :param By default: :param import_empty_nt_sequences is set to True.: :param import_empty_aa_sequences: imports sequences which have an empty amino acid sequence field; can be True or False; for analysis on :type import_empty_aa_sequences: bool :param amino acid sequences: :type amino acid sequences: import only non-empty amino acid sequences :param this parameter should be False: :type this parameter should be False: import only non-empty amino acid sequences :param region_type: Which part of the sequence to import. When IMGT_CDR3 is specified, immuneML assumes the IMGT :type region_type: str :param junction: :type junction: including leading C and trailing Y/F amino acids :param amino acids will be removed from the sequences to retrieve the IMGT CDR3 sequence. Specifying any other value: :param will result in importing the sequences as they are.: :param Valid values for region_type are the names of the RegionType enum.: :param column_mapping: A mapping where the keys are the column names in the input file, and the values must be :type column_mapping: dict :param mapped to the following fields: <chain>_amino_acid_sequence, <chain>_nucleotide_sequence, <chain>_v_gene, :param <chain>_j_gene: :param identifier: :param epitope.: :param The possible names that can be filled in for <chain> are given in Chain: :param Any column namme other than the sequence: :param v/j genes and identifier will be set as metadata fields to the: :param Receptors: :param and can subsequently be used as labels in immuneML instructions.: :param For TCR alpha-beta receptor import: :param a column mapping could for example look like this: :param .. indent with spaces: :param .. code-block:: yaml: cdr3_a_aa: alpha_amino_acid_sequence

cdr3_b_aa: beta_amino_acid_sequence cdr3_a_nucseq: alpha_nucleotide_sequence cdr3_b_nucseq: beta_nucleotide_sequence v_a_gene: alpha_v_gene v_b_gene: beta_v_gene j_a_gene: alpha_j_gene j_b_gene: beta_j_gene clone_id: identifier epitope: epitope # metadata field

Parameters

  • column_mapping_synonyms (dict) – This is a column mapping that can be used if a column could have alternative names.

  • file (The formatting is the same as column_mapping. If some columns specified in column_mapping are not found in the) –

:param : :param the columns specified in column_mapping_synonyms are instead attempted to be loaded.: :param columns_to_load: Optional; specifies which columns to load from the input file. This may be useful if :type columns_to_load: list :param the input files contain many unused columns. If no value is specified: :param all columns are loaded.: :param separator: Required parameter. Column separator, for example “t” or “,”. :type separator: str :param organism: The organism that the receptors came from. This will be set as a parameter in the ReceptorDataset object. :type organism: str

YAML specification:

my_receptor_dataset:
    format: SingleLineReceptor
    params:
        path: path/to/files/
        receptor_chains: TRA_TRB # what chain pair to import
        separator: "\t" # column separator
        import_empty_nt_sequences: True # keep sequences even though the nucleotide sequence might be empty
        import_empty_aa_sequences: False # filter out sequences if they don't have sequence_aa set
        region_type: IMGT_CDR3 # what part of the sequence to import
        columns_to_load: # which subset of columns to load from the file
        - subject
        - epitope
        - count
        - v_a_gene
        - j_a_gene
        - cdr3_a_aa
        - v_b_gene
        - j_b_gene
        - cdr3_b_aa
        - clone_id
        column_mapping: # column mapping file: immuneML
            cdr3_a_aa: alpha_amino_acid_sequence
            cdr3_b_aa: beta_amino_acid_sequence
            cdr3_a_nucseq: alpha_nucleotide_sequence
            cdr3_b_nucseq: beta_nucleotide_sequence
            v_a_gene: alpha_v_gene
            v_b_gene: beta_v_gene
            j_a_gene: alpha_j_gene
            j_b_gene: beta_j_gene
            clone_id: identifier
            epitope: epitope
            organism: mouse
static get_documentation()[source]
static import_dataset(params, dataset_name: str) → immuneML.data_model.dataset.ReceptorDataset.ReceptorDataset[source]
static make_gene_columns(df: pandas.DataFrame, genes: list, chain_name=None)[source]

immuneML.IO.dataset_import.TenxGenomicsImport module

class immuneML.IO.dataset_import.TenxGenomicsImport.TenxGenomicsImport[source]

Bases: immuneML.IO.dataset_import.DataImport.DataImport

Imports data from the 10x Genomics Cell Ranger analysis pipeline into a Repertoire-, Sequence- or ReceptorDataset. RepertoireDatasets should be used when making predictions per repertoire, such as predicting a disease state. SequenceDatasets or ReceptorDatasets should be used when predicting values for unpaired (single-chain) and paired immune receptors respectively, like antigen specificity.

The files that should be used as input are named ‘Clonotype consensus annotations (CSV)’, as described here: https://support.10xgenomics.com/single-cell-vdj/software/pipelines/latest/output/annotation#consensus

Note: by default the 10xGenomics field ‘umis’ is used to define the immuneML field counts. If you want to use the 10x Genomics field reads instead, this can be changed in the column_mapping (set reads: counts). Furthermore, the 10xGenomics field clonotype_id is used for the immuneML field cell_id.

Parameters

  • path (str) – This is the path to a directory with 10xGenomics files to import. By default path is set to the current working directory.

  • is_repertoire (bool) – If True, this imports a RepertoireDataset. If False, it imports a SequenceDataset or

  • default (ReceptorDataset. By) –

  • True. (is_repertoire is set to) –

  • metadata_file (str) – Required for RepertoireDatasets. This parameter specifies the path to the metadata file.

  • filename (This is a csv file with columns) –

  • instructions. (subject_id and arbitrary other columns which can be used as labels in) –

  • metadata (For setting Sequence- or ReceptorDataset) –

  • ignored (metadata_file is) –

  • instead. (see metadata_column_mapping) –

  • paired (str) – Required for Sequence- or ReceptorDatasets. This parameter determines whether to import a

  • SequenceDataset (paired = False) or a ReceptorDataset (paired = True) –

  • ReceptorDataset (In a) –

  • together (two sequences with chain types specified by receptor_chains are paired) –

  • 'clonotype_id'. (based on the identifier given in the 10xGenomics column named) –

  • receptor_chains (str) – Required for ReceptorDatasets. Determines which pair of chains to import for each Receptor.

:param Valid values for receptor_chains are the names of the ChainPair enum.: :param If receptor_chains is not provided: :type If receptor_chains is not provided: only one chain pair type allowed per repertoire :param the chain pair is automatically detected: :type the chain pair is automatically detected: only one chain pair type allowed per repertoire :param import_illegal_characters: Whether to import sequences that contain illegal characters, i.e., characters :type import_illegal_characters: bool :param that do not appear in the sequence alphabet: :type that do not appear in the sequence alphabet: amino acids including stop codon ‘*’, or nucleotides :param applied to the sequence type of interest (when running immuneML in amino acid mode: :param only entries with illegal: :param characters in the amino acid sequence are removed). By default import_illegal_characters is False.: :param import_empty_nt_sequences: imports sequences which have an empty nucleotide sequence field; can be True or False. :type import_empty_nt_sequences: bool :param By default: :param import_empty_nt_sequences is set to True.: :param import_empty_aa_sequences: imports sequences which have an empty amino acid sequence field; can be True or False; for analysis on :type import_empty_aa_sequences: bool :param amino acid sequences: :type amino acid sequences: import only non-empty amino acid sequences :param this parameter should be False: :type this parameter should be False: import only non-empty amino acid sequences :param region_type: Which part of the sequence to import. By default, this value is set to IMGT_CDR3. This means the :type region_type: str :param first and last amino acids are removed from the CDR3 sequence: :param as 10xGenomics uses IMGT junction as CDR3. Specifying: :param any other value will result in importing the sequences as they are.: :param Valid values for region_type are the names of the RegionType enum.: :param column_mapping: A mapping from 10xGenomics column names to immuneML’s internal data representation. :type column_mapping: dict :param For 10xGenomics: :param this is by default set to: :param .. indent with spaces: :param .. code-block:: yaml: cdr3: sequence_aas

cdr3_nt: sequences v_gene: v_genes j_gene: j_genes umis: counts chain: chains clonotype_id: cell_ids consensus_id: sequence_identifiers

Parameters

  • if (A custom column mapping can be specified here if necessary (for example; adding additional data fields) –

  • file (The formatting is the same as column_mapping. If some columns specified in column_mapping are not found in the) –

  • names) (or using alternative column) –

  • Repertoire.FIELDS (Valid immuneML fields that can be specified here are defined by) –

  • column_mapping_synonyms (dict) – This is a column mapping that can be used if a column could have alternative names.

  • file –

:param : :param the columns specified in column_mapping_synonyms are instead attempted to be loaded.: :param For 10xGenomics format: :param there is no default column_mapping_synonyms.: :param metadata_column_mapping: Specifies metadata for Sequence- and ReceptorDatasets. This should specify a :type metadata_column_mapping: dict :param mapping similar to column_mapping where keys are 10xGenomics column names and values are the names that are internally: :param used in immuneML as metadata fields. These metadata fields can be used as prediction labels for Sequence-: :param and ReceptorDatasets. For 10xGenomics format: :param there is no default metadata_column_mapping.: :param For setting RepertoireDataset metadata: :param metadata_column_mapping is ignored: :param see metadata_file instead.: :param separator: Column separator, for 10xGenomics this is by default “,”. :type separator: str

YAML specification:

my_10x_dataset:
    format: 10xGenomics
    params:
        path: path/to/files/
        is_repertoire: True # whether to import a RepertoireDataset
        metadata_file: path/to/metadata.csv # metadata file for RepertoireDataset
        paired: False # whether to import SequenceDataset (False) or ReceptorDataset (True) when is_repertoire = False
        receptor_chains: TRA_TRB # what chain pair to import for a ReceptorDataset
        metadata_column_mapping: # metadata column mapping 10xGenomics: immuneML for SequenceDataset
            tenx_column_name1: metadata_label1
            tenx_column_name2: metadata_label2
        import_illegal_characters: False # remove sequences with illegal characters for the sequence_type being used
        import_empty_nt_sequences: True # keep sequences even though the nucleotide sequence might be empty
        import_empty_aa_sequences: False # filter out sequences if they don't have sequence_aa set
        # Optional fields with 10xGenomics-specific defaults, only change when different behavior is required:
        separator: "," # column separator
        region_type: IMGT_CDR3 # what part of the sequence to import
        column_mapping: # column mapping 10xGenomics: immuneML
            cdr3: sequence_aas
            cdr3_nt: sequences
            v_gene: v_genes
            j_gene: j_genes
            umis: counts
            chain: chains
            clonotype_id: cell_ids
            consensus_id: sequence_identifiers
static get_documentation()[source]
static import_dataset(params: dict, dataset_name: str) → immuneML.data_model.dataset.Dataset.Dataset[source]
static import_receptors(df, params)[source]
static preprocess_dataframe(df: pandas.DataFrame, params: immuneML.IO.dataset_import.DatasetImportParams.DatasetImportParams)[source]

immuneML.IO.dataset_import.VDJdbImport module

class immuneML.IO.dataset_import.VDJdbImport.VDJdbImport[source]

Bases: immuneML.IO.dataset_import.DataImport.DataImport

Imports data in VDJdb format into a Repertoire-, Sequence- or ReceptorDataset. RepertoireDatasets should be used when making predictions per repertoire, such as predicting a disease state. SequenceDatasets or ReceptorDatasets should be used when predicting values for unpaired (single-chain) and paired immune receptors respectively, like antigen specificity.

Parameters

  • path (str) – This is the path to a directory with VDJdb files to import. By default path is set to the current working directory.

  • is_repertoire (bool) – If True, this imports a RepertoireDataset. If False, it imports a SequenceDataset or

  • default (ReceptorDataset. By) –

  • True. (is_repertoire is set to) –

  • metadata_file (str) – Required for RepertoireDatasets. This parameter specifies the path to the metadata file.

  • filename (This is a csv file with columns) –

  • instructions. (subject_id and arbitrary other columns which can be used as labels in) –

  • metadata (For setting Sequence- or ReceptorDataset) –

  • ignored (metadata_file is) –

  • instead. (see metadata_column_mapping) –

  • paired (str) – Required for Sequence- or ReceptorDatasets. This parameter determines whether to import a

  • SequenceDataset (paired = False) or a ReceptorDataset (paired = True) –

  • ReceptorDataset (In a) –

  • together (two sequences with chain types specified by receptor_chains are paired) –

  • 'complex.id'. (based on the identifier given in the VDJdb column named) –

  • receptor_chains (str) – Required for ReceptorDatasets. Determines which pair of chains to import for each Receptor.

:param Valid values for receptor_chains are the names of the ChainPair enum.: :param If receptor_chains is not provided: :type If receptor_chains is not provided: only one chain pair type allowed per repertoire :param the chain pair is automatically detected: :type the chain pair is automatically detected: only one chain pair type allowed per repertoire :param import_illegal_characters: Whether to import sequences that contain illegal characters, i.e., characters :type import_illegal_characters: bool :param that do not appear in the sequence alphabet: :type that do not appear in the sequence alphabet: amino acids including stop codon ‘*’, or nucleotides :param applied to the sequence type of interest (when running immuneML in amino acid mode: :param only entries with illegal: :param characters in the amino acid sequence are removed). By default import_illegal_characters is False.: :param import_empty_nt_sequences: imports sequences which have an empty nucleotide sequence field; can be True or False. :type import_empty_nt_sequences: bool :param By default: :param import_empty_nt_sequences is set to True.: :param import_empty_aa_sequences: imports sequences which have an empty amino acid sequence field; can be True or False; for analysis on :type import_empty_aa_sequences: bool :param amino acid sequences: :type amino acid sequences: import only non-empty amino acid sequences :param this parameter should be False: :type this parameter should be False: import only non-empty amino acid sequences :param region_type: Which part of the sequence to import. By default, this value is set to IMGT_CDR3. This means the :type region_type: str :param first and last amino acids are removed from the CDR3 sequence: :param as VDJdb uses IMGT junction as CDR3. Specifying: :param any other value will result in importing the sequences as they are.: :param Valid values for region_type are the names of the RegionType enum.: :param column_mapping: A mapping from VDJdb column names to immuneML’s internal data representation. :type column_mapping: dict :param For VDJdb: .. indent with spaces

V: v_alleles
J: j_alleles
CDR3: sequence_aas
complex.id: sequence_identifiers
Gene: chains
Parameters

  • to (this is by default set) – 

    V: v_alleles
J: j_alleles
CDR3: sequence_aas
complex.id: sequence_identifiers
Gene: chains

  • if (A custom column mapping can be specified here if necessary (for example; adding additional data fields) –

  • file (The formatting is the same as column_mapping. If some columns specified in column_mapping are not found in the) –

  • names) (or using alternative column) –

  • Repertoire.FIELDS (Valid immuneML fields that can be specified here are defined by) –

  • column_mapping_synonyms (dict) – This is a column mapping that can be used if a column could have alternative names.

  • file –

:param : :param the columns specified in column_mapping_synonyms are instead attempted to be loaded.: :param For VDJdb format: :param there is no default column_mapping_synonyms.: :param metadata_column_mapping: Specifies metadata for Sequence- and ReceptorDatasets. This should specify :type metadata_column_mapping: dict :param a mapping where keys are VDJdb column names and values are the names that are internally used in immuneML: :param as metadata fields.: :param For VDJdb format: .. indent with spaces

Epitope: epitope
Epitope gene: epitope_gene
Epitope species: epitope_species
Parameters

  • to (this parameter is by default set) – 

    Epitope: epitope
Epitope gene: epitope_gene
Epitope species: epitope_species

  • epitope (This means that) –

  • for (epitope_gene and epitope_species can be specified as prediction labels) –

  • well. (Sequence- and ReceptorDatasets. Custom metadata labels can be defined here as) –

  • metadata (For setting RepertoireDataset) –

  • ignored (metadata_column_mapping is) –

  • instead. (see metadata_file) –

  • separator (str) – Column separator, for VDJdb this is by default “t”.

YAML specification:

my_vdjdb_dataset:
    format: VDJdb
    params:
        path: path/to/files/
        is_repertoire: True # whether to import a RepertoireDataset
        metadata_file: path/to/metadata.csv # metadata file for RepertoireDataset
        paired: False # whether to import SequenceDataset (False) or ReceptorDataset (True) when is_repertoire = False
        receptor_chains: TRA_TRB # what chain pair to import for a ReceptorDataset
        import_illegal_characters: False # remove sequences with illegal characters for the sequence_type being used
        import_empty_nt_sequences: True # keep sequences even though the nucleotide sequence might be empty
        import_empty_aa_sequences: False # filter out sequences if they don't have sequence_aa set
        # Optional fields with VDJdb-specific defaults, only change when different behavior is required:
        separator: "\t" # column separator
        region_type: IMGT_CDR3 # what part of the sequence to import
        column_mapping: # column mapping VDJdb: immuneML
            V: v_genes
            J: j_genes
            CDR3: sequence_aas
            complex.id: sequence_identifiers
            Gene: chains
        metadata_column_mapping: # metadata column mapping VDJdb: immuneML
            Epitope: epitope
            Epitope gene: epitope_gene
            Epitope species: epitope_species
KEY_MAPPING = {'subject.id': 'subject_id'}
static extract_meta_columns(df: pandas.DataFrame, params) → pandas.DataFrame[source]

extracts values from meta columns in VDJdb format to separate columns in the data frame, using VDJdbImport.KEY_MAPPING

Note: the KEY_MAPPING values cannot be manually set, but correspond to the standard names as defined by the AIRR Community

Parameters

  • df – data frame of from file[s] in VDJdb which have already been preprocessed

  • params – parameters from importing dataset which includes the information on metadata column mapping

Returns

the data frame with additional columns where the metadata (if present) were extracted

static get_documentation()[source]
static get_sequence_identifiers(receptor_identifiers, chains)[source]
static import_dataset(params: dict, dataset_name: str) → immuneML.data_model.dataset.Dataset.Dataset[source]
static import_receptors(df, params)[source]
static preprocess_dataframe(df: pandas.DataFrame, params: immuneML.IO.dataset_import.DatasetImportParams.DatasetImportParams)[source]

Module contents