Standardize and append a batch of data#

Here, we’ll learn

how to standardize a less well curated dataset
how to append it as a new array shard to the growing versioned dataset

import lamindb as ln
import lnschema_bionty as lb

ln.settings.verbosity = "hint"
ln.track()

💡 lamindb instance: testuser1/test-scrna

💡 notebook imports: lamindb==0.64.2 lnschema_bionty==0.36.1

💡 saved: Transform(uid='ManDYgmftZ8Cz8', name='Standardize and append a batch of data', short_name='scrna2', version='0', type=notebook, updated_at=2023-12-22 11:25:39 UTC, created_by_id=1)

💡 saved: Run(uid='RCjze5MCnLTqsjL2o5jC', run_at=2023-12-22 11:25:39 UTC, transform_id=2, created_by_id=1)

💡 tracked pip freeze > /home/runner/.cache/lamindb/run_env_pip_RCjze5MCnLTqsjL2o5jC

Standardize a data shard#

Let’s now consider a dataset with less-well curated features:

adata = ln.dev.datasets.anndata_pbmc68k_reduced()
adata

We are still working with human data, and can globally instruct bionty to assume human:

lb.settings.organism = "human"

Standardize & validate genes #

This data shard is indexed by gene symbols which we’ll want to map on Ensemble ids:

adata.var.head()

Show code cell output Hide code cell output

	n_counts	highly_variable
index
HES4	1153.387451	True
TNFRSF4	304.358154	True
SSU72	2530.272705	False
PARK7	7451.664062	False
RBP7	272.811035	True

Let’s inspect the identifiers:

lb.Gene.inspect(adata.var.index, lb.Gene.symbol)

Let’s first standardize the gene symbols from synonyms:

adata.var.index = lb.Gene.standardize(adata.var.index, lb.Gene.symbol)
validated = lb.Gene.validate(adata.var.index, lb.Gene.symbol)

💡 standardized 749/765 terms

✅ 749 terms (97.90%) are validated for symbol

❗ 16 terms (2.10%) are not validated for symbol: RP11-782C8.1, RP11-277L2.3, RP11-156E8.1, GPX1, RP3-467N11.1, SOD2, RP11-390E23.6, RP11-489E7.4, RP11-291B21.2, RP11-620J15.3, TMBIM4-1, AC084018.1, RN7SL1, SNORD3B-2, CTD-3138B18.5, IGLL5

We only want to register data with validated genes:

adata_validated = adata[:, validated].copy()

Now that all symbols are validated, let’s convert them to Ensembl ids via standardize(). Note that this is an ambiguous mapping and the first match is kept because the keep arg of .standardize() defaults to "first":

adata_validated.var["ensembl_gene_id"] = lb.Gene.standardize(
    adata_validated.var.index,
    field=lb.Gene.symbol,
    return_field=lb.Gene.ensembl_gene_id,
)
adata_validated.var.index.name = "symbol"
adata_validated.var = adata_validated.var.reset_index().set_index("ensembl_gene_id")
adata_validated.var.head()

Show code cell output Hide code cell output

💡 standardized 749/749 terms

	symbol	n_counts	highly_variable
ensembl_gene_id
ENSG00000188290	HES4	1153.387451	True
ENSG00000186827	TNFRSF4	304.358154	True
ENSG00000160075	SSU72	2530.272705	False
ENSG00000116288	PARK7	7451.664062	False
ENSG00000162444	RBP7	272.811035	True

Here, we’ll use .raw:

adata_validated.raw = adata.raw[:, validated].to_adata()
adata_validated.raw.var.index = adata_validated.var.index

Standardize & validate cell types #

Inspection shows none of the terms are validated:

inspector = lb.CellType.inspect(adata_validated.obs.cell_type)

Let us search the cell type names from the public ontology, and add the name found in the AnnData object as a synonym to the top match found in the public ontology.

bionty = lb.CellType.bionty()  # access the public ontology through bionty
name_mapper = {}
for name in adata_validated.obs.cell_type.unique():
    # search the public ontology and use the ontology id of the top match
    ontology_id = bionty.search(name).iloc[0].ontology_id
    # create a record by loading the top match from bionty
    record = lb.CellType.from_bionty(ontology_id=ontology_id)
    name_mapper[name] = record.name  # map the original name to standardized name
    record.save()  # save the record
    # add the original name as a synonym, so that next time, we can just run .standardize()
    record.add_synonym(name)

Show code cell output Hide code cell output

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000451'

💡 also saving parents of CellType(uid='9JGbXeUA', name='dendritic cell', ontology_id='CL:0000451', description='A Cell Of Hematopoietic Origin, Typically Resident In Particular Tissues, Specialized In The Uptake, Processing, And Transport Of Antigens To Lymph Nodes For The Purpose Of Stimulating An Immune Response Via T Cell Activation. These Cells Are Lineage Negative (Cd3-Negative, Cd19-Negative, Cd34-Negative, And Cd56-Negative).', updated_at=2023-12-22 11:25:45 UTC, bionty_source_id=21, created_by_id=1)

✅ created 2 CellType records from Bionty matching ontology_id: 'CL:0000842', 'CL:0000145'

❗ now recursing through parents: this only happens once, but is much slower than bulk saving

💡 you can switch this off via: lb.settings.auto_save_parents = False

💡 also saving parents of CellType(uid='3iDuKkvT', name='mononuclear cell', ontology_id='CL:0000842', synonyms='mononuclear leukocyte', description='A Leukocyte With A Single Non-Segmented Nucleus In The Mature Form.', updated_at=2023-12-22 11:25:46 UTC, bionty_source_id=21, created_by_id=1)

✅ created 2 CellType records from Bionty matching ontology_id: 'CL:0000738', 'CL:0000226'

💡 also saving parents of CellType(uid='MkrH0gsX', name='leukocyte', ontology_id='CL:0000738', synonyms='white blood cell|leucocyte', description='An Achromatic Cell Of The Myeloid Or Lymphoid Lineages Capable Of Ameboid Movement, Found In Blood Or Other Tissue.', updated_at=2023-12-22 11:25:46 UTC, bionty_source_id=21, created_by_id=1)

✅ created 3 CellType records from Bionty matching ontology_id: 'CL:0000219', 'CL:0002242', 'CL:0000988'

💡 also saving parents of CellType(uid='ek6M40lJ', name='motile cell', ontology_id='CL:0000219', description='A Cell That Moves By Its Own Activities.', updated_at=2023-12-22 11:25:47 UTC, bionty_source_id=21, created_by_id=1)

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000003'

💡 also saving parents of CellType(uid='VT73gpK2', name='native cell', ontology_id='CL:0000003', description='A Cell That Is Found In A Natural Setting, Which Includes Multicellular Organism Cells 'In Vivo' (I.E. Part Of An Organism), And Unicellular Organisms 'In Environment' (I.E. Part Of A Natural Environment).', updated_at=2023-12-22 11:25:47 UTC, bionty_source_id=21, created_by_id=1)

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000000'

💡 also saving parents of CellType(uid='P0lR3Ehm', name='nucleate cell', ontology_id='CL:0002242', description='A Cell Containing At Least One Nucleus.', updated_at=2023-12-22 11:25:47 UTC, bionty_source_id=21, created_by_id=1)

💡 also saving parents of CellType(uid='Q0aQr5JB', name='hematopoietic cell', ontology_id='CL:0000988', synonyms='haematopoietic cell|hemopoietic cell|haemopoietic cell', description='A Cell Of A Hematopoietic Lineage.', updated_at=2023-12-22 11:25:47 UTC, bionty_source_id=21, created_by_id=1)

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0002371'

💡 also saving parents of CellType(uid='QMAH6IlS', name='somatic cell', ontology_id='CL:0002371', description='A Cell Of An Organism That Does Not Pass On Its Genetic Material To The Organism'S Offspring (I.E. A Non-Germ Line Cell).', updated_at=2023-12-22 11:25:48 UTC, bionty_source_id=21, created_by_id=1)

💡 also saving parents of CellType(uid='6ADi80vK', name='single nucleate cell', ontology_id='CL:0000226', description='A Cell With A Single Nucleus.', updated_at=2023-12-22 11:25:46 UTC, bionty_source_id=21, created_by_id=1)

💡 also saving parents of CellType(uid='Iv0Y02ZA', name='professional antigen presenting cell', ontology_id='CL:0000145', description='A Cell Capable Of Processing And Presenting Lipid And Protein Antigens To T Cells In Order To Initiate An Immune Response.', updated_at=2023-12-22 11:25:46 UTC, bionty_source_id=21, created_by_id=1)

💡 also saving parents of CellType(uid='9JGbXeUA', name='dendritic cell', ontology_id='CL:0000451', synonyms='Dendritic cells', description='A Cell Of Hematopoietic Origin, Typically Resident In Particular Tissues, Specialized In The Uptake, Processing, And Transport Of Antigens To Lymph Nodes For The Purpose Of Stimulating An Immune Response Via T Cell Activation. These Cells Are Lineage Negative (Cd3-Negative, Cd19-Negative, Cd34-Negative, And Cd56-Negative).', updated_at=2023-12-22 11:25:48 UTC, bionty_source_id=21, created_by_id=1)

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0001201'

💡 also saving parents of CellType(uid='CIS4VJI0', name='B cell, CD19-positive', ontology_id='CL:0001201', synonyms='B-cell, CD19-positive|B lymphocyte, CD19-positive|CD19-positive B cell|B-lymphocyte, CD19-positive|CD19+ B cell', description='A B Cell That Is Cd19-Positive.', updated_at=2023-12-22 11:25:49 UTC, bionty_source_id=21, created_by_id=1)

✅ created 2 CellType records from Bionty matching ontology_id: 'CL:0001200', 'CL:0000236'

❗ now recursing through parents: this only happens once, but is much slower than bulk saving

💡 you can switch this off via: lb.settings.auto_save_parents = False

💡 also saving parents of CellType(uid='YsKb7oIy', name='lymphocyte of B lineage, CD19-positive', ontology_id='CL:0001200', description='A Lymphocyte Of B Lineage That Is Cd19-Positive.', updated_at=2023-12-22 11:25:49 UTC, bionty_source_id=21, created_by_id=1)

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000945'

💡 also saving parents of CellType(uid='Z0yFV7vU', name='lymphocyte of B lineage', ontology_id='CL:0000945', description='A Lymphocyte Of B Lineage With The Commitment To Express An Immunoglobulin Complex.', updated_at=2023-12-22 11:25:50 UTC, bionty_source_id=21, created_by_id=1)

💡 also saving parents of CellType(uid='cx8VcggA', name='B cell', ontology_id='CL:0000236', synonyms='B-lymphocyte|B lymphocyte|B-cell', description='A Lymphocyte Of B Lineage That Is Capable Of B Cell Mediated Immunity.', updated_at=2023-12-22 11:25:49 UTC, bionty_source_id=21, created_by_id=1)

💡 also saving parents of CellType(uid='CIS4VJI0', name='B cell, CD19-positive', ontology_id='CL:0001201', synonyms='B-cell, CD19-positive|B lymphocyte, CD19-positive|CD19+ B cell|CD19-positive B cell|B-lymphocyte, CD19-positive|CD19+ B', description='A B Cell That Is Cd19-Positive.', updated_at=2023-12-22 11:25:50 UTC, bionty_source_id=21, created_by_id=1)

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0001087'

💡 also saving parents of CellType(uid='6VQXlWS7', name='effector memory CD4-positive, alpha-beta T cell, terminally differentiated', ontology_id='CL:0001087', synonyms='CD4-positive TEMRA|CD4+ TEMRA', description='A Cd4-Positive, Alpha Beta Memory T Cell With The Phenotype Cd45Ra-Positive, Cd45Ro-Negative, And Ccr7-Negative.', updated_at=2023-12-22 11:25:50 UTC, bionty_source_id=21, created_by_id=1)

✅ created 2 CellType records from Bionty matching ontology_id: 'CL:4030002', 'CL:0000897'

❗ now recursing through parents: this only happens once, but is much slower than bulk saving

💡 you can switch this off via: lb.settings.auto_save_parents = False

💡 also saving parents of CellType(uid='ylUbqlrS', name='effector memory CD45RA-positive, alpha-beta T cell, terminally differentiated', ontology_id='CL:4030002', synonyms='TEMRA cell|terminally differentiated effector memory CD45RA+ T cells|terminally differentiated effector memory cells re-expressing CD45RA', description='An Alpha-Beta Memory T Cell With The Phenotype Cd45Ra-Positive.', updated_at=2023-12-22 11:25:51 UTC, bionty_source_id=21, created_by_id=1)

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000791'

💡 also saving parents of CellType(uid='WKpZjuYS', name='mature alpha-beta T cell', ontology_id='CL:0000791', synonyms='mature alpha-beta T-cell|mature alpha-beta T lymphocyte|mature alpha-beta T-lymphocyte', description='A Alpha-Beta T Cell That Has A Mature Phenotype.', updated_at=2023-12-22 11:25:51 UTC, bionty_source_id=21, created_by_id=1)

✅ loaded 1 CellType record matching ontology_id: 'CL:0000789'

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0002419'

💡 also saving parents of CellType(uid='2C5PhwrW', name='mature T cell', ontology_id='CL:0002419', synonyms='mature T-cell|CD3e-positive T cell', description='A T Cell That Expresses A T Cell Receptor Complex And Has Completed T Cell Selection.', updated_at=2023-12-22 11:25:52 UTC, bionty_source_id=21, created_by_id=1)

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000084'

💡 also saving parents of CellType(uid='BxNjby0x', name='T cell', ontology_id='CL:0000084', synonyms='T-cell|T-lymphocyte|T lymphocyte', description='A Type Of Lymphocyte Whose Defining Characteristic Is The Expression Of A T Cell Receptor Complex.', updated_at=2023-12-22 11:25:52 UTC, bionty_source_id=21, created_by_id=1)

💡 also saving parents of CellType(uid='s6Ag7R5U', name='CD4-positive, alpha-beta memory T cell', ontology_id='CL:0000897', synonyms='CD4-positive, alpha-beta memory T lymphocyte|CD4-positive, alpha-beta memory T-lymphocyte|CD4-positive, alpha-beta memory T-cell', description='A Cd4-Positive, Alpha-Beta T Cell That Has Differentiated Into A Memory T Cell.', updated_at=2023-12-22 11:25:51 UTC, bionty_source_id=21, created_by_id=1)

✅ created 2 CellType records from Bionty matching ontology_id: 'CL:0000813', 'CL:0000624'

💡 also saving parents of CellType(uid='Re00kg0W', name='memory T cell', ontology_id='CL:0000813', synonyms='memory T-cell|memory T lymphocyte|memory T-lymphocyte', description='A Long-Lived, Antigen-Experienced T Cell That Has Acquired A Memory Phenotype Including Distinct Surface Markers And The Ability To Differentiate Into An Effector T Cell Upon Antigen Reexposure.', updated_at=2023-12-22 11:25:53 UTC, bionty_source_id=21, created_by_id=1)

💡 also saving parents of CellType(uid='05vQoepH', name='CD4-positive, alpha-beta T cell', ontology_id='CL:0000624', synonyms='CD4-positive, alpha-beta T lymphocyte|CD4-positive, alpha-beta T-cell|CD4-positive, alpha-beta T-lymphocyte', description='A Mature Alpha-Beta T Cell That Expresses An Alpha-Beta T Cell Receptor And The Cd4 Coreceptor.', updated_at=2023-12-22 11:25:53 UTC, bionty_source_id=21, created_by_id=1)

💡 also saving parents of CellType(uid='6VQXlWS7', name='effector memory CD4-positive, alpha-beta T cell, terminally differentiated', ontology_id='CL:0001087', synonyms='CD4+/CD45RO+ Memory|CD4+ TEMRA|CD4-positive TEMRA', description='A Cd4-Positive, Alpha Beta Memory T Cell With The Phenotype Cd45Ra-Positive, Cd45Ro-Negative, And Ccr7-Negative.', updated_at=2023-12-22 11:25:53 UTC, bionty_source_id=21, created_by_id=1)

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000910'

💡 also saving parents of CellType(uid='OxsmyL44', name='cytotoxic T cell', ontology_id='CL:0000910', synonyms='cytotoxic T lymphocyte|cytotoxic T-lymphocyte|cytotoxic T-cell', description='A Mature T Cell That Differentiated And Acquired Cytotoxic Function With The Phenotype Perforin-Positive And Granzyme-B Positive.', updated_at=2023-12-22 11:25:53 UTC, bionty_source_id=21, created_by_id=1)

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000911'

❗ now recursing through parents: this only happens once, but is much slower than bulk saving

💡 you can switch this off via: lb.settings.auto_save_parents = False

💡 also saving parents of CellType(uid='yvHkIrVI', name='effector T cell', ontology_id='CL:0000911', synonyms='effector T-cell|effector T-lymphocyte|effector T lymphocyte', description='A Differentiated T Cell With Ability To Traffic To Peripheral Tissues And Is Capable Of Mounting A Specific Immune Response.', updated_at=2023-12-22 11:25:54 UTC, bionty_source_id=21, created_by_id=1)

💡 also saving parents of CellType(uid='OxsmyL44', name='cytotoxic T cell', ontology_id='CL:0000910', synonyms='cytotoxic T-lymphocyte|CD8+ Cytotoxic T|cytotoxic T lymphocyte|cytotoxic T-cell', description='A Mature T Cell That Differentiated And Acquired Cytotoxic Function With The Phenotype Perforin-Positive And Granzyme-B Positive.', updated_at=2023-12-22 11:25:54 UTC, bionty_source_id=21, created_by_id=1)

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000919'

💡 also saving parents of CellType(uid='ORD0dMdt', name='CD8-positive, CD25-positive, alpha-beta regulatory T cell', ontology_id='CL:0000919', synonyms='CD8+CD25+ T cell|CD8+CD25+ T-cell|CD8+CD25+ T(reg)|CD8-positive, CD25-positive, alpha-beta regulatory T-lymphocyte|CD8+CD25+ T lymphocyte|CD8-positive, CD25-positive, alpha-beta regulatory T-cell|CD8+CD25+ T-lymphocyte|CD8+CD25+ Treg|CD8-positive, CD25-positive Treg|CD8-positive, CD25-positive, alpha-beta regulatory T lymphocyte', description='A Cd8-Positive Alpha Beta-Positive T Cell With The Phenotype Foxp3-Positive And Having Suppressor Function.', updated_at=2023-12-22 11:25:55 UTC, bionty_source_id=21, created_by_id=1)

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000795'

❗ now recursing through parents: this only happens once, but is much slower than bulk saving

💡 you can switch this off via: lb.settings.auto_save_parents = False

💡 also saving parents of CellType(uid='oTsFrhYW', name='CD8-positive, alpha-beta regulatory T cell', ontology_id='CL:0000795', synonyms='CD8-positive, alpha-beta regulatory T-cell|CD8-positive Treg|CD8-positive T(reg)|CD8+ regulatory T cell|CD8-positive, alpha-beta regulatory T lymphocyte|CD8-positive, alpha-beta Treg|CD8-positive, alpha-beta regulatory T-lymphocyte|CD8+ T(reg)|CD8+ Treg', description='A Cd8-Positive, Alpha-Beta T Cell That Regulates Overall Immune Responses As Well As The Responses Of Other T Cell Subsets Through Direct Cell-Cell Contact And Cytokine Release.', updated_at=2023-12-22 11:25:55 UTC, bionty_source_id=21, created_by_id=1)

✅ loaded 1 CellType record matching ontology_id: 'CL:0000815'

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000625'

💡 also saving parents of CellType(uid='VnKkQsME', name='CD8-positive, alpha-beta T cell', ontology_id='CL:0000625', synonyms='CD8-positive, alpha-beta T-cell|CD8-positive, alpha-beta T lymphocyte|CD8-positive, alpha-beta T-lymphocyte', description='A T Cell Expressing An Alpha-Beta T Cell Receptor And The Cd8 Coreceptor.', updated_at=2023-12-22 11:25:56 UTC, bionty_source_id=21, created_by_id=1)

💡 also saving parents of CellType(uid='ORD0dMdt', name='CD8-positive, CD25-positive, alpha-beta regulatory T cell', ontology_id='CL:0000919', synonyms='CD4+/CD25 T Reg|CD8+CD25+ T-lymphocyte|CD8+CD25+ T lymphocyte|CD8+CD25+ T cell|CD8-positive, CD25-positive Treg|CD8+CD25+ Treg|CD8-positive, CD25-positive, alpha-beta regulatory T-lymphocyte|CD8+CD25+ T(reg)|CD8+CD25+ T-cell|CD8-positive, CD25-positive, alpha-beta regulatory T-cell|CD8-positive, CD25-positive, alpha-beta regulatory T lymphocyte', description='A Cd8-Positive Alpha Beta-Positive T Cell With The Phenotype Foxp3-Positive And Having Suppressor Function.', updated_at=2023-12-22 11:25:56 UTC, bionty_source_id=21, created_by_id=1)

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0002057'

💡 also saving parents of CellType(uid='O0AQiAuv', name='CD14-positive, CD16-negative classical monocyte', ontology_id='CL:0002057', synonyms='CD16-negative monocyte|CD16- monocyte', description='A Classical Monocyte That Is Cd14-Positive, Cd16-Negative, Cd64-Positive, Cd163-Positive.', updated_at=2023-12-22 11:25:56 UTC, bionty_source_id=21, created_by_id=1)

✅ loaded 1 CellType record matching ontology_id: 'CL:0000860'

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0001054'

❗ now recursing through parents: this only happens once, but is much slower than bulk saving

💡 you can switch this off via: lb.settings.auto_save_parents = False

💡 also saving parents of CellType(uid='E5LdQF00', name='CD14-positive monocyte', ontology_id='CL:0001054', description='A Monocyte That Expresses Cd14 And Is Negative For The Lineage Markers Cd3, Cd19, And Cd20.', updated_at=2023-12-22 11:25:57 UTC, bionty_source_id=21, created_by_id=1)

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000576'

💡 also saving parents of CellType(uid='YzV7Qgmj', name='monocyte', ontology_id='CL:0000576', description='Myeloid Mononuclear Recirculating Leukocyte That Can Act As A Precursor Of Tissue Macrophages, Osteoclasts And Some Populations Of Tissue Dendritic Cells.', updated_at=2023-12-22 11:25:57 UTC, bionty_source_id=21, created_by_id=1)

✅ loaded 2 CellType records matching ontology_id: 'CL:0000842', 'CL:0011026'

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000766'

💡 also saving parents of CellType(uid='40onq0tm', name='myeloid leukocyte', ontology_id='CL:0000766', description='A Cell Of The Monocyte, Granulocyte, Or Mast Cell Lineage.', updated_at=2023-12-22 11:25:58 UTC, bionty_source_id=21, created_by_id=1)

✅ loaded 1 CellType record matching ontology_id: 'CL:0000738'

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0000763'

💡 also saving parents of CellType(uid='g1zY6vUW', name='myeloid cell', ontology_id='CL:0000763', description='A Cell Of The Monocyte, Granulocyte, Mast Cell, Megakaryocyte, Or Erythroid Lineage.', updated_at=2023-12-22 11:25:58 UTC, bionty_source_id=21, created_by_id=1)

💡 also saving parents of CellType(uid='O0AQiAuv', name='CD14-positive, CD16-negative classical monocyte', ontology_id='CL:0002057', synonyms='CD14+ Monocytes|CD16- monocyte|CD16-negative monocyte', description='A Classical Monocyte That Is Cd14-Positive, Cd16-Negative, Cd64-Positive, Cd163-Positive.', updated_at=2023-12-22 11:25:58 UTC, bionty_source_id=21, created_by_id=1)

✅ created 1 CellType record from Bionty matching ontology_id: 'CL:0002101'

💡 also saving parents of CellType(uid='a6U0yA2P', name='CD38-positive naive B cell', ontology_id='CL:0002101', synonyms='CD38-positive naive B-cell|CD38+ naive B cell|CD38+ naive B lymphocyte|CD38+ naive B-cell|CD38-positive naive B lymphocyte|CD38-positive naive B-lymphocyte|CD38+ naive B-lymphocyte', description='A Cd38-Positive Naive B Cell Is A Mature B Cell That Has The Phenotype Cd38-Positive, Surface Igd-Positive, Surface Igm-Positive, And Cd27-Negative, And That Has Not Yet Been Activated By Antigen In The Periphery.', updated_at=2023-12-22 11:25:59 UTC, bionty_source_id=21, created_by_id=1)

💡 also saving parents of CellType(uid='a6U0yA2P', name='CD38-positive naive B cell', ontology_id='CL:0002101', synonyms='CD38-positive naive B-cell|CD38+ naive B-cell|CD38-positive naive B lymphocyte|CD38+ naive B-lymphocyte|CD38-positive naive B-lymphocyte|CD8+/CD45RA+ Naive Cytotoxic|CD38+ naive B lymphocyte|CD38+ naive B cell', description='A Cd38-Positive Naive B Cell Is A Mature B Cell That Has The Phenotype Cd38-Positive, Surface Igd-Positive, Surface Igm-Positive, And Cd27-Negative, And That Has Not Yet Been Activated By Antigen In The Periphery.', updated_at=2023-12-22 11:25:59 UTC, bionty_source_id=21, created_by_id=1)

We can now standardize cell type names using the search-based mapper:

adata_validated.obs.cell_type = adata_validated.obs.cell_type.map(name_mapper)

Now, all cell types are validated:

validated = lb.CellType.validate(adata_validated.obs.cell_type)
assert all(validated)

✅ 9 terms (100.00%) are validated for name

We don’t want to store any of the other metadata columns:

for column in ["n_genes", "percent_mito", "louvain"]:
    adata.obs.drop(column, axis=1)

Register #

experimental_factors = lb.ExperimentalFactor.lookup()
organism = lb.Organism.lookup()
features = ln.Feature.lookup()

artifact = ln.Artifact.from_anndata(
    adata_validated,
    description="10x reference adata",
    field=lb.Gene.ensembl_gene_id,
)

As we do not want to manage the remaining unvalidated terms in registries, we can save and annotate the artifact:

artifact.save()
artifact.labels.add(adata_validated.obs.cell_type, features.cell_type)
artifact.labels.add(organism.human, feature=features.organism)
artifact.labels.add(
    experimental_factors.single_cell_rna_sequencing, feature=features.assay
)
artifact.describe()

✅ saved 2 feature sets for slots: 'var','obs'

✅ storing artifact 'VbND5kUT2jBNcK1iICSO' at '/home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna/.lamindb/VbND5kUT2jBNcK1iICSO.h5ad'

✅ loaded: FeatureSet(uid='5wmaXji2xoIIjTW0c8Li', n=1, registry='core.Feature', hash='SMcPr9uZdZXIUbrmVZgh', updated_at=2023-12-22 11:25:32 UTC, created_by_id=1)

✅ linked new feature 'organism' together with new feature set FeatureSet(uid='5wmaXji2xoIIjTW0c8Li', n=1, registry='core.Feature', hash='SMcPr9uZdZXIUbrmVZgh', updated_at=2023-12-22 11:26:00 UTC, created_by_id=1)

💡 no file links to it anymore, deleting feature set FeatureSet(uid='5wmaXji2xoIIjTW0c8Li', n=1, registry='core.Feature', hash='SMcPr9uZdZXIUbrmVZgh', updated_at=2023-12-22 11:26:00 UTC, created_by_id=1)

✅ linked new feature 'assay' together with new feature set FeatureSet(uid='fGqbKDo1LQcCaRObgqgc', n=2, registry='core.Feature', hash='X2ktoQzWvquazFsSgCik', updated_at=2023-12-22 11:26:00 UTC, created_by_id=1)

Artifact(uid='VbND5kUT2jBNcK1iICSO', suffix='.h5ad', accessor='AnnData', description='10x reference adata', size=853388, hash='eKH1ljAEh7Kd81-o2H4A7w', hash_type='md5', visibility=1, key_is_virtual=True, updated_at=2023-12-22 11:26:00 UTC)

Provenance:
  🗃️ storage: Storage(uid='kFrUCXTg', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna', type='local', updated_at=2023-12-22 11:25:11 UTC, created_by_id=1)
  💫 transform: Transform(uid='ManDYgmftZ8Cz8', name='Standardize and append a batch of data', short_name='scrna2', version='0', type=notebook, updated_at=2023-12-22 11:25:39 UTC, created_by_id=1)
  👣 run: Run(uid='RCjze5MCnLTqsjL2o5jC', run_at=2023-12-22 11:25:39 UTC, transform_id=2, created_by_id=1)
  👤 created_by: User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at=2023-12-22 11:25:11 UTC)
Features:
  var: FeatureSet(uid='qqD3HJwIKtgwJ2ITFyuK', n=749, type='number', registry='bionty.Gene', hash='ZL6ScVsUK3gvyaiIVPVr', updated_at=2023-12-22 11:25:59 UTC, created_by_id=1)
    'IL18', 'NPM3', 'S100A9', 'S100A8', 'CNN2', 'ARHGAP45', 'RNF34', 'GPX4', 'S100A6', 'ADISSP', 'S100A4', 'FAM174C', 'SIT1', 'CCDC107', 'RSL1D1', 'TLN1', 'HES4', 'TNFRSF17', 'PCNA', 'RAB13', ...
  obs: FeatureSet(uid='9Yi3Yrq2TawXOzRGoVZP', n=1, registry='core.Feature', hash='Lumlh13XvCeHWghLinRL', updated_at=2023-12-22 11:26:00 UTC, created_by_id=1)
    🔗 cell_type (9, bionty.CellType): 'dendritic cell', 'CD38-positive naive B cell', 'B cell, CD19-positive', 'effector memory CD4-positive, alpha-beta T cell, terminally differentiated', 'CD16-positive, CD56-dim natural killer cell, human', 'CD4-positive, alpha-beta T cell', 'cytotoxic T cell', 'CD8-positive, CD25-positive, alpha-beta regulatory T cell', 'CD14-positive, CD16-negative classical monocyte'
  external: FeatureSet(uid='fGqbKDo1LQcCaRObgqgc', n=2, registry='core.Feature', hash='X2ktoQzWvquazFsSgCik', updated_at=2023-12-22 11:26:00 UTC, created_by_id=1)
    🔗 assay (1, bionty.ExperimentalFactor): 'single-cell RNA sequencing'
    🔗 organism (1, bionty.Organism): 'human'
Labels:
  🏷️ organism (1, bionty.Organism): 'human'
  🏷️ cell_types (9, bionty.CellType): 'dendritic cell', 'CD38-positive naive B cell', 'B cell, CD19-positive', 'effector memory CD4-positive, alpha-beta T cell, terminally differentiated', 'CD16-positive, CD56-dim natural killer cell, human', 'CD4-positive, alpha-beta T cell', 'cytotoxic T cell', 'CD8-positive, CD25-positive, alpha-beta regulatory T cell', 'CD14-positive, CD16-negative classical monocyte'
  🏷️ experimental_factors (1, bionty.ExperimentalFactor): 'single-cell RNA sequencing'

artifact.view_flow()

_images/3258ca2a82ff88bde4293aa2fc95307fabc88087e0b020c42c1a694659d32002.svg

Append the shard to the dataset#

Query the previous dataset:

dataset_v1 = ln.Dataset.filter(name="My versioned scRNA-seq dataset", version="1").one()

Create a new version of the dataset by sharding it across the new artifact and the artifact underlying version 1 of the dataset:

dataset_v2 = ln.Dataset(
    [artifact, dataset_v1.artifact],
    is_new_version_of=dataset_v1,
)
dataset_v2.save()
dataset_v2.labels.add_from(artifact)
dataset_v2.labels.add_from(dataset_v1)

Version 2 of the dataset covers significantly more conditions.

dataset_v2.describe()

Dataset(uid='yIwUH2JUgdJ8sandeDLl', name='My versioned scRNA-seq dataset', version='2', hash='BOAf0T5UbN_iOe3fQDyq', visibility=1, updated_at=2023-12-22 11:26:01 UTC)

Provenance:
  💫 transform: Transform(uid='ManDYgmftZ8Cz8', name='Standardize and append a batch of data', short_name='scrna2', version='0', type=notebook, updated_at=2023-12-22 11:25:39 UTC, created_by_id=1)
  👣 run: Run(uid='RCjze5MCnLTqsjL2o5jC', run_at=2023-12-22 11:25:39 UTC, transform_id=2, created_by_id=1)
  🔖 initial_version: Dataset(uid='kJLMc6NiYPhmG31eM77C', name='My versioned scRNA-seq dataset', version='1', hash='9sXda5E7BYiVoDOQkTC0KB', visibility=1, updated_at=2023-12-22 11:25:33 UTC, transform_id=1, run_id=1, artifact_id=1, created_by_id=1)
  👤 created_by: User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at=2023-12-22 11:25:11 UTC)
Features:
  var: FeatureSet(uid='IJdCwYowHCLhFRDjL6Cm', n=36390, type='number', registry='bionty.Gene', hash='rMZltwoBCMdVPVR8x6nJ', updated_at=2023-12-22 11:25:30 UTC, created_by_id=1)
    'MIR1302-2HG', 'FAM138A', 'OR4F5', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'OR4F29', 'None', 'OR4F16', 'None', 'LINC01409', 'FAM87B', 'LINC01128', 'LINC00115', 'FAM41C', 'None', ...
  obs: FeatureSet(uid='c1iGKBV0EhgauNo9AXwU', n=4, registry='core.Feature', hash='g1m0hshChNIbAUcLGIWa', updated_at=2023-12-22 11:25:31 UTC, created_by_id=1)
    🔗 cell_type (40, bionty.CellType): 'dendritic cell', 'CD38-positive naive B cell', 'B cell, CD19-positive', 'effector memory CD4-positive, alpha-beta T cell, terminally differentiated', 'CD4-positive, alpha-beta T cell', 'cytotoxic T cell', 'CD8-positive, CD25-positive, alpha-beta regulatory T cell', 'CD14-positive, CD16-negative classical monocyte', 'classical monocyte', 'T follicular helper cell', ...
    🔗 assay (4, bionty.ExperimentalFactor): '10x 3' v3', '10x 5' v2', '10x 5' v1', 'single-cell RNA sequencing'
    🔗 tissue (17, bionty.Tissue): 'blood', 'thoracic lymph node', 'spleen', 'lung', 'mesenteric lymph node', 'lamina propria', 'liver', 'jejunal epithelium', 'omentum', 'bone marrow', ...
    🔗 donor (12, core.ULabel): 'D496', '621B', 'A29', 'A36', 'A35', '637C', 'A52', 'A37', 'D503', '640C', ...
  external: FeatureSet(uid='fGqbKDo1LQcCaRObgqgc', n=2, registry='core.Feature', hash='X2ktoQzWvquazFsSgCik', updated_at=2023-12-22 11:26:00 UTC, created_by_id=1)
    🔗 assay (4, bionty.ExperimentalFactor): '10x 3' v3', '10x 5' v2', '10x 5' v1', 'single-cell RNA sequencing'
    🔗 organism (1, bionty.Organism): 'human'
Labels:
  🏷️ organism (1, bionty.Organism): 'human'
  🏷️ tissues (17, bionty.Tissue): 'blood', 'thoracic lymph node', 'spleen', 'lung', 'mesenteric lymph node', 'lamina propria', 'liver', 'jejunal epithelium', 'omentum', 'bone marrow', ...
  🏷️ cell_types (40, bionty.CellType): 'dendritic cell', 'CD38-positive naive B cell', 'B cell, CD19-positive', 'effector memory CD4-positive, alpha-beta T cell, terminally differentiated', 'CD4-positive, alpha-beta T cell', 'cytotoxic T cell', 'CD8-positive, CD25-positive, alpha-beta regulatory T cell', 'CD14-positive, CD16-negative classical monocyte', 'classical monocyte', 'T follicular helper cell', ...
  🏷️ experimental_factors (4, bionty.ExperimentalFactor): '10x 3' v3', '10x 5' v2', '10x 5' v1', 'single-cell RNA sequencing'
  🏷️ ulabels (12, core.ULabel): 'D496', '621B', 'A29', 'A36', 'A35', '637C', 'A52', 'A37', 'D503', '640C', ...
  🏷️ artifacts (2, core.Artifact): 'scrna/conde22.h5ad', 'None'

View the flow:

dataset_v2.view_flow()

_images/d15f61d9557f5b7cb62cb4e8ac520e4debb9f0e82bf07fb1229984af1168fd18.svg