Gene ontologies#

lamindb provides access to the following public gene ontologies through lnschema-bionty:

  1. Ensembl

  2. NCBI Gene

Here we show how to access and search gene ontologies to standardize new data.

Setup#

import lamindb_setup

lamindb_setup.init(storage="./test-gene", schema="bionty")
2023-12-22 11:23:10,799:INFO - HTTP Request: GET https://hub.lamin.ai/rest/v1/instance?select=%2A%2C%20account%21inner%21fk_instance_account_id_account%28%2A%29&account.handle=eq.testuser1&name=eq.test-gene "HTTP/1.1 200 OK"
2023-12-22 11:23:11,036:INFO - HTTP Request: GET https://hub.lamin.ai/rest/v1/account?select=%2A&handle=eq.testuser1 "HTTP/1.1 200 OK"
2023-12-22 11:23:11,138:INFO - HTTP Request: GET https://hub.lamin.ai/rest/v1/instance?select=%2A&account_id=eq.29cff183-c34d-445f-b6cf-31fb3b566158&name=eq.test-gene "HTTP/1.1 200 OK"
βœ… saved: User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at=2023-12-22 11:23:17 UTC)
βœ… saved: Storage(uid='81mfUU3k', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-gene', type='local', updated_at=2023-12-22 11:23:17 UTC, created_by_id=1)
πŸ’‘ loaded instance: testuser1/test-gene
πŸ’‘ did not register local instance on hub
# !lamin init --storage ./test-gene --schema bionty
import lnschema_bionty as lb
import pandas as pd

# adds an entry "human" into an empty instance
lb.settings.organism = "human"

Bionty objects#

Let us create a public knowledge accessor with bionty(), which chooses a default public knowledge source from BiontySource. It’s a Bionty object, which you can think about as a less-capable registry:

gene_bt = lb.Gene.bionty(organism="human")
gene_bt
Gene
Organism: human
Source: ensembl, release-110
#terms: 75719

πŸ“– Gene.df(): ontology reference table
πŸ”Ž Gene.lookup(): autocompletion of terms
🎯 Gene.search(): free text search of terms
βœ… Gene.validate(): strictly validate values
🧐 Gene.inspect(): full inspection of values
πŸ‘½ Gene.standardize(): convert to standardized names
πŸͺœ Gene.diff(): difference between two versions
πŸ”— Gene.ontology: Pronto.Ontology object

As for registries, you can export the ontology as a DataFrame:

df = gene_bt.df()
df.head()
ensembl_gene_id symbol ncbi_gene_id biotype description synonyms
0 ENSG00000000003 TSPAN6 7105 protein_coding tetraspanin 6 [Source:HGNC Symbol;Acc:HGNC:11858] T245|TSPAN-6|TM4SF6
1 ENSG00000000005 TNMD 64102 protein_coding tenomodulin [Source:HGNC Symbol;Acc:HGNC:17757] TEM|CHM1L|BRICD4|MYODULIN|TENDIN
2 ENSG00000000419 DPM1 8813 protein_coding dolichyl-phosphate mannosyltransferase subunit... MPDS|CDGIE
3 ENSG00000000457 SCYL3 57147 protein_coding SCY1 like pseudokinase 3 [Source:HGNC Symbol;A... PACE-1|PACE1
4 ENSG00000000460 C1orf112 55732 protein_coding chromosome 1 open reading frame 112 [Source:HG... APOLO1|FLIP|FLJ10706

Unlike registries, you can also export it as a Pronto object via gene_bt.ontology.

Look up terms#

As for registries, terms can be looked up with auto-complete:

lookup = gene_bt.lookup()

The . accessor provides normalized terms (lower case, only contains alphanumeric characters and underscores):

lookup.tcf7
Gene(ensembl_gene_id='ENSG00000081059', symbol='TCF7', ncbi_gene_id='6932', biotype='protein_coding', description='transcription factor 7 [Source:HGNC Symbol;Acc:HGNC:11639]', synonyms='TCF-1')

To look up the exact original strings, convert the lookup object to dict and use the [] accessor:

lookup_dict = lookup.dict()
lookup_dict["TCF7"]
Gene(ensembl_gene_id='ENSG00000081059', symbol='TCF7', ncbi_gene_id='6932', biotype='protein_coding', description='transcription factor 7 [Source:HGNC Symbol;Acc:HGNC:11639]', synonyms='TCF-1')

By default, the name field is used to generate lookup keys. You can specify another field to look up:

lookup = gene_bt.lookup(gene_bt.ncbi_gene_id)

If multiple entries are matched, they are returned as a list:

lookup.bt_100126572
Gene(ensembl_gene_id='ENSG00000203733', symbol='GJE1', ncbi_gene_id='100126572', biotype='protein_coding', description='gap junction protein epsilon 1 [Source:HGNC Symbol;Acc:HGNC:33251]', synonyms='CX23')

Search terms#

Search behaves in the same way as it does for registries:

gene_bt = lb.Gene.bionty()
gene_bt.search("TP53").head(3)
ensembl_gene_id ncbi_gene_id biotype description synonyms __ratio__
symbol
TP53 ENSG00000141510 7157 protein_coding tumor protein p53 [Source:HGNC Symbol;Acc:HGNC... LFS1|P53 100.0
TP53TG3GP ENSG00000261274 None unprocessed_pseudogene TP53 target 3 family member G, pseudogene [Sou... 90.0
TP53TG1 ENSG00000182165 None lncRNA TP53 target 1 [Source:HGNC Symbol;Acc:HGNC:17026] TP53AP1|H_RG012D21.9|LINC00096|TP53LC12 90.0

By default, search also covers synonyms:

gene_bt.search("PDL1").head(3)
ensembl_gene_id ncbi_gene_id biotype description synonyms __ratio__
symbol
CD274 ENSG00000120217 29126 protein_coding CD274 molecule [Source:HGNC Symbol;Acc:HGNC:17... B7-H1|B7H1|PDL1|PD-L1|PDCD1LG1|B7-H 100.0
GAPDHP73 ENSG00000226540 None processed_pseudogene glyceraldehyde-3-phosphate dehydrogenase pseud... GAPDL19|GAPDHL19 90.0
IGKV1-16 ENSG00000282282 None IG_V_gene immunoglobulin kappa variable 1-16 [Source:HGN... IGKV116|L1 90.0

You can turn this off synonym by passing synonyms_field=None:

gene_bt.search("PDL1", synonyms_field=None).head(3)
ensembl_gene_id ncbi_gene_id biotype description synonyms __ratio__
symbol
SPDL1 ENSG00000040275 54908 protein_coding spindle apparatus coiled-coil protein 1 [Sourc... HSPINDLY|FLJ20364|CCDC99 88.888889
PKD2L1 ENSG00000107593 9033 protein_coding polycystin 2 like 1, transient receptor potent... TRPP3|PCL|PKDL|PKD2L 80.000000
PKD1L1 ENSG00000158683 168507 protein_coding polycystin 1 like 1, transient receptor potent... PRO19563 80.000000

Search another field (default is .name):

gene_bt.search("tumor protein p53", field=gene_bt.description).head()
ensembl_gene_id symbol ncbi_gene_id biotype synonyms __ratio__
description
tyrosinase related protein 1 [Source:HGNC Symbol;Acc:HGNC:12450] ENSG00000107165 TYRP1 7306 protein_coding B-PROTEIN|CAS2|CATB|TYRP|OCA3|TRP|GP75 61.538462
novel protein ENSG00000258674 None None protein_coding 60.000000
novel protein ENSG00000289012 None None lncRNA 60.000000
novel protein ENSG00000289653 None None protein_coding 60.000000
novel protein ENSG00000289565 None None protein_coding 60.000000

Standardize gene identifiers#

Let us generate a DataFrame that stores a number of gene identifiers, some of which corrupted:

data = {
    "gene symbol": ["A1CF", "A1BG", "FANCD1", "corrupted"],
    "ncbi id": ["29974", "1", "5133", "corrupted"],
    "ensembl_gene_id": [
        "ENSG00000148584",
        "ENSG00000121410",
        "ENSG00000188389",
        "ENSGcorrupted",
    ],
}
df_orig = pd.DataFrame(data).set_index("ensembl_gene_id")
df_orig
gene symbol ncbi id
ensembl_gene_id
ENSG00000148584 A1CF 29974
ENSG00000121410 A1BG 1
ENSG00000188389 FANCD1 5133
ENSGcorrupted corrupted corrupted

First we can check whether any of our values are validated against the ontology reference:

validated = gene_bt.validate(df_orig.index, gene_bt.ensembl_gene_id)
df_orig.index[~validated]
❗ 1 term (25.00%) is not validated: ENSGcorrupted
Index(['ENSGcorrupted'], dtype='object', name='ensembl_gene_id')

Next, we validate which symbols are mappable against the ontology:

# based on NCBI gene ID
gene_bt.validate(df_orig["ncbi id"], gene_bt.ncbi_gene_id)
❗ 1 term (25.00%) is not validated: corrupted
array([ True,  True,  True, False])
# based on Gene symbols
validated_symbols = gene_bt.validate(df_orig["gene symbol"], gene_bt.symbol)
df_orig["gene symbol"][~validated_symbols]
❗ 2 terms (50.00%) are not validated: FANCD1, corrupted
ensembl_gene_id
ENSG00000188389       FANCD1
ENSGcorrupted      corrupted
Name: gene symbol, dtype: object

Here, 2 of the gene symbols are not validated. Inspect why:

gene_bt.inspect(df_orig["gene symbol"], gene_bt.symbol);
❗ 2 terms (50.00%) are not validated for symbol: FANCD1, corrupted
   detected 1 terms with synonym: FANCD1
β†’  standardize terms via .standardize()

Logging suggests to use .standardize():

mapped_symbol_synonyms = gene_bt.standardize(df_orig["gene symbol"])
mapped_symbol_synonyms
['A1CF', 'A1BG', 'BRCA2', 'corrupted']

Optionally, you can return a mapper in the form of {synonym1: standardized_name1, ...}:

gene_bt.standardize(df_orig["gene symbol"], return_mapper=True)
{'FANCD1': 'BRCA2'}

We can use the standardized symbols as the new standardized index:

df_curated = df_orig.reset_index()
df_curated.index = mapped_symbol_synonyms
df_curated
ensembl_gene_id gene symbol ncbi id
A1CF ENSG00000148584 A1CF 29974
A1BG ENSG00000121410 A1BG 1
BRCA2 ENSG00000188389 FANCD1 5133
corrupted ENSGcorrupted corrupted corrupted

You can convert identifiers by passing return_field to standardize():

gene_bt.standardize(
    df_curated.index,
    field=gene_bt.symbol,
    return_field=gene_bt.ensembl_gene_id,
)
['ENSG00000148584', 'ENSG00000121410', 'ENSG00000139618', 'corrupted']

And return mappable identifiers as a dict:

gene_bt.standardize(
    df_curated.index,
    field=gene_bt.symbol,
    return_field=gene_bt.ensembl_gene_id,
    return_mapper=True,
)
{'A1BG': 'ENSG00000121410',
 'BRCA2': 'ENSG00000139618',
 'A1CF': 'ENSG00000148584'}

Ontology source versions#

For any given entity, we can choose from a number of versions:

lb.BiontySource.filter(entity="Gene").df()
uid entity organism currently_used source source_name version url md5 source_website updated_at created_by_id
id
9 o36k Gene human True ensembl Ensembl release-110 s3://bionty-assets/df_human__ensembl__release-... 832f3947e83664588d419608a469b528 https://www.ensembl.org 2023-12-22 11:23:17.151087+00:00 1
10 mIAU Gene human False ensembl Ensembl release-109 s3://bionty-assets/human_ensembl_release-109_G... 72da9968c74e96d136a489a6102a4546 https://www.ensembl.org 2023-12-22 11:23:17.151120+00:00 1
11 VTEw Gene mouse True ensembl Ensembl release-110 s3://bionty-assets/df_mouse__ensembl__release-... fa4ce130f2929aefd7ac3bc8eaf0c4de https://www.ensembl.org 2023-12-22 11:23:17.151154+00:00 1
12 bN62 Gene mouse False ensembl Ensembl release-109 s3://bionty-assets/mouse_ensembl_release-109_G... 08a1165061151b270b985317322bd2ed https://www.ensembl.org 2023-12-22 11:23:17.151187+00:00 1
13 Uhnp Gene saccharomyces cerevisiae True ensembl Ensembl release-110 s3://bionty-assets/df_saccharomyces cerevisiae... 2e59495a3e87ea6575e408697dd73459 https://www.ensembl.org 2023-12-22 11:23:17.151219+00:00 1

When instantiating a Bionty object, we can choose a source or version:

bionty_source = lb.BiontySource.filter(
    source="ensembl", version="release-110", organism="human"
).one()
gene_bt = lb.Gene.bionty(bionty_source=bionty_source)
gene_bt
Gene
Organism: human
Source: ensembl, release-110
#terms: 75719

πŸ“– Gene.df(): ontology reference table
πŸ”Ž Gene.lookup(): autocompletion of terms
🎯 Gene.search(): free text search of terms
βœ… Gene.validate(): strictly validate values
🧐 Gene.inspect(): full inspection of values
πŸ‘½ Gene.standardize(): convert to standardized names
πŸͺœ Gene.diff(): difference between two versions
πŸ”— Gene.ontology: Pronto.Ontology object

The currently used ontologies can be displayed using:

lb.BiontySource.filter(currently_used=True).df()
uid entity organism currently_used source source_name version url md5 source_website updated_at created_by_id
id
1 zvGR Organism vertebrates True ensembl Ensembl release-110 https://ftp.ensembl.org/pub/release-110/specie... f3faf95648d3a2b50fd3625456739706 https://www.ensembl.org 2023-12-22 11:23:17.150789+00:00 1
4 TE9h Organism bacteria True ensembl Ensembl release-57 https://ftp.ensemblgenomes.ebi.ac.uk/pub/bacte... ee28510ed5586ea7ab4495717c96efc8 https://www.ensembl.org 2023-12-22 11:23:17.150917+00:00 1
5 OZIG Organism fungi True ensembl Ensembl release-57 http://ftp.ensemblgenomes.org/pub/fungi/releas... dbcde58f4396ab8b2480f7fe9f83df8a https://www.ensembl.org 2023-12-22 11:23:17.150951+00:00 1
6 W07m Organism metazoa True ensembl Ensembl release-57 http://ftp.ensemblgenomes.org/pub/metazoa/rele... 424636a574fec078a61cbdddb05f9132 https://www.ensembl.org 2023-12-22 11:23:17.150985+00:00 1
7 AVh3 Organism plants True ensembl Ensembl release-57 https://ftp.ensemblgenomes.ebi.ac.uk/pub/plant... eadaa1f3e527e4c3940c90c7fa5c8bf4 https://www.ensembl.org 2023-12-22 11:23:17.151019+00:00 1
8 MdBu Organism all True ncbitaxon NCBItaxon Ontology 2023-06-20 s3://bionty-assets/df_all__ncbitaxon__2023-06-... 00d97ba65627f1cd65636d2df22ea76c https://github.com/obophenotype/ncbitaxon 2023-12-22 11:23:17.151052+00:00 1
9 o36k Gene human True ensembl Ensembl release-110 s3://bionty-assets/df_human__ensembl__release-... 832f3947e83664588d419608a469b528 https://www.ensembl.org 2023-12-22 11:23:17.151087+00:00 1
11 VTEw Gene mouse True ensembl Ensembl release-110 s3://bionty-assets/df_mouse__ensembl__release-... fa4ce130f2929aefd7ac3bc8eaf0c4de https://www.ensembl.org 2023-12-22 11:23:17.151154+00:00 1
13 Uhnp Gene saccharomyces cerevisiae True ensembl Ensembl release-110 s3://bionty-assets/df_saccharomyces cerevisiae... 2e59495a3e87ea6575e408697dd73459 https://www.ensembl.org 2023-12-22 11:23:17.151219+00:00 1
14 000Q Protein human True uniprot Uniprot 2023-03 s3://bionty-assets/df_human__uniprot__2023-03_... 1c46e85c6faf5eff3de5b4e1e4edc4d3 https://www.uniprot.org 2023-12-22 11:23:17.151253+00:00 1
16 tD7O Protein mouse True uniprot Uniprot 2023-03 s3://bionty-assets/df_mouse__uniprot__2023-03_... 9d5e9a8225011d3218e10f9bbb96a46c https://www.uniprot.org 2023-12-22 11:23:17.151319+00:00 1
18 vqWI CellMarker human True cellmarker CellMarker 2.0 s3://bionty-assets/human_cellmarker_2.0_CellMa... d565d4a542a5c7e7a06255975358e4f4 http://bio-bigdata.hrbmu.edu.cn/CellMarker 2023-12-22 11:23:17.151386+00:00 1
19 ypPK CellMarker mouse True cellmarker CellMarker 2.0 s3://bionty-assets/mouse_cellmarker_2.0_CellMa... 189586732c63be949e40dfa6a3636105 http://bio-bigdata.hrbmu.edu.cn/CellMarker 2023-12-22 11:23:17.151419+00:00 1
20 2Zjk CellLine all True clo Cell Line Ontology 2022-03-21 https://data.bioontology.org/ontologies/CLO/su... ea58a1010b7e745702a8397a526b3a33 https://bioportal.bioontology.org/ontologies/CLO 2023-12-22 11:23:17.151452+00:00 1
21 4shh CellType all True cl Cell Ontology 2023-08-24 http://purl.obolibrary.org/obo/cl/releases/202... 46e7dd89421f1255cf0191eca1548f73 https://obophenotype.github.io/cell-ontology 2023-12-22 11:23:17.151485+00:00 1
25 LmWQ Tissue all True uberon Uberon multi-species anatomy ontology 2023-09-05 http://purl.obolibrary.org/obo/uberon/releases... abcee3ede566d1311d758b853ccdf5aa http://obophenotype.github.io/uberon 2023-12-22 11:23:17.151616+00:00 1
29 zMWv Disease all True mondo Mondo Disease Ontology 2023-08-02 http://purl.obolibrary.org/obo/mondo/releases/... 7f33767422042eec29f08b501fc851db https://mondo.monarchinitiative.org 2023-12-22 11:23:17.151748+00:00 1
33 cxPr Disease human True doid Human Disease Ontology 2023-03-31 http://purl.obolibrary.org/obo/doid/releases/2... 64f083a1e47867c307c8eae308afc3bb https://disease-ontology.org 2023-12-22 11:23:17.151880+00:00 1
35 2wto ExperimentalFactor all True efo The Experimental Factor Ontology 3.57.0 http://www.ebi.ac.uk/efo/releases/v3.57.0/efo.owl 2ecafc69b3aba7bdb31ad99438505c05 https://bioportal.bioontology.org/ontologies/EFO 2023-12-22 11:23:17.151946+00:00 1
37 3SSF Phenotype human True hp Human Phenotype Ontology 2023-06-17 https://github.com/obophenotype/human-phenotyp... 65e8d96bc81deb893163927063b10c06 https://hpo.jax.org 2023-12-22 11:23:17.152012+00:00 1
40 nwdt Phenotype mammalian True mp Mammalian Phenotype Ontology 2023-05-31 https://github.com/mgijax/mammalian-phenotype-... be89052cf6d9c0b6197038fe347ef293 https://github.com/mgijax/mammalian-phenotype-... 2023-12-22 11:23:17.152110+00:00 1
41 zAfB Phenotype zebrafish True zp Zebrafish Phenotype Ontology 2022-12-17 https://github.com/obophenotype/zebrafish-phen... 03430b567bf153216c0fa4c3440b3b24 https://github.com/obophenotype/zebrafish-phen... 2023-12-22 11:23:17.152143+00:00 1
43 p1co Phenotype all True pato Phenotype And Trait Ontology 2023-05-18 http://purl.obolibrary.org/obo/pato/releases/2... bd472f4971492109493d4ad8a779a8dd https://github.com/pato-ontology/pato 2023-12-22 11:23:17.152209+00:00 1
44 h0rU Pathway all True go Gene Ontology 2023-05-10 https://data.bioontology.org/ontologies/GO/sub... e9845499eadaef2418f464cd7e9ac92e http://geneontology.org 2023-12-22 11:23:17.152242+00:00 1
46 fxHJ BFXPipeline all True lamin Bioinformatics Pipeline 1.0.0 s3://bionty-assets/bfxpipelines.json a7eff57a256994692fba46e0199ffc94 https://lamin.ai 2023-12-22 11:23:17.152309+00:00 1
47 chfO Drug all True dron Drug Ontology 2023-03-10 https://data.bioontology.org/ontologies/DRON/s... 75e86011158fae76bb46d96662a33ba3 https://bioportal.bioontology.org/ontologies/DRON 2023-12-22 11:23:17.152342+00:00 1
48 7JhT DevelopmentalStage human True hsapdv Human Developmental Stages 2020-03-10 http://aber-owl.net/media/ontologies/HSAPDV/11... 52181d59df84578ed69214a5cb614036 https://github.com/obophenotype/developmental-... 2023-12-22 11:23:17.152375+00:00 1
49 JIKv DevelopmentalStage mouse True mmusdv Mouse Developmental Stages 2020-03-10 http://aber-owl.net/media/ontologies/MMUSDV/9/... 5bef72395d853c7f65450e6c2a1fc653 https://github.com/obophenotype/developmental-... 2023-12-22 11:23:17.152408+00:00 1
50 clid Ethnicity human True hancestro Human Ancestry Ontology 3.0 https://github.com/EBISPOT/hancestro/raw/3.0/h... 76dd9efda9c2abd4bc32fc57c0b755dd https://github.com/EBISPOT/hancestro 2023-12-22 11:23:17.152441+00:00 1
51 rsbG BioSample all True ncbi NCBI BioSample attributes 2023-09 s3://bionty-assets/df_all__ncbi__2023-09__BioS... 918db9bd1734b97c596c67d9654a4126 https://www.ncbi.nlm.nih.gov/biosample/docs/at... 2023-12-22 11:23:17.152474+00:00 1
Hide code cell content
!lamin delete --force test-gene
!rm -r test-gene
πŸ’‘ deleting instance testuser1/test-gene
βœ…     deleted instance settings file: /home/runner/.lamin/instance--testuser1--test-gene.env
βœ…     instance cache deleted
βœ…     deleted '.lndb' sqlite file
❗     consider manually deleting your stored data: /home/runner/work/lamin-usecases/lamin-usecases/docs/test-gene