← All skills

cellxgene-census

data

Query the CZ CELLxGENE Census for versioned public single-cell and spatial transcriptomics data — cell metadata, expression slices, summary counts, source H5AD downloads, and embeddings across organisms, tissues, diseases, and cell types.

CZ CELLxGENE Census

Overview

The CZ CELLxGENE Census provides programmatic access to a comprehensive, versioned collection of standardized single-cell and spatial transcriptomics data from CZ CELLxGENE Discover. This skill enables efficient querying and analysis of public Census releases without downloading whole datasets first.

The 2025-11-08 stable LTS release declares, in census_info/summary:

  • 217,768,036 total cells and 125,463,259 unique cells
  • 1,845 datasets, whose dataset_total_cell_count values sum to exactly that total
  • Human, mouse, marmoset, rhesus macaque, and chimpanzee in census_data; human and mouse in census_spatial_sequencing
  • Standardized metadata (cell types, tissues, diseases, donors) and raw plus normalized expression layers
  • Source H5AD lookup/download helpers, pre-calculated summary counts, and hosted embeddings
  • Integration with AnnData, Scanpy, TileDB-SOMA, TileDB-SOMA-ML, and other analysis tools

Nothing here needs an account or a key.

The declared totals span two collections — every query reaches only one

This is the trap, and it is silent. A Census release is two top-level experiment collections:

collection 2025-11-08 cells contents
census_data 212,080,059 dissociated single-cell, five organisms
census_spatial_sequencing 5,687,977 Visium and Slide-seqV2, human and mouse
declared total_cell_count 217,768,036 both

census_info/summary and census_info/summary_cell_counts — the numbers you would naturally check an answer against — count both. Every query helper defaults to modality="census_data" and reaches only the first. There is no warning, no error, and no hint in the result.

The cost is not evenly spread, which is why one worked example never catches it:

filter (is_primary_data == True) census_data declared unique you get
mouse tissue_general == 'kidney' 398,589 2,399,190 17%
mouse tissue_general == 'colon' 3,509 43,445 8%
human tissue_general == 'inguinal part of abdomen' 0 14,976 0%
human tissue_general == 'kidney' 1,463,942 2,501,535 59%
human cell_type == 'B cell' 1,406,088 1,475,370 95%
human tissue_general == 'brain' 28,967,109 28,967,109 100%
human tissue_general == 'blood' 25,872,063 25,872,063 100%

Brain and blood are exact because no spatial dataset covers them. Twenty-three human and six mouse tissue_general labels are not exact, and one reads as if the tissue is absent entirely.

get_obs, get_var and get_anndata all take modality=; axis_query is reached through the collection you index. Query both and add, or say in writing which one you meant:

def n_cells(census, organism, filt, modality="census_data"):
    return len(cellxgene_census.get_obs(census, organism, value_filter=filt,
                                        column_names=["soma_joinid"], modality=modality))

F = "tissue_general == 'kidney' and is_primary_data == True"
n_cells(census, "mus_musculus", F)                                  # 398,589
n_cells(census, "mus_musculus", F, "census_spatial_sequencing")     # 2,000,601

adata = cellxgene_census.get_anndata(census=census, organism="Homo sapiens",
                                     obs_value_filter="dataset_id == '4cceac62-9513-42a4-90e5-2878dbb0192c'",
                                     modality="census_spatial_sequencing")

The two collections do not share a gene space — human census_data has 61,497 features and human census_spatial_sequencing has 43,386 — so concatenate on the intersection, not blindly.

When to Use This Skill

This skill should be used when:

  • Querying single-cell expression data by cell type, tissue, or disease
  • Exploring available single-cell datasets and metadata
  • Training machine learning models on single-cell data
  • Performing large-scale cross-dataset analyses
  • Integrating Census data with scanpy or other analysis frameworks
  • Computing statistics across millions of cells
  • Accessing pre-calculated embeddings or model predictions

Installation and Setup

Install the Census API:

uv pip install "cellxgene-census==1.17.*"

Reading data needs nothing else — no account, no key, no cloud credentials. 1.17.0 opens every release currently listed, from 2023-05-15 to 2025-11-08.

For spatial workflows. to_spatialdata() warns and then raises ModuleNotFoundError: No module named 'spatialdata' without this — the warning alone does not stop the call:

uv pip install "cellxgene-census[spatial]==1.17.*" "spatialdata[extra]>=0.2.5"

For PyTorch model training, use TileDB-SOMA-ML. The old cellxgene_census.experimental.ml loaders are deprecated:

uv pip install "cellxgene-census==1.17.*" tiledbsoma-ml

Core Workflow Patterns

Eight patterns, each with code, are in references/core_workflow_patterns.md:

  1. Opening the Census — always pin census_version so an analysis stays reproducible.
  2. Exploring Census information — available datasets, cell counts, and summary tables.
  3. Querying expression data — small to medium scale into an AnnData.
  4. Large-scale queries — out-of-core processing when the slice will not fit in memory.
  5. Machine learning with PyTorch — the Census data loaders.
  6. Spatial Census data — accessing spatial assays.
  7. Integration with Scanpy — handing a Census slice to a standard Scanpy workflow.
  8. Multi-dataset integration — combining datasets and handling batch effects.

Key Concepts and Best Practices

Always Filter for Primary Data

Unless analyzing duplicates, always include is_primary_data == True in queries to avoid counting cells multiple times:

obs_value_filter="cell_type == 'B cell' and is_primary_data == True"

is_primary_data == True matches summary_cell_counts.unique_cell_count; no filter at all matches total_cell_count. Both of those declared totals include the spatial collection, so the B cell filter above returns 1,406,088 from census_data against 1,475,370 declared — see The declared totals span two collections.

One dataset can be entirely non-primary: all 146 cells of 0895c838-e550-48a3-a777-dbcd35d30272 carry is_primary_data == False, so adding the filter to a single-dataset query returns an empty result with no error. Deduplicate across datasets; do not assume any one dataset survives it.

Specify Census Version for Reproducibility

Always specify the Census version in production analyses:

census = cellxgene_census.open_soma(census_version="2025-11-08")

soma_joinid Is Release-Local and Gets Reassigned

This is the version trap with teeth, because it fails silently. soma_joinid is a row offset in one build, not an identity. Between 2023-05-15 and 2025-11-08, 551 of the 553 datasets present in both releases have a different dataset soma_joinid — only 2 of 562 ids still point at the same dataset. Cell ids move too: the first cell of dataset 00476f9f-ebc1-4b72-b541-32f912ce36ea is soma_joinid 20,321,147 in the 2023 build and 1,971,589 in the 2025 one, with the same 10,099 cells in both.

So a saved soma_joinid list — from a size estimate, an obs_coords subsample, a cached index — reads different cells against a different release, with no error and no warning.

  • Persist dataset_id and observation_joinid, which are stable, not soma_joinid.
  • observation_joinid does not exist in pre-2025 releases, so a workflow that must span old and new builds has to key on dataset_id plus the source H5AD's own barcodes.
  • Re-derive soma_joinid inside the release you are querying, every time.

Estimate Query Size Before Loading

get_obs does not truncate and does not cap rows. Unfiltered human brain is 48,581,856 cells and it returns all 48,581,856 in seconds, in about 2.5 GB of RSS, with one column selected. So counting first is cheap and reliable, and it is the only thing standing between you and a get_anndata that tries to materialise millions of cells:

n_cells = len(cellxgene_census.get_obs(
    census, "homo_sapiens",
    value_filter="tissue_general == 'brain' and is_primary_data == True",
    column_names=["soma_joinid"],          # one column — do not read the whole obs frame
))
print(f"Query will return {n_cells:,} cells")

Above roughly 100k cells, move to axis_query, or subsample. A var_value_filter alone does not save you. get_anndata carries a multi-gigabyte fixed cost whatever the slice — a 3,364-cell, three-gene query peaked at 4.0 GB of RSS and took 319 s — and grows from there with the cells, not the genes: 224,666 cells on those same three genes passed 6 GB, and 437,482 cells on sixteen genes was past 9 GB and still going after nineteen minutes. Several filters that read as ordinary examples are far over that line in this release: tissue_general in ['lung','liver','kidney'] is 9,257,011 primary cells, cell_type == 'neuron' is 3,858,369, and cell_type == 'macrophage' and tissue_general in ['lung','liver','brain'] is 807,855.

To subsample, take the soma_joinids from get_obs and pass them back as obs_coords. Use the ids within the same open release and do not persist them — see the section above.

import numpy as np

ids = cellxgene_census.get_obs(census, "homo_sapiens", value_filter=F,
                               column_names=["soma_joinid"]).soma_joinid.to_numpy()
keep = np.sort(np.random.default_rng(0).choice(ids, size=50_000, replace=False))
adata = cellxgene_census.get_anndata(census=census, organism="Homo sapiens", obs_coords=keep)

An Unknown Value Returns Nothing; an Unknown Column Raises

The asymmetry matters because only one half is visible:

# Not a value in this release's `tissue` vocabulary -> empty DataFrame, no error, no warning
cellxgene_census.get_obs(census, "homo_sapiens",
                         value_filter="tissue == 'peripheral blood mononuclear cell'")   # 0 rows

# Not a column -> SOMAError: 'Column tissue_generall does not exist in schema'
cellxgene_census.get_obs(census, "homo_sapiens", value_filter="tissue_generall == 'brain'")

Take the vocabulary from the release rather than from memory. summary_cell_counts carries every assay, cell_type, disease, self_reported_ethnicity, sex, suspension_type, tissue and tissue_general label with its declared counts, so it is both the vocabulary and the ground truth — for seven of those eight.

suspension_type is the exception, and it fails in the opposite direction from the spatial bug this skill is about. The table lists a single label, cell, carrying the organism totals; nucleus and na are absent, and nucleus is another 37.7M primary human cells. Read the vocabulary for that one field off the obs enumeration instead, or you will conclude nucleus is not a legal value and over-count cell. Checked 2026-08-19 on 2025-11-08: summary_cell_counts gives ['cell'] where obs gives three values.

For the other seven:

counts = census["census_info"]["summary_cell_counts"].read().concat().to_pandas()
h = counts[counts.organism.eq("homo_sapiens")]
sorted(h[h.category.eq("tissue_general")].label)   # 71 labels; 'peripheral blood mononuclear cell' is not one

Use tissue_general for Broader Groupings

The tissue_general field provides coarser categories than tissue — 71 human labels against 423 — which is useful for cross-tissue analyses. The two vocabularies are not interchangeable, and putting a label in the wrong one returns zero cells rather than an error. thymus and cerebral cortex are tissue labels with no tissue_general counterpart, so tissue_general == 'thymus' is silently empty; cortex exists in both.

# Broader grouping
obs_value_filter="tissue_general == 'immune system'"

# Specific tissue, taken from the 423 labels this release actually carries
obs_value_filter="tissue == 'blood'"

disease Holds Multi-Labels, So == Is a Subset

A cell donated by someone with several recorded conditions carries all of them in one string, ||-delimited. Equality matches the single-condition label only:

# 608,235 primary cells
"disease == 'Alzheimer disease' and is_primary_data == True"

The Census declares 2,736,680 primary Alzheimer cells across the 18 labels that mention the disease — dementia || Alzheimer disease alone holds 1,486,303. Equality hands back 22%. There is no contains operator in the filter grammar, so enumerate the labels and use in:

d = counts[counts.organism.eq("homo_sapiens") & counts.category.eq("disease")]
labels = d.loc[d.label.str.contains("Alzheimer disease"), "label"].tolist()
cellxgene_census.get_obs(census, "homo_sapiens",
                         value_filter=f"disease in {labels} and is_primary_data == True",
                         column_names=["soma_joinid"])          # 2,736,680 — matches the declared sum

disease == 'COVID-19' happens to be exact in this release, because no multi-label mentions it. That is luck, not a rule — check before relying on it.

Select Only Needed Columns

Minimize data transfer by specifying only required metadata columns:

obs_column_names=["cell_type", "tissue_general", "disease"]  # Not all columns

The columns come back as pandas categoricals carrying the release's whole dictionary, not just the values present. A small slice's cell_type still carries all 898 human levels present in census_data, and a 200-cell two-tissue slice still has all 36 mouse tissue_general levels.

This is not merely untidy — it breaks downstream tools. sc.tl.rank_genes_groups(adata, groupby="tissue_general") on that 200-cell slice raises ValueError: Could not calculate statistics for groups forelimb, reproductive system, brain, … since they only contain one sample, naming 34 tissues the slice does not contain. Drop the unused levels first:

adata.obs["tissue_general"] = adata.obs.tissue_general.cat.remove_unused_categories()  # 36 -> 2
adata.obs.groupby("cell_type", observed=True).size()      # observed=True does the same for a groupby

Check Dataset Presence for Gene-Specific Queries

When analyzing specific genes, verify which datasets measured them. get_presence_matrix takes no gene filter — its signature is (census, organism, measurement_name="RNA", modality="census_data"), and passing var_value_filter= is a TypeError. It returns the whole n_datasets x n_genes boolean matrix (1,845 x 61,497 for human here), row-indexed by the dataset's soma_joinid and column-indexed by the gene's. Slice it yourself:

import numpy as np

var = cellxgene_census.get_var(census, "homo_sapiens", column_names=["soma_joinid", "feature_name"])
presence = cellxgene_census.get_presence_matrix(census, "homo_sapiens")   # (1845, 61497)

idx = var.loc[var.feature_name.isin(["CD4", "CD8A"]), "soma_joinid"].to_numpy()
both = np.asarray(presence[:, idx].sum(axis=1)).ravel() == len(idx)       # 1,067 of 1,845 datasets

datasets = census["census_info"]["datasets"].read().concat().to_pandas()
usable = datasets.loc[datasets.soma_joinid.isin(np.where(both)[0]), "dataset_id"].tolist()

Two-Step Workflow: Explore Then Query

First explore metadata to understand available data, then query expression:

# Step 1: Explore what's available
metadata = cellxgene_census.get_obs(
    census, "homo_sapiens",
    value_filter="disease == 'COVID-19' and is_primary_data == True",
    column_names=["cell_type", "tissue_general"]
)
print(metadata.value_counts())

# Step 2: Query based on findings
adata = cellxgene_census.get_anndata(
    census=census,
    organism="Homo sapiens",
    obs_value_filter="disease == 'COVID-19' and cell_type == 'T cell' and is_primary_data == True",
)

Available Metadata Fields

Cell Metadata (obs)

Key fields for filtering:

  • cell_type, cell_type_ontology_term_id
  • tissue, tissue_general, tissue_ontology_term_id
  • disease, disease_ontology_term_id
  • assay, assay_ontology_term_id
  • donor_id, sex, self_reported_ethnicity
  • development_stage, development_stage_ontology_term_id
  • dataset_id
  • is_primary_data (Boolean: True = unique cell)
  • tissue_type, suspension_type, observation_joinid
  • raw_sum, nnz, raw_mean_nnz, raw_variance_nnz, n_measured_vars (per-cell summaries)

Spatial observations carry three more: in_tissue, array_row, array_col.

There is no organism column. Organism is the collection key, not a filterable field — value_filter="organism == 'Homo sapiens'" raises SOMAError: 'Column organism does not exist in schema'. Human and mouse obs schemas are otherwise identical, so a filter written for one works unchanged on the other. Confirm the organisms a release carries with list(census["census_data"].keys()) and list(census["census_spatial_sequencing"].keys()).

Gene Metadata (var)

  • feature_id (Ensembl gene ID, e.g., "ENSG00000161798")
  • feature_name (Gene symbol, e.g., "FOXP2")
  • feature_type
  • feature_length (Gene length in base pairs)
  • nnz, n_measured_obs (availability summaries useful for checking sparsity and coverage)

Reference Documentation

This skill includes detailed reference documentation:

references/census_schema.md

Comprehensive documentation of:

  • Census data structure and organization
  • All available metadata fields
  • Value filter syntax and operators
  • SOMA object types
  • Data inclusion criteria

When to read: When you need detailed schema information, full list of metadata fields, or complex filter syntax.

references/common_patterns.md

Examples and patterns for:

  • Exploratory queries (metadata only)
  • Small-to-medium queries (AnnData)
  • Large queries (out-of-core processing)
  • PyTorch integration
  • Spatial Census access patterns
  • Scanpy integration workflows
  • Multi-dataset integration
  • Best practices and common pitfalls

When to read: When implementing specific query patterns, looking for code examples, or troubleshooting common issues.

Common Use Cases

Use Case 1: Explore Cell Types in a Tissue

Metadata only, so 6,167,731 cells is fine. Lung has 275,274 spatial cells too, so read both if the answer is meant to be the tissue rather than the dissociated half of it.

with cellxgene_census.open_soma(census_version="2025-11-08") as census:
    F = "tissue_general == 'lung' and is_primary_data == True"
    cells = cellxgene_census.get_obs(census, "homo_sapiens", value_filter=F,
                                     column_names=["cell_type"])
    spatial = cellxgene_census.get_obs(census, "homo_sapiens", value_filter=F,
                                       column_names=["cell_type"],
                                       modality="census_spatial_sequencing")
    print(cells["cell_type"].value_counts())
    print(f"{len(cells):,} dissociated + {len(spatial):,} spatial")

Use Case 2: Query Marker Gene Expression

cell_type in ['T cell', 'B cell'] and is_primary_data == True is 2,673,107 cells; adding tissue_general == 'lung' still leaves 224,666, and that slice passed 6 GB of RSS on three genes. Narrow the cells, not just the genes. The same filter over bone marrow is 3,364.

with cellxgene_census.open_soma(census_version="2025-11-08") as census:
    adata = cellxgene_census.get_anndata(
        census=census,
        organism="Homo sapiens",
        var_value_filter="feature_name in ['CD4', 'CD8A', 'CD19']",
        obs_value_filter="cell_type in ['T cell', 'B cell'] and tissue_general == 'bone marrow' "
                         "and is_primary_data == True",
        obs_column_names=["cell_type", "assay", "donor_id"],
    )                                                  # 3,364 x 3 — 2,020 B cells, 1,344 T cells

Even that took 319 s and 4.0 GB. get_anndata is never cheap; it is only ever cheaper. Size varies wildly by tissue for the same filter: spleen 42,196, kidney 47,245, lymph node 115,795, liver 168,816 — and tissue_general == 'thymus' is 0, because thymus is a tissue label and not a tissue_general one. Count before you load, every time.

Use Case 3: Train Cell Type Classifier

is_primary_data == True with nothing else is all 96,591,226 primary human cells in census_data — a real full-atlas run, not a demo. It streams, so it does not blow memory, but add a tissue or assay filter unless that is genuinely the training set you want. The loader reaches census_data only, so a classifier trained this way has seen no spatial cells.

import tiledbsoma as soma
from tiledbsoma_ml import ExperimentDataset, experiment_dataloader

with cellxgene_census.open_soma(census_version="2025-11-08") as census:
    experiment = census["census_data"]["homo_sapiens"]
    with experiment.axis_query(
        measurement_name="RNA",
        obs_query=soma.AxisQuery(value_filter="is_primary_data == True"),
    ) as query:
        dataset = ExperimentDataset(
            query=query,
            layer_name="raw",
            obs_column_names=["cell_type"],
            batch_size=128,
            shuffle=True,
        )
        dataloader = experiment_dataloader(dataset)

        for X, obs in dataloader:
            labels = obs["cell_type"]
            # Training logic
            pass

Use Case 4: Cross-Tissue Analysis

That obs filter is 807,855 primary cells — eight times the get_anndata guidance above — and with no var_value_filter it asks for 807,855 x 61,497. Cutting to lung and liver still leaves 437,482, and a 16-gene panel does not rescue it, because the cost is in the cells. Count, then subsample by soma_joinid, then load.

import numpy as np, scanpy as sc

PANEL = ["MRC1", "MARCO", "CD5L", "TIMD4", "VSIG4", "SIGLEC1", "SPP1", "APOE",
         "C1QA", "C1QB", "FABP4", "TREM2", "LYVE1", "FOLR2", "CD163", "MSR1"]
F = ("cell_type == 'macrophage' and tissue_general in ['lung', 'liver'] "
     "and is_primary_data == True")

with cellxgene_census.open_soma(census_version="2025-11-08") as census:
    ids = cellxgene_census.get_obs(census, "homo_sapiens", value_filter=F,
                                   column_names=["soma_joinid"]).soma_joinid.to_numpy()
    keep = np.sort(np.random.default_rng(0).choice(ids, size=50_000, replace=False))

    adata = cellxgene_census.get_anndata(
        census=census,
        organism="Homo sapiens",
        var_value_filter=f"feature_name in {PANEL}",
        obs_coords=keep,                                        # not obs_value_filter
        obs_column_names=["tissue_general", "assay", "donor_id"],
    )                                                           # 50,000 x 16

    # Without this, rank_genes_groups raises: the categorical still has all 70 tissues
    adata.obs["tissue_general"] = adata.obs.tissue_general.cat.remove_unused_categories()

    sc.pp.normalize_total(adata, target_sum=1e4); sc.pp.log1p(adata)
    sc.tl.rank_genes_groups(adata, groupby="tissue_general", method="wilcoxon")

obs_coords takes soma_joinid values, so the filter is applied once, in the cheap metadata pass, and the expensive pass reads only the rows you kept. assay and donor_id are in obs_column_names for a reason: the tissues are not balanced across assays or donors, so the top of that ranking is partly a batch effect. Check the crosstab before believing it.

Get the files

Two routes end in files on disk, and they answer different questions.

The source H5AD is the dataset as CELLxGENE Discover curated it, with the full gene space and every obs column — not the Census's harmonised subset. Check what lands against the dataset_total_cell_count the release declares rather than assuming they agree; that is what the assert below is for. Both datasets fetched here matched exactly.

import json, os, anndata as ad, cellxgene_census

VERSION, OUT = "2025-11-08", "Data/cellxgene-census"

def fetch_source_h5ads(dataset_ids, outdir=OUT, version=VERSION):
    """Source H5ADs on disk, each checked against the cell count the release declares."""
    os.makedirs(outdir, exist_ok=True)
    with cellxgene_census.open_soma(census_version=version) as census:
        frame = census["census_info"]["datasets"].read().concat().to_pandas().set_index("dataset_id")
    manifest = []
    for did in dataset_ids:
        if did not in frame.index:
            raise KeyError(f"{did} is not in Census {version}. get_source_h5ad_uri raises the same "
                           f"KeyError('Unknown dataset_id') for a withdrawn dataset and for a typo — "
                           f"check an older LTS release before assuming the id is wrong.")
        row, dest = frame.loc[did], os.path.join(outdir, f"{did}.h5ad")
        cellxgene_census.download_source_h5ad(did, to_path=dest, census_version=version,
                                              progress_bar=False)
        n_obs = ad.read_h5ad(dest, backed="r").n_obs
        assert n_obs == row.dataset_total_cell_count, \
            f"{did}: {n_obs} cells on disk, {row.dataset_total_cell_count} declared"
        manifest.append({"dataset_id": did, "title": row.dataset_title, "path": dest,
                         "cells": int(n_obs), "bytes": os.path.getsize(dest),
                         "citation": row.citation})
        print(f"  {n_obs:>7,} cells  {os.path.getsize(dest):>9,} B  {row.dataset_title[:46]}")
    with open(os.path.join(outdir, "manifest.json"), "w") as fh:
        json.dump({"census_version": version, "datasets": manifest}, fh, indent=2)
    return manifest

fetch_source_h5ads([
    "0895c838-e550-48a3-a777-dbcd35d30272",   # human liver B cells — every cell is_primary_data False
    "4eb29386-de81-452f-b3c0-e00844e8c7fd",   # Slide-seqV2 mouse — lives in census_spatial_sequencing
])

      146 cells  5,382,132 B  Healthy human liver: B cells
   10,888 cells  4,396,363 B  Spatial transcriptomics in mouse: Puck_191112_

The second of those is the reminder: a spatial dataset downloads from the same helper, but no census_data query will ever find its cells.

A cross-dataset slice is what the Census is for — a filter no single H5AD answers. Write the AnnData straight out, and pass modality= when the cells are spatial:

with cellxgene_census.open_soma(census_version=VERSION) as census:
    adata = cellxgene_census.get_anndata(
        census=census, organism="Mus musculus",
        obs_value_filter="tissue_general == 'optic cup' and is_primary_data == True")
    adata.write_h5ad(f"{OUT}/optic_cup.h5ad")                       # 146 x 53,384

    sp = cellxgene_census.get_anndata(
        census=census, organism="Homo sapiens",
        obs_value_filter="dataset_id == '4cceac62-9513-42a4-90e5-2878dbb0192c'",
        modality="census_spatial_sequencing")
    sp.write_h5ad(f"{OUT}/thymus_visium.h5ad")                      # 4,992 x 43,386

Every dataset carries a citation string in the datasets frame — publication DOI, dataset version URL and collection URL. Keep it with the file; it is the attribution the depositors are owed. Terms are per-collection on CELLxGENE Discover, so check the collection page before redistributing.

Try it

A self-contained check against the 2025-11-08 stable LTS, plus one read of the oldest LTS build to show that soma_joinid moves. Public data, no account, no key. Metadata only — row identifiers, one presence matrix, and three non-zero expression values. No large expression slices.

Data — Census releases 2025-11-08 (the current stable) and 2023-05-15 (the oldest LTS), both listed here:

https://census.cellxgene.cziscience.com/cellxgene-census/v1/release.json

Openly accessible; the directory flags both as LTS with do_not_delete. Individual datasets carry per-collection terms on CELLxGENE Discover. Last confirmed reachable 2026-08-18.

uv pip install "cellxgene-census==1.17.*"

import cellxgene_census, tiledbsoma as soma

VERSION = "2025-11-08"          # the current stable LTS

def n_cells(census, organism, filt, modality="census_data"):
    """Cells matching a filter. `modality` selects the collection — this is the trap."""
    return len(cellxgene_census.get_obs(census, organism, value_filter=filt,
                                        column_names=["soma_joinid"], modality=modality))

with cellxgene_census.open_soma(census_version=VERSION) as census:
    # 1. What the release declares about itself.
    summary  = census["census_info"]["summary"].read().concat().to_pandas().set_index("label")["value"]
    counts   = census["census_info"]["summary_cell_counts"].read().concat().to_pandas()
    datasets = census["census_info"]["datasets"].read().concat().to_pandas()
    declared = lambda org, cat, lab: int(counts[counts.organism.eq(org) & counts.category.eq(cat)
                                                & counts.label.eq(lab)].unique_cell_count.iloc[0])

    # 2. Spatial lives in its own collection; the declared totals count both.
    data_obs    = sum(census["census_data"][o].obs.count for o in census["census_data"].keys())
    spatial_obs = sum(census["census_spatial_sequencing"][o].obs.count
                      for o in census["census_spatial_sequencing"].keys())

    # 3. The counter-example: mouse kidney is mostly Slide-seqV2.
    F = "tissue_general == 'kidney' and is_primary_data == True"
    mk_data    = n_cells(census, "mus_musculus", F)
    mk_spatial = n_cells(census, "mus_musculus", F, "census_spatial_sequencing")
    mk_declared = declared("mus_musculus", "tissue_general", "kidney")

    # 4. A tissue that reads as absent from census_data entirely.
    G = "tissue_general == 'inguinal part of abdomen' and is_primary_data == True"
    ig_data    = n_cells(census, "homo_sapiens", G)
    ig_spatial = n_cells(census, "homo_sapiens", G, "census_spatial_sequencing")

    # 5. An unknown VALUE returns an empty frame; an unknown COLUMN raises.
    pbmc = n_cells(census, "homo_sapiens", "tissue == 'peripheral blood mononuclear cell'")
    try:
        n_cells(census, "homo_sapiens", "tissue_generall == 'brain'"); bad_column = "no error"
    except Exception as e:
        bad_column = type(e).__name__

    # 6. `disease` holds ' || '-delimited multi-labels, so `==` is a subset.
    alz_eq  = n_cells(census, "homo_sapiens", "disease == 'Alzheimer disease' and is_primary_data == True")
    d       = counts[counts.organism.eq("homo_sapiens") & counts.category.eq("disease")]
    alz_lab = d.loc[d.label.str.contains("Alzheimer disease"), "label"].tolist()
    alz_all = int(d.loc[d.label.str.contains("Alzheimer disease"), "unique_cell_count"].sum())
    alz_in  = n_cells(census, "homo_sapiens", f"disease in {alz_lab} and is_primary_data == True")

    # 7. X("raw").tables() yields stored non-zeros only — dividing by len() is not the mean.
    with census["census_data"]["mus_musculus"].axis_query(
        measurement_name="RNA",
        obs_query=soma.AxisQuery(value_filter="tissue_general == 'optic cup' and is_primary_data == True"),
        var_query=soma.AxisQuery(value_filter="feature_name in ['Rho', 'Sox2', 'Pax6']"),
    ) as q:
        nnz, total, cells, genes = 0, 0.0, q.n_obs, q.n_vars
        for batch in q.X("raw").tables():
            v = batch["soma_data"].to_numpy(); nnz += len(v); total += float(v.sum())
    mean_nonzero, mean_true = total / nnz, total / (cells * genes)

    # 8. get_presence_matrix takes no gene filter, and returns datasets x ALL genes.
    try:
        cellxgene_census.get_presence_matrix(census, "homo_sapiens", var_value_filter="feature_name == 'CD4'")
        presence_err = "accepted (unexpected)"
    except TypeError:
        presence_err = "TypeError"
    presence = cellxgene_census.get_presence_matrix(census, "homo_sapiens")

    # 9. A withdrawn dataset_id and a nonsense one fail identically.
    def uri_err(d):
        try:
            cellxgene_census.get_source_h5ad_uri(d, census_version=VERSION); return "found"
        except KeyError as e:
            return f"KeyError {e}"
    withdrawn = uri_err("0c774045-26a7-40f8-9b07-6742d3c771c0")   # in the 2023-05-15 LTS, gone from this one
    nonsense  = uri_err("00000000-0000-0000-0000-000000000000")

# 10. soma_joinid is a row offset in ONE build, not an identity.
with cellxgene_census.open_soma(census_version="2023-05-15") as old:
    old_ds = old["census_info"]["datasets"].read().concat().to_pandas().set_index("dataset_id").soma_joinid
new_ds = datasets.set_index("dataset_id").soma_joinid
shared = old_ds.index.intersection(new_ds.index)
moved  = int((old_ds[shared] != new_ds[shared]).sum())

assert data_obs + spatial_obs == int(summary["total_cell_count"]), "declared total covers both collections"
assert data_obs != int(summary["total_cell_count"]), "census_data alone must not reconcile"
assert mk_data + mk_spatial == mk_declared, "the two collections partition the declared count"
assert mk_data < mk_declared / 4, "census_data alone hands back under a quarter of mouse kidney"
assert ig_data == 0 and ig_spatial > 0, "a spatial-only tissue reads as empty from census_data"
assert pbmc == 0, "an unknown VALUE returns an empty frame with no error"
assert bad_column == "SOMAError", "an unknown COLUMN does raise"
assert alz_eq < alz_all / 4, "== on disease misses the ' || ' multi-label cells"
assert alz_in == alz_all, "enumerating the labels and using `in` recovers all of them"
assert mean_nonzero > 100 * mean_true, "dividing by len(values) is a mean over expressing cells"
assert presence_err == "TypeError", "get_presence_matrix has no var_value_filter"
assert presence.shape[0] == len(datasets), "presence rows are datasets, indexed by soma_joinid"
assert withdrawn == nonsense, "a withdrawn id is indistinguishable from a typo"
assert moved > 0.9 * len(shared), "soma_joinid is release-local — do not persist it"

print(f"release                            : {summary['census_build_date']} (schema {summary['census_schema_version']})")
print(f"declared total / unique cells      : {int(summary['total_cell_count']):,} / {int(summary['unique_cell_count']):,}")
print(f"census_data + spatial obs          : {data_obs:,} + {spatial_obs:,} = {data_obs + spatial_obs:,}")
print(f"datasets in release                : {len(datasets):,}")
print(f"mouse kidney, primary              : census_data {mk_data:,} + spatial {mk_spatial:,} = {mk_declared:,} declared")
print(f"human inguinal part of abdomen     : census_data {ig_data:,} + spatial {ig_spatial:,}")
print(f"tissue == 'peripheral blood mono…' : {pbmc} rows, no error | bad column -> {bad_column}")
print(f"disease == 'Alzheimer disease'     : {alz_eq:,}; in [{len(alz_lab)} labels] {alz_in:,} of {alz_all:,} declared")
print(f"optic cup Rho/Sox2/Pax6            : {nnz} non-zeros over {cells}x{genes}; mean {mean_nonzero:.3f} vs {mean_true:.5f}")
print(f"get_presence_matrix(var_value_filter): {presence_err}; unfiltered shape {presence.shape}")
print(f"withdrawn id / nonsense id         : {withdrawn} / {nonsense}")
print(f"dataset soma_joinid vs 2023-05-15  : {moved} of {len(shared)} shared datasets moved")

Expect

Invariants — these hold across releases, and a failure means the skill is wrong:

  • The declared total_cell_count never equals census_data alone. It is census_data plus census_spatial_sequencing, and no default-modality query can reach the second term.
  • Mouse kidney from census_data alone is under a quarter of the declared count, and the two collections sum to it exactly. The assertion is a ratio, not a fixed number, so it keeps holding as the Census grows. This is what stops anyone reintroducing "just query census_data", which hands a reader 398,589 of 2,399,190 cells with nothing to signal the loss.
  • A tissue can be entirely spatial — inguinal part of abdomen is 0 cells from census_data and 14,976 from the spatial collection.
  • An unknown value is silent and an unknown column raises. Only the second failure is visible.
  • disease == '…' is a subset, and disease in [every label mentioning it] recovers the declared total exactly.
  • X("raw").tables() yields stored non-zeros, so sum / len(values) is a mean over expressing cells and overstates by more than two orders of magnitude here.
  • get_presence_matrix rejects var_value_filter and returns one row per dataset.
  • A withdrawn dataset id and a nonsense one raise the identical KeyError.
  • soma_joinid is release-local. Nearly every dataset shared between 2023-05-15 and 2025-11-08 sits at a different id. A persisted id list silently selects different cells in a different release, which is why the skill says to store dataset_id instead.

Observed 2026-08-18 against Census 2025-11-08 — these move with each release:

release                            : 2025-11-08 (schema 2.4.0)
declared total / unique cells      : 217,768,036 / 125,463,259
census_data + spatial obs          : 212,080,059 + 5,687,977 = 217,768,036
datasets in release                : 1,845
mouse kidney, primary              : census_data 398,589 + spatial 2,000,601 = 2,399,190 declared
human inguinal part of abdomen     : census_data 0 + spatial 14,976
tissue == 'peripheral blood mono…' : 0 rows, no error | bad column -> SOMAError
disease == 'Alzheimer disease'     : 608,235; in [18 labels] 2,736,680 of 2,736,680 declared
optic cup Rho/Sox2/Pax6            : 2 non-zeros over 146x3; mean 1.500 vs 0.00685
get_presence_matrix(var_value_filter): TypeError; unfiltered shape (1845, 61497)
withdrawn id / nonsense id         : KeyError 'Unknown dataset_id' / KeyError 'Unknown dataset_id'
dataset soma_joinid vs 2023-05-15  : 551 of 553 shared datasets moved

Troubleshooting

The Count Does Not Match the Census Summary

  • The declared totals include census_spatial_sequencing; your query almost certainly did not. Re-run with modality="census_spatial_sequencing" and add.
  • is_primary_data == True corresponds to summary_cell_counts.unique_cell_count; no filter at all corresponds to total_cell_count. Mixing them up looks like a bug in the query.
  • A disease, cell_type or tissue value that appears in several ||-delimited labels is matched by == only in its solo label.
  • A misspelled value silently matches nothing. Compare against summary_cell_counts labels.

Query Returns Too Many Cells

  • Add more specific filters to reduce scope
  • Use tissue instead of tissue_general for finer granularity
  • Filter by specific dataset_id if known
  • Switch to out-of-core processing for large queries

Memory Errors

  • Reduce query scope with more restrictive filters
  • Select fewer genes with var_value_filter
  • Use out-of-core processing with axis_query()
  • Process data in batches

Duplicate Cells in Results

  • Always include is_primary_data == True in filters
  • Check if intentionally querying across multiple datasets

Gene Not Found

  • Verify gene name spelling (case-sensitive)
  • Try Ensembl ID with feature_id instead of feature_name
  • Check dataset presence matrix to see if gene was measured
  • Some genes may have been filtered during Census construction

Version Inconsistencies

  • Always specify census_version explicitly
  • Use same version across all analyses
  • Check release notes for version-specific changes
  • stable and latest are moving aliases. On 2026-08-18 stable was 2025-11-08 and latest was 2025-11-17; ten builds were listed, of which six carry the LTS flag. Weekly non-LTS builds are deleted, so an analysis pinned to one stops resolving.
  • Older releases have fewer collections and fewer organisms. 2023-05-15 is 562 datasets, two organisms and no census_spatial_sequencing; 2025-01-30 gains the spatial collection; 2025-11-08 gains marmoset, macaque and chimpanzee. Code that indexes census["census_spatial_sequencing"] raises KeyError on a pre-2025 release. One client version reads all of them — cellxgene-census 1.17.0 opened 2023-05-15, 2024-07-01, 2025-01-30 and 2025-11-08 without complaint.

A dataset_id Is Not Found

get_source_h5ad_uri raises KeyError('Unknown dataset_id') for a withdrawn dataset and for a typo alike — the message does not distinguish them. Nine datasets in the 2023-05-15 LTS are absent from 2025-11-08. Check the datasets frame of an older release before concluding the id is wrong. A dataset_id filter on obs is worse: it returns zero rows with no error either way.

Sources