Karenina — grading answers against templates and rubrics

Karenina turns "was that answer any good?" into two questions it can answer separately:

• Did it get the facts right? An answer template is a Pydantic model whose fields carry their own ground truth and their own comparison rule.
• Was it well-made? A rubric is a set of traits scored over the response text — cited, hedged, formatted, on-topic.

The two are independent on purpose. A response can name the right gene and argue for it badly, or reason beautifully to the wrong gene, and you want to see which one happened.

The useful property for automation is that most of this is deterministic. Regex and callable traits, and every template comparison below, are ordinary Python — no model call, no key, same answer every time. Only the LLM-judge traits need a provider.

What you need

• Python 3.11 or newer.
• Install from the repository. There is no PyPI release as of 2026-09-12, so pip install karenina will not find it.
• A model-provider API key — only for LLM-judge traits. Everything else in this page runs offline. The one section that needs a key says so before its code block.

Expect a large dependency tree (it pulls a full agent/LLM stack), so allow a few minutes.

pip install "git+https://github.com/biocypher/karenina.git"
python -c "import karenina; print(karenina.__version__)"

Answer templates — ground truth on the field

Subclass BaseAnswer and declare each field with VerifiedField, giving it the correct value and the rule for comparing against it. Karenina generates verify() and verify_granular() for you.

from karenina.schemas import BaseAnswer
from karenina.schemas.entities.verified_field import VerifiedField
from karenina.schemas.primitives import ExactMatch, NumericExact

class CFAnswer(BaseAnswer):
    gene: str = VerifiedField(
        description="HGNC symbol of the causal gene",
        ground_truth="CFTR",
        verify_with=ExactMatch(),
    )
    chromosome: int = VerifiedField(
        description="Chromosome carrying the gene",
        ground_truth=7,
        verify_with=NumericExact(),
    )

print(CFAnswer(gene="cftr", chromosome=7).verify())     # True
print(CFAnswer(gene="SCNN1A", chromosome=12).verify())  # False

VerifiedField is not re-exported from karenina.schemas; import it from karenina.schemas.entities.verified_field as above.

The comparison rules live in karenina.schemas.primitives and cover the usual shapes — ExactMatch, NumericExact, NumericTolerance, NumericRange, ContainsAll, ContainsAny, RegexMatch, SetContainment, DateMatch, and composition helpers (AllOf, AnyOf, AtLeastN).

Two scores, not one

verify() is all-or-nothing. verify_granular() returns the weighted fraction of fields that passed, as a float — not, despite the name, a per-field breakdown.

from karenina.schemas import BaseAnswer
from karenina.schemas.entities.verified_field import VerifiedField
from karenina.schemas.primitives import ExactMatch, NumericExact

class Weighted(BaseAnswer):
    gene: str = VerifiedField(description="gene", ground_truth="CFTR",
                              verify_with=ExactMatch(), weight=3.0)
    chromosome: int = VerifiedField(description="chr", ground_truth=7,
                                    verify_with=NumericExact(), weight=1.0)

right_gene_wrong_chr = Weighted(gene="CFTR", chromosome=99)
print(right_gene_wrong_chr.verify())           # False
print(right_gene_wrong_chr.verify_granular())  # 0.75  -> 3 of 4 weight units

Weights are what stop a throwaway field from costing as much as the answer itself. Leave weight off and every field counts 1.

Two traps that pass wrong answers quietly

Both of these produce a template that looks strict and is not. They are worth knowing before you trust a score.

NumericTolerance is relative by default. tolerance=0.5 means ±50%, not ±0.5. Against a ground truth of 7.2 it accepts anything up to 10.8.

from karenina.schemas import BaseAnswer
from karenina.schemas.entities.verified_field import VerifiedField
from karenina.schemas.primitives import NumericTolerance

class Relative(BaseAnswer):
    v: float = VerifiedField(description="HbA1c", ground_truth=7.2,
                             verify_with=NumericTolerance(tolerance=0.5))

class Absolute(BaseAnswer):
    v: float = VerifiedField(description="HbA1c", ground_truth=7.2,
                             verify_with=NumericTolerance(tolerance=0.5, mode="absolute"))

for v in (7.5, 8.9, 10.7, 10.9):
    print(v, Relative(v=v).verify(), Absolute(v=v).verify())
# 7.5  True  True
# 8.9  True  False   <- 24% off, still "within tolerance"
# 10.7 True  False
# 10.9 False False

Pass mode="absolute" whenever you mean units.

Unknown keyword arguments to a primitive are dropped silently. ExactMatch has no case_sensitive parameter — it has normalize, which defaults to lowercasing and stripping. So ExactMatch(case_sensitive=True) is accepted, changes nothing, and leaves you case-insensitive.

from karenina.schemas import BaseAnswer
from karenina.schemas.entities.verified_field import VerifiedField
from karenina.schemas.primitives import ExactMatch

print(ExactMatch(case_sensitive=True).model_dump())
# {'normalize': ['lowercase', 'strip']}  -- the argument vanished

class Loose(BaseAnswer):
    g: str = VerifiedField(description="gene", ground_truth="CFTR",
                           verify_with=ExactMatch())

class Strict(BaseAnswer):
    g: str = VerifiedField(description="gene", ground_truth="CFTR",
                           verify_with=ExactMatch(normalize=[]))

print(Loose(g="cftr").verify(), Strict(g="cftr").verify())  # True False

Check Primitive.model_dump() when a comparison behaves unexpectedly — it shows what the primitive actually holds. normalize=[] is how you get a case-sensitive match.

Rubric traits over the response text

A Rubric holds five lists of traits. Two of them run with no model at all.

RegexRubricTrait.evaluate(text) returns a bool. invert_result=True turns a pattern into a prohibition, which is how you check that something is absent.

from karenina.schemas import Rubric, RegexRubricTrait

rubric = Rubric(regex_traits=[
    RegexRubricTrait(name="cites_pmid", pattern=r"PMID:\s*\d{6,8}",
                     description="cites at least one PMID"),
    RegexRubricTrait(name="no_hedging", pattern=r"\b(?:might|maybe|possibly)\b",
                     case_sensitive=False, invert_result=True,
                     description="states a conclusion without hedging"),
])

for text in ["The causal gene is CFTR (PMID: 2475911).", "It might possibly be CFTR."]:
    print({t.name: t.evaluate(text) for t in rubric.regex_traits})
# {'cites_pmid': True, 'no_hedging': True}
# {'cites_pmid': False, 'no_hedging': False}

Callable traits take an arbitrary function, which is where domain checks belong — a symbol that must appear in a reference set, a dose that must be physiological. Build them with from_callable and give kind="boolean" or kind="score".

import re
from karenina.schemas import CallableRubricTrait

KNOWN = {"CFTR", "TP53", "KRAS", "BRCA1"}

def only_known_genes(text: str) -> bool:
    found = set(re.findall(r"\b[A-Z][A-Z0-9]{2,}\b", text))
    return bool(found) and found <= KNOWN

def n_citations(text: str) -> int:
    return len(re.findall(r"PMID:\s*\d+", text))

t1 = CallableRubricTrait.from_callable(name="only_known_genes", func=only_known_genes,
                                       kind="boolean", description="no invented symbols")
t2 = CallableRubricTrait.from_callable(name="n_citations", func=n_citations,
                                       kind="score", min_score=0, max_score=5)

print(t1.deserialize_callable()("CFTR and TP53 are implicated."))   # True
print(t1.deserialize_callable()("FAKEGENE1 is implicated."))        # False
print(t2.deserialize_callable()("PMID: 123456 and PMID: 654321"))   # 2

A callable trait stores pickled code, and deserialize_callable() executes it. The library warns on every call. A rubric is therefore as dangerous as a script — load one only from somewhere you trust, and do not accept rubric files from users the way you would accept a CSV.

LLM-judge traits

LLMRubricTrait is the one that needs a provider. It is for judgements no regex reaches — whether the reasoning follows, whether a caveat was warranted. Traits are declared the same way; scoring them requires a configured model and will spend API credits.

Set your provider key in the environment first (OPENAI_API_KEY, ANTHROPIC_API_KEY or GOOGLE_API_KEY, depending on provider). Declaring the traits, as below, needs no key and costs nothing — it is scoring them against responses that calls the provider, and that step was deliberately not exercised when this page was verified.

from karenina.schemas import Rubric, LLMRubricTrait

rubric = Rubric(llm_traits=[
    LLMRubricTrait(
        name="mechanism_explained",
        kind="boolean",
        description="States the mechanism linking the gene to the phenotype, not just the name.",
    ),
    LLMRubricTrait(
        name="calibration",
        kind="score",
        min_score=1,
        max_score=5,
        description="Confidence is proportionate to the evidence actually cited.",
    ),
])
print(rubric.get_llm_trait_names())

Prefer a deterministic trait wherever one will do. A regex costs nothing, never drifts between model versions, and cannot disagree with itself on a re-run — and an eval whose own grader is stochastic is an eval you have to re-run to trust.

Try it

Fully offline — no key, no network, no data to fetch. Saves the file, runs it, and checks every assertion including the relative-tolerance trap.

Data — generated inline; there is no external dataset to obtain.

Run

pip install "git+https://github.com/biocypher/karenina.git"

cat > karenina_tryit.py <<'PY'
from karenina.schemas import BaseAnswer, Rubric, RegexRubricTrait
from karenina.schemas.entities.verified_field import VerifiedField
from karenina.schemas.primitives import ExactMatch, NumericExact, NumericTolerance

class CFAnswer(BaseAnswer):
    gene: str = VerifiedField(description="causal gene", ground_truth="CFTR",
                              verify_with=ExactMatch(), weight=3.0)
    chromosome: int = VerifiedField(description="chromosome", ground_truth=7,
                                    verify_with=NumericExact(), weight=1.0)

assert CFAnswer(gene="cftr", chromosome=7).verify() is True
partial = CFAnswer(gene="CFTR", chromosome=99)
assert partial.verify() is False
assert partial.verify_granular() == 0.75

class Rel(BaseAnswer):
    v: float = VerifiedField(description="HbA1c", ground_truth=7.2,
                             verify_with=NumericTolerance(tolerance=0.5))
class Abs(BaseAnswer):
    v: float = VerifiedField(description="HbA1c", ground_truth=7.2,
                             verify_with=NumericTolerance(tolerance=0.5, mode="absolute"))
assert Rel(v=8.9).verify() is True     # relative: 8.9 is within +50% of 7.2
assert Abs(v=8.9).verify() is False    # absolute: 8.9 is not within 7.2 +/- 0.5

rubric = Rubric(regex_traits=[
    RegexRubricTrait(name="cites_pmid", pattern=r"PMID:\s*\d{6,8}"),
    RegexRubricTrait(name="no_hedging", pattern=r"\b(?:might|maybe|possibly)\b",
                     case_sensitive=False, invert_result=True),
])
good = {t.name: t.evaluate("The causal gene is CFTR (PMID: 2475911).")
        for t in rubric.regex_traits}
weak = {t.name: t.evaluate("It might possibly be CFTR.")
        for t in rubric.regex_traits}
assert good == {"cites_pmid": True, "no_hedging": True}
assert weak == {"cites_pmid": False, "no_hedging": False}

print("granular (3 of 4 weight units):", partial.verify_granular())
print("good answer traits:", good)
print("weak answer traits:", weak)
print("trait names:", rubric.get_trait_names())
print("OK")
PY

python karenina_tryit.py

Expect

granular (3 of 4 weight units): 0.75
good answer traits: {'cites_pmid': True, 'no_hedging': True}
weak answer traits: {'cites_pmid': False, 'no_hedging': False}
trait names: ['cites_pmid', 'no_hedging']
OK

Invariants (a failure means this page is wrong): verify() is all-or-nothing while verify_granular() returns the weighted pass fraction; a relative tolerance is wider than the same number read as units; invert_result flips a regex trait.

Observed against karenina 0.1.0 (git 140584a) on Python 3.11.15, 2026-09-12. A changed default — normalize on ExactMatch, or mode on NumericTolerance — would change these values, and that is drift to investigate rather than a bug in the page.

Limits

• No PyPI release as of 2026-09-12; installation is from the repository, which means you are pinning a commit rather than a version. karenina.__version__ reports 0.1.0.
• verify_granular() returns a float, so if you want to know which field failed you have to compare fields yourself.
• Deterministic traits do not judge meaning. no_hedging above fires on the word, not the stance — a confidently wrong answer passes it.
• The library also carries multi-turn conversation and agent-trace primitives (AllTurns, LastTurn, TraceContains, TraceLength) and a Benchmark object with SQL storage. This page covers single-answer scoring only.