Late Interaction on turbopuffer

What the namespaces hold, and how the query flows through them.

Companion page: the Top-K problem in late interaction →

1. What lives in the database

Two namespaces. One dense vector per document. One ColBERT vector per token.

2. Query flow (the funnel)

You can't ColBERT-score every doc. So narrow the field with cheap dense ANN first, then apply ColBERT to the survivors.

Stage 1 is fast but blurry (one similarity per doc), so the right answer might land at rank 47 instead of rank 1.
Stage 2 is slow but sharp (~480 similarities per doc), and can only fit if we first narrow to 100 candidates.

3. MaxSim, in tiny numbers

The ColBERT score for one (query, doc) pair is the sum of per-query-token best matches. Here it is with 3-dim unit vectors so the arithmetic is doable in your head.

Set up the vectors

Query: "make money"                      Doc A: "earn cash"
q1 = "make"  = [0.6, 0.8, 0.0]           d1 = "earn"  = [0.5, 0.7, 0.1]
q2 = "money" = [0.0, 0.5, 0.9]           d2 = "cash"  = [0.1, 0.4, 0.9]

Compute every pairwise dot product

Dot product of two unit vectors is a similarity in [–1, 1].

make · earn  = 0.6×0.5 + 0.8×0.7 + 0.0×0.1 = 0.86
make · cash  = 0.6×0.1 + 0.8×0.4 + 0.0×0.9 = 0.38
money · earn = 0.0×0.5 + 0.5×0.7 + 0.9×0.1 = 0.44
money · cash = 0.0×0.1 + 0.5×0.4 + 0.9×0.9 = 0.99

"Max": each query token keeps its best doc match

q1 = "make"  → max(0.86, 0.38) = 0.86   (best: "earn")
q2 = "money" → max(0.44, 0.99) = 0.99   (best: "cash")

"Sum": add up across query tokens

Doc A score = 0.86 + 0.99 = 1.85

Compare to another doc

make  → best match "earn"  = 0.86
money → best match "cash"  = 0.99
                       ────
              score = 1.85

make  → best match "buy"   = 0.21
money → best match "shoes" = 0.18
                       ────
              score = 0.39

Doc A wins because every query word found a strong match, even though "earn" and "cash" aren't the same words as "make" and "money". Dense retrieval averages this signal away; MaxSim keeps each word independent.

What real ColBERT does

4. How turbopuffer executes stage 2

Naive path: fetch every doc's token vectors to the client, compute MaxSim locally. Lots of network, lots of math.

Server-side path (what the guide uses): for each of the 32 query token vectors, send an ANN query into the token namespace, filtered to the 100 candidate doc IDs. turbopuffer runs the dot products in its ANN engine and returns the nearest doc tokens with distances attached. The client never multiplies vectors; it just aggregates scores.

for each query_token q (32 total):
    hits = token_ns.query(
        rank_by=("vector", "ANN", q),            # turbopuffer computes q · v
        filters=("doc_id", "In", candidate_doc_ids),
        top_k=1500
    )
    # Each hit comes back as (doc_id, $dist)
    # where $dist = 1 − (q · v). Flip it back:
    for row in hits:
        sim = 1.0 - row["$dist"]                 # recover similarity
        if sim > best_per_doc[row["doc_id"]]:
            best_per_doc[row["doc_id"]] = sim    # max per doc

    for doc_id, sim in best_per_doc.items():
        scores[doc_id] += sim                    # sum across query tokens

The only multiplication in your code is not a multiplication at all. It's the 1 − $dist subtraction to convert turbopuffer's distance back into similarity. All actual vector math (the dot products that define similarity) runs inside turbopuffer's ANN engine.

Those 32 queries are batched 16 at a time into multi_query, so stage 2 is 2 API calls regardless of candidate count. No raw doc vectors cross the network.

5. Putting it together for one query

Query: "How can I make money online free of cost?"

Stage 1: dense returns top 100, ordered by cosine similarity:
  rank  doc_id  text
  ──────────────────────────────────────────────────────────
  1     273     "How do I earn money online without investment?"
  2     891     "Best ways to make passive income"
  3     42      "Free online income opportunities"
  ...
  10    0       "How do I make money online?"          ← actual best match
  ...
  47    9921    "Online business ideas for beginners"
  ...
  100   5043    "How to budget your monthly expenses"

Stage 2: ColBERT rescores those 100:
  rank  doc_id  colbert_score
  ──────────────────────────────────────────────────────────
  1     0       14.7   "How do I make money online?"   ← promoted from rank 10
  2     273     13.9   "How do I earn money online..."
  3     42      13.2   "Free online income opportunities"
  ...

ColBERT promoted doc 0 because every query word (make, money, online) found an exact token match. Dense ranked it #10 because the extra query words (free, of, cost) diluted the averaged similarity. MaxSim doesn't care about averages. Each query word picks its own best match, independently.

6. How we measure quality: MRR@10

One query is an anecdote. To compare dense vs. ColBERT across a whole dataset, we need a score that summarizes "did search rank the right answer near the top?" across many queries. The standard metric is MRR@10, Mean Reciprocal Rank at 10.

6a. The scoring rule

For each test query you know the correct answer. Find where it lands in your results. The closer to position 1, the more credit you get.

The "@10" just means "look at the top 10 results only, anything past that scores zero." MRR@5 or MRR@100 work the same way with different cutoffs.

6b. Worked example: 5 test queries

Imagine you have 5 test queries, and for each one you know the correct answer. Run them all, find the rank of the right answer, compute 1/rank, average.

Higher is better. 1.0 is perfect (every correct answer was rank 1). 0.0 means the correct answer was never in the top 10.

6c. What the numbers feel like

6d. The actual result on Quora

Running MRR@10 on 100 known duplicate pairs from the Quora dataset:

Both scores are high because Quora questions are short, and a single OpenAI vector captures short text well. ColBERT's per-token precision doesn't get a chance to shine when there's no lost detail to recover. On longer documents (support articles, contracts, product catalogs) the gap flips in ColBERT's favor.

7. Looking ahead: multi-vector support

The current implementation works, but it leaks complexity into user code: two namespaces, an ID-hack to link them, a doc_id filter, 32 token-level ANN searches batched into 2 multi_query calls, and client-side aggregation. All of that exists because turbopuffer stores one vector per row today. If rows could hold a list of vectors, most of that complexity collapses.

7a. The data model

7b. The schema that makes it work

The server can only score multi-vector columns if it knows the column holds a list of vectors. That's what the schema declares:

ns.write(
    upsert_rows=[...],
    distance_metric={
        "dense":          "cosine_distance",
        "colbert_tokens": "cosine_distance",
    },
    schema={
        "dense":          {"type": "vector[1536]"},     # single vector per row
        "colbert_tokens": {"type": "vector[128][]"},    # ← list of 128-dim vectors
        "text":           {"type": "string"},
    },
)

vector[128][] tells the server two things: (1) build an ANN index across all tokens from all rows, with back-references to the row they came from; (2) this column is eligible to appear as the first argument to a multi-vector operator like MaxSim.

7c. The query: one call, two stages

results = ns.query(
    rank_by=[
        ("dense", "ANN", dense_vec(query)),           # stage 1
        ("colbert_tokens", "MaxSim", q_tokens),       # stage 2
    ],
    candidates=100,        # width of stage-1 pool
    top_k=10,              # number of stage-2 results to return
    include_attributes=["text"],
)

7d. How the server executes it

Execute ANN index lookup on the `dense` column with query D.
Result: [(row_id, dist)] × 100 candidates

  row_id    dist
  ────────  ──────
  273       0.12
  891       0.14
  42        0.16
  ...
  0         0.31   ← actual best, buried at rank ~10
  ...
  5043      0.42

for row_id in candidates:
    doc_tokens[row_id] = storage.get(row_id, "colbert_tokens")
    # e.g. doc_tokens[0] = [[0.18,-0.04,...], [0.09,0.31,...], ... 10 vecs]

Q = q_tokens   # shape (32, 128)

for row_id in candidates:
    V = doc_tokens[row_id]                    # (N, 128)
    sim_matrix = Q @ V.T                      # (32, N), one matmul
    row.score = sim_matrix.max(axis=1).sum()  # row-wise max, then sum

sort candidates by score desc
return top 10 with requested attributes

7e. The response shape

Because stage 2 is the final ranker, each row comes back with a single $score, the aggregated MaxSim value. No per-query-token distances, no client math:

[
  { "id": 0,   "$score": 14.72, "text": "How do I make money online?" },
  { "id": 273, "$score": 13.90, "text": "How do I earn money online..." },
  { "id": 42,  "$score": 13.21, "text": "Free online income opportunities" },
  ...
]

For debugging or research use cases (like "why did this doc rank here?"), an opt-in explain mode could return the 32 per-query-token alignments per doc:

{
  "id": 0, "$score": 14.72,
  "$breakdown": [
    { "q_idx": 5, "best_doc_tok": 5, "sim": 0.96 },  # "make" → "make"
    { "q_idx": 6, "best_doc_tok": 6, "sim": 0.97 },  # "money" → "money"
    { "q_idx": 7, "best_doc_tok": 7, "sim": 0.95 },  # "online" → "online"
    ... 29 more ...
  ]
}

7f. Side-by-side

Built as part of the turbopuffer late interaction guide · full guide →