Embeddings don't understand your Well ID

Three numbers

The finding

One result shaped everything else here: embeddings cannot locate identifiers, quantities, or group membership. Not “do it badly” — there is no mechanism. A well number, a seismic line name, a derived trace length, a coordinate system: each is a key, and a key is precisely what a similarity score cannot resolve. So each one got an exact lookup, and each lookup is measured below against the ranked baseline it replaced.

Around that sits a deliberately small pipeline: walk a folder, read what can be read, chunk it, embed it, answer with citations. One DuckDB file holds the text, the vectors and the extracted facts. Both models are fixed and cheap on purpose — openai/text-embedding-3-small embeds and inclusionai/ling-3.0-flash answers, through OpenRouter. Comparing models is not the point of this.

The readers

Seismic, measured

Without a reader, libmagic cannot name a SEG-Y file at all, so it is dropped with a recorded reason and nothing inside it is searchable. With one, the same 54 files yield 119 passages and 741 facts, every stored number checked against the range its field allows.

Ground truth comes from the headers themselves: a file whose card reads CLIENT: LITHOPROBE belongs in the answer set for a question about Lithoprobe. A seismic archive is many files per survey, so these questions have several right answers and are scored against the set rather than one blessed document.

The zeroes are the interesting rows. Trace length is derived — sample interval times sample count — so the number appears nowhere in the file and nothing in the index resembles the question. Ranking has nothing to grip; a lookup answers it exactly. The unit has to appear in the question too, so “18000 barrels” is never read as a trace length. A sample interval is stored twice, in microseconds and milliseconds, so a question in either unit matches whichever way the file was written.

Sample counts are the weakest case for a lookup and still worth it: 2000 is real text in the header, so ranking already found the right files most of the time — and put the wrong one first a quarter of the time.

Well numbers have no canonical length

Vendors write fourteen digits. State agencies write ten. A map layer sometimes writes seven, with the state code left off entirely. Each is what that system holds, and picking one as “correct” throws away the sidetrack and completion codes the longer forms carry. So no length is imposed: separators are stripped, the digits are kept as written, and two numbers name the same well when one runs on from the other — sharing a start, where both begin at the state, or an end, where one has dropped it.

A fragment can be starred like a glob, in a question or in --where:

The last matters more than it looks. A fourteen-digit number carries a sidetrack after the well, so a fragment lifted out of a map layer lands in the middle of it: 2506325 ends 2506325 and sits inside 490250632500. Only the double star reaches both.

A label in prose, a column in a table

API numbers are detected in any document — a completion report, a loader log, a scanned permit — and validated against this project's own table of state and county codes. In prose a label must precede the digits, which is what produced the zero in the third card above. In a .dbf column no label is required, and that asymmetry is the point: a column is evidence, a sentence isn't. If most of a column validates, it is a column of well numbers whatever it happens to be called. In this corpus GeoGraphix layers keep them under DataId and WellID; across every other numeric column — TypeId, ObjectID, ParentCode, ASR_ID — not one value validates. That found 6,908 well numbers across four layers, none of them searchable before.

The state and county are already in the digits

The numbering puts two digits of state and three of county at the front, so a stored well number already says where it is. Until that is read back out, nobody can search for it by name: “what LAS files are in TX” previously matched nothing whatsoever and returned Wyoming logs.

Only 12- and 14-digit values are trusted for it. Ten is excluded because the last ten digits of 490250632500 are 0250632500, which parses as state 02, county 506 — a real-looking Arizona well that does not exist. Shorter values are fragments whose digits do not line up at all: Teapot_Wells writes 2500153, which is county 025 and well 00153 with the state gone. A wrong county is worse than no county. Across the corpus this reads WY and Natrona off the two layers storing twelve digits, and correctly nothing off the two storing seven.

A two-letter code is only recognised written in capitals, because half of them are ordinary words — IN, OR, ME, OK, HI, DE, LA. “Wells in or near the field” would otherwise ask for Indiana and Oregon.

Shapefiles: classify the columns by measurement

.dbf attributes are sorted by what their values do, never by what they are named, because field names are a vendor's abbreviations and there is no list to check them against. Values that repeat are a category and become a filterable fact; values nearly all distinct name individual things and go into the text; a column that is 90% one value describes the layer rather than the row; a column of bare digits is dropped, because retrieval is text and a number that names nothing cannot be searched for.

Group membership is filtered, not ranked

A projection, a state and a county each name a set of documents rather than describing one, and the rarity boost cannot deliver them: 21 layers out of 2,520 documents earn about 0.06, which will not lift anything past 68,000 chunks. An exact match narrows the candidates instead.

Whether silence is a contradiction is read from the index

The obvious way to filter — drop every document that does not carry the key — deleted every road and benchmark layer for any question naming a state, and cost the shapefile set seven points. Only a document with a parsed well number carries api_state; a roads layer saying nothing about a state is not claiming to be outside it. So the rule is asked of the data rather than written down:

My own assumption going in was that every shapefile has a .prj. A third of them do not. No list of keys is maintained for this, so a new reader that always records something gets the same treatment with no edit.

The questions people actually ask

The shapefile set asks the three things a person really puts to a pile of map layers, and scores 67% on the top hit, 81% recall@10 over 107 of them:

Every well-number question puts a right answer first. Only the starred form recovers the whole answer set, which is exactly what the wildcards are for. An earlier version of this set asked “which layer has 53 features?”, scored 0%, and was measuring the question generator rather than the system — nobody searches for an exact feature count. Asked the realistic way, “how many features does Teapot_Wells have”, it returns rank one.

Retrieval, overall

Cosine similarity, with three corrections that matter for a collection like this one: an identifier in the question is looked up rather than ranked; metadata the question names lifts the documents carrying it, weighted by rarity, so a well name held by one document outweighs state=WYOMING held by 1,375; and no single document may take more than its share of the answer — a map layer of 2,111 wells is 500 chunks that read alike, and without that cap it held every place in the top ten for any question about wells.

Read recall@5 first — top_k is 6, so that is what the model actually sees. The LAS row is what a format reader buys you. The plain-text and HTML rows are ordinary documents with no reader and no metadata, and they are the honest floor this sits on.

Sample size is why this set exists at all. At 22 questions the 95% interval on a recall figure is about ±16 points, so any change worth less than that was indistinguishable from luck. At 103 it is about ±8 — still wide enough to be careful with, and stated here rather than hidden. The whole corpus rebuilds cold in about five minutes for $0.56: 2,530 files, 67,919 chunks.

Two things built, measured, and switched off

Both are in the repository, both are wired up, and both default to off with the numbers written next to the setting. Deleting them would hide the result.

BM25 and hybrid search

DuckDB's FTS extension, fused with the dense ranking by reciprocal rank. On 39 seismic questions with the lookups disabled it halved retrieval — 36% to 18% on the top hit — and cost recall with them enabled. Asked “where is seismic line 93D” it returns a telephone-system procurement PDF, because “line” appears in 1,121 chunks and the one discriminating token appears in 138.

HNSW vector index

DuckDB's VSS extension, built at ingest over all 67,919 vectors in about six seconds. The planner only reaches an HNSW index through a bare nearest-neighbour query, which means candidates are cut before the filter and the metadata boost are applied — and those are precisely the two things this pipeline needs exact. Merging every chunk of a boosted document back in recovered most of the loss and not all of it:

It saves 0.3 seconds of SQL in a query that spends most of a second waiting on the embedding API. At 68,000 vectors the brute-force scan is the better trade. At ten million it would not be, and the switch is already there.

What this does not do, and why

Where it stops

Two gaps are left open deliberately. Neither is exotic, and both are the kind of thing that turns a demonstration into something an asset team would actually run:

• Legacy Office formats are unreadable. 81 .doc, 61 .xls and 58 .ppt files in this corpus are skipped with a recorded reason — and in a real data room that stratum is where the partnership agreements, the AFEs and the decades of internal memos live. textutil covers .doc on macOS and headless LibreOffice covers all three; neither is wired in, so the pipeline reports the gap instead of quietly pretending the corpus was complete.
• Company names are only half-normalised. A name is tidied as far as its trailing legal suffix, and no further, because ENCANA appeared as three spellings across five files and split every answer set on its own. Real normalisation — ampersands against “and”, abbreviations, subsidiaries, former names, the whole merger history of an operator — is a substantially larger job with its own authority file, and pretending a regex covers it would be worse than saying it does not. Every operator-based answer on this page carries that caveat.

The pattern in both is the same one the rest of the project argues for: a gap that is named and measured is worth more than a gap that is papered over, because the first one tells you what to buy next and the second one tells you nothing.

Untrusted text in the prompt

Retrieved chunks come from files nobody vetted, and they go into a prompt. Each source is fenced with a random token generated per request, and that token is stripped from the source text before it is inserted — so a document cannot close the fence and start addressing the model as the operator.

Running it

uv run geo-mini-rag ingest             # read data/raw, embed via OpenRouter
uv run geo-mini-rag stats              # what got indexed, and why the rest did not
uv run geo-mini-rag search "question"  # retrieval only, one tiny embedding call
uv run geo-mini-rag ask "question"     # answer with citations

To see what the pipeline reads out of a single file, with no API key and no database:

python -m geo_mini_rag.ep.inspect data/raw/las/some.las
python -m geo_mini_rag.ep.inspect --facts data/raw/gis/layer.shp