Four weeks of testing a local GraphRAG pipeline

Over four weeks and about $300 in rented GPU time, I ran ten rounds of experiments on Orb, my desktop note-taking app. I wanted to see how well its local question-answering pipeline actually performed.

As it turned out, the single biggest performance jump came from a bug rather than a clever strategy. The reranker was assigning the exact same score to every result it felt confident about. Fixing that single bug jumped the answer F1 score on HotpotQA from 0.52 to 0.69 and cut question-answering time by 2.5x.

What Orb is and what I tested

Orb is a desktop note-taking app that answers questions using your own notes, completely offline on your computer.

When you save a note, a small local language model reads it and extracts entities and relationships into a local knowledge graph. When you ask a question, Orb searches through that graph, reranks the relevant snippets, and lets an answering model investigate step by step before giving you an answer:

I went into these experiments with two main goals:

1. Help the pipeline answer more multi-hop questions correctly.
2. See if a smaller model could handle graph extraction as well as Gemma 4 12B, so Orb could run smoothly on everyday laptops.

How I tested

Every experiment ran Orb's real backend end to end on a rented GPU: ingest the notes, build the index, ask the questions, and score the answers.

Datasets: I tested on two multi-hop question datasets, where each question comes with its own small collection of notes. HotpotQA (2-hop, 100 questions, 990 notes) was the easier benchmark. MuSiQue (2 to 4 hops, 50 questions, 526 longer Wikipedia passages) was the tough one. Each passage was saved as an individual note, just like real notes in the app.

One strict rule: No fixing model output behind the scenes. No regex patches, no text cleanup, and no silent retries to hide errors. If an extraction was malformed, that note was lost. If an answer came back broken, it scored zero. This makes my numbers lower than many published benchmarks, but it reflects what a user would actually experience.

The testing harness: Built on a dedicated branch (orb-testing), the harness included:

• Configuration sweeps: Defined a baseline setup and tested changes one variable at a time. Experiments that shared an index reused it to save time.
• A model call cache: Every LLM response, embedding, and reranker score was cached by its exact input. Repeating a run cost zero GPU time and returned bit-for-bit identical results.
• Granular retrieval metrics: Tracked whether a gold note made it into the index, whether search retrieved it, and whether it made it to the final answering model. This separated extraction failures from search failures.
• Oracle runs: Fed the gold notes straight to the answering model without any search step. This showed the absolute ceiling for each model.

Statistics: Answers were evaluated using Exact Match (EM) and token F1. Comparisons were paired question by question, using bootstrap confidence intervals for F1 and McNemar's test for exact matches. On a 50-question sample, differences smaller than roughly 0.05 F1 are mostly noise, so I only counted a change as an improvement if its confidence interval stayed above zero.

Models tested: Gemma 4 (E2B, E4B, 12B, 26B-A4B), Qwen 3.5 (0.8B to 9B), Phi-4 14B, and Granite 4.2 8B. All were quantised to 4-bit and run through llama.cpp. For retrieval, I used Qwen3-Embedding and Qwen3-Reranker (0.6B, 4B, and 8B versions).

The ten rounds

Each round answered the questions raised by the one before it. F1 numbers in different rounds use different question samples, so compare within a row rather than down a column.

Round	Dates (2026)	Question	What it found
0	Sep 28	Where does ingestion time go?	Constraining extraction to JSON using a grammar was about 4x slower because llama.cpp evaluated the grammar on the CPU token by token.
1	Sep 29-30	Which model should extract the graph?	Gemma 4 12B scored 0.767 F1 versus 0.429 for E4B on 20 HotpotQA questions. Qwen 3.5 4B failed completely because it added plain-text reasoning before its JSON.
2	Oct 1-2	Does that hold at scale?	Yes. Gemma 4 12B scored 0.670 versus 0.416 on 50 fresh questions. E4B could not extract MuSiQue at all. The extraction model mattered far more than the answering model.
3	Oct 2-3	Can a smaller model extract as well?	Qwen 3.5 9B had an early lead at 20 questions that disappeared at 50. On MuSiQue, half the gold notes never reached the index, showing answers were lost during extraction rather than search.
4	Oct 3-5	Can extraction failures be cut?	Adding a closing reminder about relationship formatting and using a JSON grammar both reduced failures. Two-pass extraction and smaller chunks made things worse.
5	Oct 4-7	Do the fixes hold, served in parallel?	Combining the reminder and JSON grammar boosted MuSiQue gold notes in the index from 52% to 67%. Switching to a smaller embedder and reranker lost 6 points of exact match.
6	Oct 10-11	Can a new model beat the 12B?	Gemma 4 12B QAT beat the baseline on HotpotQA (0.681 vs 0.527) but struggled on MuSiQue (0.152 vs 0.220). Phi-4, Granite 4.2, and Gemma 26B-A4B were not strong enough replacements.
7	Oct 11-12	Which retrieval strategies from the literature help?	Storing raw passages, running keyword search on them, querying the question directly, and reordering evidence all helped. PageRank expansion did not. Oracle tests showed the answering model was the real bottleneck.
8	Oct 12-13	What does fixing the reranker change?	Everything: HotpotQA F1 jumped from 0.524 to 0.686 on the exact same questions, running 2.5x faster. With the reranker fixed, the round 7 retrieval tweaks added just 0.02.
9	Oct 14	Does that hold on the hard dataset?	On MuSiQue, the retrieval strategies made a big difference, jumping F1 from 0.223 to 0.342. Larger answering models pushed past E4B's ceiling up to 0.41.

What I found

Retrieval is no longer the main bottleneck holding back Orb. The answering model is. Getting to this point came down to two major shifts: fixing a sneaky reranker bug and making sure each note's raw text remained searchable.

1. A hidden reranker bug skewed every early result

Orb uses Qwen3-Reranker-8B running locally in llama.cpp. To score relevance, it asks the model if a document is relevant and calculates the probability of "yes".

My original implementation asked llama.cpp for the top five next-token log probabilities. It only computed the relative score if both "yes" and "no" appeared in those top five tokens. Whenever the model was very confident, the minority token fell outside the top five. When that happened, the code fell back to crude hardcoded numbers: 0.9 for "yes" and 0.1 for "no".

Because of this, every clearly relevant candidate tied at 0.9. They ended up sorted by their original arrival order instead of true relevance.

Directly inspecting the logits for "yes" and "no" fixed the issue. On the same 50 HotpotQA questions, F1 jumped from 0.524 to 0.686, and exact match rose from 42% to 56% (+8 correct, -1 wrong). It also made question answering 2.5 times faster (dropping from 274 to 110 seconds) because llama-cpp-python no longer needed to copy the full vocabulary logits for every token. This bug had been quietly sitting in the pipeline through rounds 1 to 7.

2. Most answers were lost during extraction, not search

In round 3, only 49% of the gold reference passages in MuSiQue ever made it into the graph index. Once in the index, search successfully retrieved about 90% of them. The problem was that half the notes were being dropped before search even started.

These losses came from rejected extractions: the model produced invalid JSON, or it created a relationship pointing to an entity it forgot to define. Because of my strict no-repair rule, any malformed extraction meant the note was thrown out entirely.

Adding a simple closing prompt reminding the model about relationship rules, paired with a JSON grammar, boosted the indexing success rate to 67% in round 5.

3. Always keep raw text searchable

The best graph-RAG implementations in the research literature keep original raw text chunks alongside the graph. Initially, Orb relied solely on the extracted graph.

I changed the pipeline to chunk and index each note's raw text before running graph extraction. That way, even if graph extraction fails, the text stays searchable. Adding these text chunks to BM25 keyword search, tagged with the note title, raised MuSiQue F1 from 0.223 to 0.342 in round 9. This was the first algorithmic change whose confidence interval decisively cleared zero.

4. The answering model sets the performance ceiling

When you hand Gemma 4 E4B the gold notes directly (an oracle test), it scores 0.718 F1 on HotpotQA and 0.357 on MuSiQue. The full pipeline now practically matches that ceiling, scoring 0.716 and 0.342.

In fact, simply dumping all available notes for a question directly into E4B's context window without any retrieval yields nearly the same score in one-thirtieth of the time. To get better answers from here, we need a smarter answering model, not better search:

Qwen 3.5 9B matched E4B's speed while delivering better answers. Gemma 4 12B was slightly better still, but took 25% longer.

5. The model refuses to hallucinate, but communicates poorly

I tested 50 HotpotQA questions where none of the relevant notes existed in the index. The research loop behaved responsibly and never hallucinated a fake answer.

However, once the model ran out of search steps, it simply returned its last intermediate finding. In 19 to 29 of those 50 tests, the response sounded misleadingly like an answer (for example, starting with "The documents confirm that..."). The fix here is prompting and response formatting, not better retrieval.

What did not work

Most ideas did not survive a larger sample size. The recurring pattern was an early lead on 20 questions that completely vanished at 50 questions.

Idea	What happened
A smaller extractor (Gemma 4 E2B/E4B, Qwen 3.5 0.8B-9B)	Each model extracted fewer usable notes than the 12B. The most common failure was creating relationships between items the model never defined as entities. Qwen 3.5 9B led at 20 questions (F1 0.696) but lost at 50 (0.555 vs 0.683).
Granite 4.2 8B as extractor	Successfully ingested 76% of HotpotQA notes (compared to 98% for the 12B) and only 19% of MuSiQue notes.
Phi-4 14B as extractor	Best of four new candidates on HotpotQA (0.601), but worst on MuSiQue (0.119).
Two-pass extraction (entities first, then relationships)	Caused more failures rather than fewer. The second pass frequently referenced entities the first pass never extracted.
Smaller chunks for long notes	A single failed chunk still caused the entire note to fail.
Personalized PageRank over the graph (HippoRAG 2 style)	Produced mixed results across datasets and was noticeably slower.
Community summaries (GraphRAG style)	Offered no accuracy gain on HotpotQA and was very slow to compute.
More research steps (increasing from 5 to 8)	Made no difference on either dataset.
A cheaper retrieval stack (0.6B embedder and reranker, no graph expansion)	Individually, changes were within noise margins. Together, they lost 6 points of exact match.

One early lesson on grammars: constrained JSON decoding was 4x slower at first and initially looked like a mistake. But once prompts were cleaned up and requests ran in parallel, it became the most reliable extraction setup for long notes. Sometimes a technique only shows its value once you fix the bottlenecks around it.

The research behind it

Two rounds of background research shaped rounds 6 and 7: one examining modern graph-RAG architectures and open models, and another looking at proven retrieval strategies.

How other systems build GraphRAG: Orb's design is closest to HippoRAG 2 and LightRAG. They use one extraction pass per document, allow free-form entity and relationship types, and combine vector search with a knowledge graph. Three insights stood out:

• High-performing systems always keep raw text passages alongside the graph. Systems that rely only on extracted entities and relationships lose 5 to 10 points of F1 on fact-based questions compared to standard dense retrieval. Adding raw passage storage to Orb fixed this gap.
• HippoRAG 2 expands search through the graph using Personalized PageRank. I implemented this, but in Orb it did not improve accuracy and slowed things down.
• Benchmarks in papers rarely treat malformed JSON as a lost document. Most papers silently repair or retry bad outputs. Counting every broken extraction as a total loss made Orb's benchmark numbers stricter than typical academic reports.

Choosing the models: The survey helped pick candidates for round 6: the Gemma 4 12B QAT build (quantisation-aware trained for 4-bit), Gemma 4 26B-A4B (a mixture of experts with about 4B active parameters), Phi-4 14B, and Granite 4.2 8B. These were chosen for their strong track record on structured data. I also established a few practical rules: disable thinking modes during extraction, stick to at least 4-bit weights, use an f16 KV cache, and target a memory budget of around 16 GB so everything fits comfortably on a 24 GB Mac.

Proven retrieval strategies: Modern RAG surveys all point to the same principle: isolate the weakest link in the pipeline and address it with low-cost fixes first. That usually means hybrid keyword and vector search, a solid reranker, and chunk-level metadata (similar to Anthropic's contextual retrieval).

They also strongly advise measuring each pipeline stage independently, especially using oracle runs on the gold evidence. In Orb, the oracle run was the exact test that proved the answering model was the real bottleneck.

In round 7, I tested six ideas one by one: adding raw passages to keyword search, prefixing notes with title headers as lightweight context, querying the original question first, reordering evidence to counter the lost-in-the-middle effect, PageRank graph expansion, and explicit refusal tests. Complex ideas that felt ill-suited for a local desktop app (like vision-based document parsing or RL-trained search agents) were left out.

Infrastructure and bugs along the way

As free credits ran out, I bounced between four different GPU providers. To keep things manageable, I wrote a small automation script for each provider: spin up an instance, set up dependencies, pull the sweep queue, stream results back to my local machine, and shut down automatically when finished or reaching a budget limit.

Provider	GPU	Rounds	Notes
RunPod	RTX 4000 Ada, RTX 4090, A40 (48 GB)	0-5	Ran up to six pods simultaneously; paid out of pocket
AWS	L40S (48 GB), g6e instances	6-8	Funded by cloud credits; hit frequent capacity shortages in us-east-1
Google Cloud	None	N/A	GPU quota requests were rejected automatically on a new account
Modal	L40S (48 GB)	9	Funded by cloud credits; serverless setup with zero server management

Running automated benchmarks on remote GPUs is messy. Here are the main things that broke:

• Prompt cache mixups in llama-server: An open bug in llama.cpp caused the prompt cache to leak context between parallel request slots. Luckily, I caught this before saving any benchmark runs. Setting --cache-ram 0 resolved it.
• A model unloading race condition: When swapping models, 16 out of 20 questions crashed silently. The evaluator averaged the score using only the 4 questions that succeeded, reporting an artificially inflated F1 of 0.938. I added concurrency locks in the backend and updated the evaluation harness so crashed queries count as an automatic zero.
• CPU over-subscription: llama-server defaulted to 96 threads inside a container that only had 8 virtual CPUs allocated. Disk writes stalled and timed out under the load.
• A broken watchdog timer: The shutdown script used integer math on fractional hours, which meant the time-limit cap never triggered. Fortunately, balance checks and external monitors stopped the machines before bills ran away.
• Lazy disk loading from snapshots: Cloud instances created from saved disk templates read files lazily from network storage. Models loaded so slowly on first boot that they missed their 15-minute startup health check twice.
• A SQLite lock timeout during parallel ingestion: Writing multiple notes concurrently locked the SQLite database, leaving notes stuck in a "processing" state forever. This was a genuine product bug in Orb, and it is now fixed.
• Home internet dropouts: A few times, my home connection dropped while managing long sweeps. One Modal job ran for 5 hours and had to be aborted because the local client script disconnected. I changed the harness to launch jobs entirely server-side.

None of these issues skewed the final results because every problem was caught by an automated check, a watchdog script, or a score that looked suspiciously high. Still, debugging them burned quite a few hours of paid GPU time.

What it cost

All in all, the project ran through roughly $300 in compute: $145 of my own money on RunPod, and around $150 in promotional cloud credits.

Provider	Spent	Paid with	Covered
RunPod	About $145	Out of pocket	Rounds 0 to 5, setup, and sanity checks
AWS	About $129	Cloud credits ($160 granted)	Rounds 6 to 8
Modal	About $22 (est.)	Cloud credits ($30 granted)	Round 9, plus an aborted run
Google Cloud, Azure	$0	N/A	No GPU quota granted
Total	About $296

I estimate that about 20% of the total spend was wasted on infrastructure hiccups: restarts from the prompt-cache bug, idle instances during home network drops, and aborted jobs.

On the bright side, the model call cache saved significantly more than that. Because every extraction, embedding, and reranker score was saved locally, rerunning an experiment replayed cached inferences for free rather than paying for duplicate GPU cycles.

Lessons learned

1. Test the components you trust most: I took the reranker for granted and never benchmarked it in isolation. Fixing its bug produced a bigger gain than all other retrieval strategies combined. Always measure each pipeline component separately before spending time tuning the system around it.
2. Run oracle tests right away: Feeding the reference gold notes directly to the answering model takes only a few minutes. It immediately reveals whether your bottleneck is search or synthesis. I waited until round 7 to do this; had I done it on day one, it would have saved me rounds 2 through 5.
3. Small sample sizes give false confidence: Multiple strategies that looked like clear winners on 20 questions completely fizzled out on 50 questions. Pair your comparisons, calculate confidence intervals, and treat F1 differences under 0.05 on 50 questions as pure noise.
4. Do not patch model mistakes in your benchmark: Post-processing hacks and silent retries mask how brittle an extraction pipeline really is. Treating every malformed output as a failed note was the only reason I discovered that answers were being lost during graph creation, not retrieval.
5. Make runs completely deterministic and replayable: Caching every model call by its exact input, combined with fixed seeds and dates, meant running the same configuration twice yielded the exact same result. That made differences easy to isolate and made reruns virtually free.
6. Be suspicious of unusually good numbers: An F1 score of 0.938 sounded incredible, but it was just a bug where 16 failed questions were silently excluded from the average. If a benchmark looks too good to be true, check the denominator.
7. Expect remote infrastructure to break: Always set hard spending limits, run watchdog processes, save results incrementally to disk, and decouple jobs from your local internet connection.

What comes next

I am rolling these improvements into Orb in order of impact:

1. The reranker logit fix
2. The SQLite concurrency lock fix
3. Storing and searching raw text passages alongside the graph
4. The extraction prompt reminder and JSON grammar
5. Better wording when the model cannot find relevant notes

Beyond those immediate updates, there are a few next steps I want to explore:

• Testing on larger benchmarks: The findings so far are based on 50 to 100 questions. I want to validate the final pipeline across several hundred questions.
• Re-evaluating extraction models: Gemma 4 12B QAT won on HotpotQA but struggled on MuSiQue, and that test ran before the reranker fix. It deserves a proper rematch.
• Finding lighter answering models: Qwen 3.5 9B answered as quickly as E4B while matching the quality of Gemma 4 12B. Finding efficient models in this class will make local question-answering much more practical on laptops.
• Improving latency: The full research loop can take a couple of minutes per query, whereas reading small collections of notes directly takes seconds. For personal notebooks with only a few hundred notes, a fast path that bypasses the multi-step research loop might be both faster and just as accurate.