♻️ EchoCache

A semantic cache in front of your LLM API: it answers repeat questions from memory instead of paying for them again — and refuses to answer a different question from cache.

Matching is 100% local (hashed character n-grams + SimHash, no model download, no network). Apache-2.0.

This page is documentation, not the running app.

Hugging Face now requires a PRO subscription to host Gradio Spaces on free CPU hardware, so this Space is published as a static page. Everything below is real — the code, the measurements and the deploy steps — but there is no live demo to click here.

To run it: take app.py and requirements.txt from this repository and either push them to a Gradio Space on paid hardware, or run python3 app.py locally. It is one file with no model download and it starts in seconds.

Space — this repo

NagaYu/EchoCache

The full application: cache, dashboard, threshold sweep, audit, JSON API. Static here; runnable anywhere.

Model

echocache-matcher

The matching core as one numpy-only file. Deterministic and rule-based — no trained weights.

Dataset

echocache-guard-benchmark

131 labelled pairs that measure wrong reuse. Scores any cache, not just this one.

The problem a similarity score cannot solve

A cosine of 0.95 means “these two strings look alike”. It does not mean “these two questions have the same answer”. Can I cancel my subscription? and Can I not cancel my subscription? are near-identical to any similarity function and have opposite answers. Serving the cached answer there is not a cache miss — it is a wrong answer, and it costs far more than the tokens it saved.

EchoCache therefore splits the decision in two. Similarity picks candidates; a MismatchGuard decides whether a candidate may actually be served, vetoing any reuse where the meaning has shifted.

Measured correctness

Scored on the companion benchmark: 131 pairs — 41 a cache may reuse, 90 high-similarity pairs it must not. Wrong reuse counts cached answers served for a question whose meaning differs.

Threshold	MismatchGuard	Accuracy	Reuse recall	Wrong reuse
0.92 (default)	on	97.7%	92.7%	0 / 90
0.92	off	93.1%	92.7%	6 / 90
0.80	on	99.2%	100%	1 / 90
0.80	off	77.9%	100%	29 / 90
0.70	off	69.5%	100%	40 / 90

The guard is what makes an aggressive threshold survivable: at 0.80 it turns a 32% wrong-answer rate into 1%. Reproduce it with python3 evaluate.py in the dataset repo.

Real decisions, recorded from the implementation

Every similarity below was produced by running the published matcher — not estimated.

Prompt pair	Similarity	Threshold	Verdict	Reason
`Hi team, how do I reset my password? Thanks in advance!` vs `How do I reset my password?` greeting and sign-off normalized away	1.0000	0.92	REUSE	exact key match
`Could you please tell me how do I reset my password?` vs `How do I reset my password?` polite request frame is boilerplate	1.0000	0.92	REUSE	exact key match
`お世話になっております。返品の方法を教えてください。よろしくお願いいたします。` vs `返品の方法を教えてください` Japanese honorific opening and closing removed	1.0000	0.92	REUSE	exact key match
`How can I reset my password?` vs `How do I reset my password?` true paraphrase, reached at a lower threshold	0.8476	0.80	REUSE	—
`Can I not cancel my subscription?` vs `Can I cancel my subscription?` negation flip	0.9101	0.70	REFUSE	negation_mismatch:q=True,c=False
`retry after 30 seconds` vs `retry after 3 seconds` quantity	0.8840	0.70	REFUSE	numeric_mismatch:q={30},c={3}
`Does the plan include 100MB of storage?` vs `Does the plan include 100GB of storage?` unit	0.9231	0.70	REFUSE	unit_mismatch:q={100mb},c={100gb}
`Does the X1 Yoga support 32GB?` vs `Does the X1 Carbon support 32GB?` product name	0.7746	0.70	REFUSE	proper_noun_mismatch:q={x1,yoga},c={carbon,x1}
`What is the 2025 tax rate?` vs `What is the 2024 tax rate?` time anchor	0.8846	0.70	REFUSE	numeric_mismatch:q={2025},c={2024}
`Why do I need to reset my password?` vs `How do I reset my password?` question type	0.7327	0.70	REFUSE	question_type_mismatch:q={why},c={how}
`Translate cheers into German` vs `Translate cheers into French` a sign-off word used as content	0.7500	0.70	REFUSE	proper_noun_mismatch:q={german},c={french}
`I forgot my login credentials, what now?` vs `How do I reset my password?` same intent, no shared surface - a known miss	0.1174	0.92	REFUSE	temporal_mismatch:q={now},c={}

How a lookup works

Exact. sha256 of the normalized prompt. Normalization folds NFKC, case, whitespace and punctuation, and removes greetings, sign-offs, disclaimers and politeness wrappers in English and Japanese. Three very different-looking prompts collapse onto one key.
SimHash band. A 64-bit fingerprint of the same character n-grams narrows thousands of entries to a handful, with a vectorized popcount. No candidate, no cosine.
Cosine + MismatchGuard. Only survivors are scored against a hashed character-n-gram vector. Anything above the threshold must still pass five checks: negation flip, quantity/unit, proper noun or model number, time anchor, question type.

Character n-grams mean no tokenizer and no word boundaries: Japanese, Chinese and Thai behave exactly like English. Measured on a laptop with 5,000 entries in one partition: ~0.2 ms per store, ~0.11 ms for an exact hit, ~0.3 ms for a full miss.

What else is in the app

Per-tenant isolation — each tenant gets its own exact-key map, entry list, vector matrix, fingerprint array and LRU. No method takes two partitions, so there is no path that can surface one tenant's entry to another.
A safety screen before storage — Luhn-validated card numbers, credential prefixes, JWTs, PEM keys, high-entropy secrets and excess contact PII are refused with a machine-readable reason code.
Threshold sweep — upload a CSV of real traffic and see, per threshold, how much you would have reused and saved, with the guard on and off. That is the number you take to a review.
Observability — hit-rate over time, cumulative savings per tenant and route, latency split by hit/miss, most-reused entries, borderline decisions, index health, CSV export. Prompt text is not logged by default.
Audit — every candidate the guard rejected, with its reason, so a wrong hit can be traced and the entry invalidated.

The JSON API

Nine endpoints, all returning an envelope. ok: true means the call completed — a miss is a successful call.

POST /gradio_api/call/lookup   {"data": ["acme", "How do I reset my password?", 0.92, "support"]}

{"ok": true, "hit": true, "stage": "exact", "similarity": 1.0,
 "entry_id": "b67ac733...", "age_sec": 0.022, "response": "Open Settings > Security > Reset password.",
 "savings": {"saved_tokens": 18, "saved_cost_usd": 0.0},
 "summary": "HIT via exact (similarity 1.0000, age 0.022s, reused 1x)"}

{"ok": true, "hit": false, "stage": "miss", "reason": "guard_rejected",
 "best_similarity": 0.9101, "threshold": 0.7,
 "guard_reasons": ["negation_mismatch:q=True,c=False"],
 "summary": "MISS (candidate was above threshold but MismatchGuard rejected it)"}

/store · /invalidate · /stats · /sweep · /audit · /export_index · /import_index · /health complete the surface. The running app builds the exact paths for your deployment in its API Docs tab, because Gradio changes them between major versions.

Run it yourself

# locally - identical behaviour to a Space
git clone https://huggingface.co/spaces/NagaYu/EchoCache
cd EchoCache
pip install -r requirements.txt
python3 app.py            # http://localhost:7860

To host it: create a Gradio Space (paid hardware or PRO), push app.py, requirements.txt and README.md, and it builds in a minute or two. No secrets are required — HF_TOKEN is optional and only enables an embedding re-rank that the cache never depends on. Full walkthrough: deploy.md · 日本語版.

Use just the matcher

from huggingface_hub import hf_hub_download
import importlib.util, sys

path = hf_hub_download("NagaYu/echocache-matcher", "echocache_matcher.py")
spec = importlib.util.spec_from_file_location("echocache_matcher", path)
m = importlib.util.module_from_spec(spec); sys.modules["echocache_matcher"] = m
spec.loader.exec_module(m)

m.match("Can I cancel my subscription?", "Can I not cancel my subscription?", 0.5)
# {'reuse': False, 'similarity': 0.9101, 'guard_reasons': ['negation_mismatch:q=True,c=False'], ...}

Limitations

Surface similarity only. Paraphrases that share meaning but not characters are not matched, by design: How do I reset my password? vs I forgot my login credentials, what now? scores 0.12. If your traffic is full of synonym-only rewordings, pair it with embeddings or accept the misses.
The guard refuses valid reuse on purpose. Any difference in the numeric-token set vetoes a hit, so a 3 day trial and a three day trial do not match. A miss costs one API call; a wrong hit costs trust.
Free-tier reality. Spaces have no persistent disk and sleep after 48h. The index lives in RAM: export it before you go idle and import it after wake-up. A cache is not a database.
Savings are estimates. Tokens are approximated locally and multiplied by the prices you configure. Useful for comparing thresholds, not for reconciling an invoice.
The benchmark is not independent. It was written alongside the matcher by the same author, and two matcher fixes came out of it. It is a regression baseline, not a third-party leaderboard.