Kanji
・Cloud / AI Agent Dev / Freelance Engineer ・Hands-on insights & trials on AWS, GCP & Autonomous AI ・From Ehime, based in Tokyo (Born 1993) Profile details
Table of Contents
When autonomous AI coding agents (such as Google Antigravity, Claude Code, and Kiro) become everyday pair-programming partners, engineers quickly hit a shared wall: every time a session ends, the agent completely forgets past architectural decisions, troubleshooting history, and hard-earned context.
To solve this, vendors have rapidly rolled out built-in memory features: Claude Code’s MEMORY.md (Auto-memory), Kiro’s official Memory feature ( kiro.dev/docs/memory ), Google Antigravity’s Brain directory ( .brain ) and /learn command, and OpenAI Codex’s context management. However, when managing large, complex codebases and production operational incident records without leaking tokens or sacrificing security and precision, major engineering bottlenecks remain.
MEMORY.md
kiro.dev/docs/memory
.brain
/learn
Initially, I tried integrating Obsidian—the popular local Markdown knowledge base (Vault)—with various community AI plugins to supply knowledge to agents. But as autonomous development loops matured, I ran into severe limitations regarding desktop GUI dependencies, poor exact-match search, and token consumption.
To solve this fundamentally, I overhauled my long-term memory architecture (Agent Memory Hub) into an autonomous hybrid stack: LanceDB (embedded vector DB) + SQLite FTS5 (full-text lexical search) + Recursive CTEs (WikiLink graph traversal) + Local Cross-Encoder (ONNX INT8 reranker) + Markdown SSOT .
In this article, I analyze the current state of agent memory, break down the four real-world walls encountered with pure Obsidian integration, detail the Gen 3 architecture design, and share measurable performance benchmarks from day-to-day operations.
Modern coding agents are exceptionally capable. Given clear prompts, they run commands, edit files, and pass tests autonomously.
However, agents are fundamentally isolated per session . As a result, critical knowledge fails to carry across sessions:
Pasting these rules repeatedly into prompts wastes valuable context window tokens and degrades reasoning precision. A session-crossing long-term memory hub capable of pinpoint retrieval on demand is indispensable.
Coding agents in active production are rapidly introducing persistence mechanisms:
The auto-memory capabilities in Claude Code ( MEMORY.md ) , Kiro ( kiro.dev/docs/memory/ ) , and Google Antigravity ( .brain & /learn ) represent huge steps forward. In Antigravity, each conversation gets a dedicated Brain directory storing execution plans, artifacts, and full thought transcripts ( transcript.jsonl ), while /learn extracts behavioral lessons into persistent skills.
kiro.dev/docs/memory/
transcript.jsonl
However, once stored knowledge scales to hundreds or thousands of notes, relying solely on flat Markdown files injected into prompts or isolated per-session Brain folders creates severe friction:
A headless, hybrid local memory hub that humans can edit in standard Markdown and agents can query in milliseconds is the missing link.
I initially built around Obsidian, the gold standard of local Markdown knowledge management.
Obsidian connects notes via [[WikiLink]] and provides interactive visual graph views. By pairing it with community plugins like Smart Connections, Obsidian Copilot, and Local REST API, I exposed the Vault to agents.
[[WikiLink]]
However, four deal-breaking bottlenecks emerged in automated workflows:
Obsidian and key REST plugins ( obsidian-local-rest-api ) explicitly require the desktop application to be running on macOS/Windows. While headless container workarounds exist to access raw Vault files, running full Obsidian plugin indexing and UI integrations headlessly is virtually impossible. In terminal CLI environments, CI/CD pipelines, Docker containers, and headless VMs where autonomous agents thrive, keeping a desktop GUI app running constantly is fragile and impractical.
obsidian-local-rest-api
Most Obsidian AI plugins rely heavily on embedding-based semantic vector search. While effective for conceptual similarity, embeddings struggle with exact matches for error codes, library versions, and short identifiers . Searching for specific strings like ERR_CERT_AUTHORITY_INVALID , exact package names, or two-letter keywords like S3 or Go frequently returned irrelevant notes due to vector approximation.
ERR_CERT_AUTHORITY_INVALID
S3
Go
Smart Connections officially notes that “semantic queries do not behave like plain text search” —meaning exact keyword matches can be missed if semantic similarity scores are low. Hybrid lexical search (such as BM25) is mandatory for software engineering accuracy.
Obsidian’s graph view is designed for human visual inspection, lacking low-latency, programmatic APIs for recursive multi-hop exploration. Without custom parsing, agents cannot perform millisecond-level dynamic GraphRAG—such as automatically pulling 1-2 hop linked incident records related to an ADR into the prompt context.
Standard RAG plugins dump top vector search chunks directly into prompts without reranking. Unfiltered chunks introduce noise, induce hallucinations, and burn context tokens. Without a local Cross-Encoder reranker , precision suffers.
To resolve these challenges, I built Gen 3: an autonomous hybrid architecture keeping Obsidian-compatible Markdown as the Single Source of Truth (SSOT), while offloading search and graph traversal to a headless local engine .
Key components of this architecture:
The canonical source of truth remains plain Markdown files stored in a Git-tracked Vault. Each note features YAML frontmatter and [[WikiLink]] syntax, ensuring full compatibility with Obsidian, VS Code, and web git viewers.
Dense semantic vector search is powered by LanceDB: - Completely embedded—no standalone background server daemon required. - On-disk columnar format processes tens of thousands of vectors in single-digit milliseconds. - Employs lightweight multilingual embedding models for accurate technical indexing.
To compensate for vector search shortcomings, SQLite FTS5 runs in parallel. Using a trigram tokenizer, it reliably catches exact error codes, library symbols, and short abbreviations with BM25 scoring.
trigram
WikiLinks are indexed into an SQLite link table. When an agent queries a note, recursive Common Table Expressions (CTEs) instantly pull 1-2 hop connected nodes in under 5 milliseconds.
Search rankings from LanceDB and SQLite are merged via Reciprocal Rank Fusion (RRF, k = 60 k=60 ). A quantized ONNX INT8 Cross-Encoder reranks candidate chunks, filtering down to the top 3-5 most pertinent chunks in under 200ms on Apple Silicon.
Operating this hybrid stack for several weeks delivered immediate qualitative and quantitative leaps:
Obsidian remains the gold standard for human thought organization. However, empowering autonomous AI coding agents requires headless operation, hybrid lexical/semantic retrieval, dynamic GraphRAG traversal, and local reranking .
By decoupling the human interface (Markdown Vault SSOT) from the agent search engine (LanceDB + SQLite FastMCP), engineers achieve the ideal harmony of human-friendly notes and blazing-fast autonomous AI recall.
![[2024-2026 History] Evolution of AI Coding Agents & Future Outlook: From Copilot to Claude Code, Antigravity, and Kiro — The Paradigm Shift from Chat to Autonomous Cowork](/thumbnails/Evolution-of-AI-Coding-Agents-from-Chat-to-Cowork.en.webp)
![[Python] How to extract Excel VBA code with Python and code in VSCode (Alternative if XVBA doesn't work)](/thumbnails/How-to-extract-Excel-VBA-code-with-Python-and-code-in-VSCode.en.webp)
