Knowledge base & RAG#

Upload documents to an agent and it can answer from them instead of from the model's general knowledge. Files go browser → bucket via a signed URL, then a server-side pipeline extracts text, splits it into overlapping chunks, embeds each chunk, and writes the vectors to Postgres (pgvector). At chat time the agent's File Retrieval Strategy decides whether those files are pasted into context wholesale, searched semantically on demand, or mounted into a sandboxed shell.

What the Knowledge tab controls#

The agent editor's Knowledge tab renders one card, Tools & Files:

• Web Fetch — adds the builtin:web_fetch tool so the agent can pull a URL mid-conversation.
• Source URLs — up to 5 URLs listed inline under Web Fetch. These are pre-fetched into context on every turn by whichever retrieval strategy is active, independently of the Web Fetch toggle.
• File search — adds the builtin:file_search tool. Both switches are staged locally and only persist when you press Save Tools.
• Reference Files — an upload button plus a drag-and-drop zone, then the list of attached files with per-file download and delete.

The File Retrieval Strategy picker is not on this tab — it lives in the agent editor's right sidebar, alongside a second copy of Tools & Files.

File retrieval strategies#

Every agent has a retrievalStrategy of direct (the default), rag, or bash. It changes which files reach the model and which tools the model gets.

Source URLs behave identically under all three: the first 5 are fetched and appended while the running total stays within 30,000 characters.

Direct#

DirectRetriever reads every attached file in full, sorts them smallest first, and appends each one whose length still fits the 18,000-character file budget. Anything that doesn't fit is skipped and sets an overflow flag. When nothing overflowed and the agent has at least one file, buildAgentContext strips file_search from the toolkit entirely — the model already has the documents, so searching them would be redundant. If something did overflow, file_search survives (assuming the toggle is on) as an escape hatch for the files that were left out.

RAG#

RagRetriever preloads nothing. It injects its own file_search tool whose description says it uses semantic search, and whose limit parameter defaults to 8 matches. It is injected when the agent has files and the File search switch is on, and it takes precedence over the built-in file_search on name collision — so under rag the tool the model sees is the retriever's semantic one. The switch used to be ignored here: the retriever injected on "does the agent have files" alone and overrode the toolkit by name, so turning File search off appeared to do nothing under RAG. Source URLs are unaffected by the switch — they are context, not a file tool.

Bash#

BashRetriever downloads each file, truncates it at 200,000 characters, and writes it to /home/user/project/ in a just-bash in-memory sandbox (filenames sanitised to [a-zA-Z0-9._-], leading dots stripped, .. collapsed). It then injects a bash tool and a context hint listing the available paths. The sandbox advertises grep, rg, awk, sed, head, tail, cat, sort, uniq, wc, cut, tr, jq, xan, yq, find, ls, and diff, with no network and no host filesystem access. Writes persist across calls within one conversation turn; the sandbox is discarded on dispose().

Per-call limits: commands over 4,000 characters are truncated, stdout is clipped to 8,000 characters and stderr to 500, and a command that runs longer than 10 seconds is rejected. Execution limits also cap call depth (50), command count (500), and loop/awk/sed iterations (5,000 each).

The indexing pipeline#

Vectors are sharded across four tables by dimension so several providers can coexist in one deployment: embeddings_1536 (OpenAI text-embedding-3-small), embeddings_384 (e5 / Google Cloud MaaS), embeddings_1024 (BGE, Arctic Embed), and embeddings (768-dim, the Ollama nomic-embed-text default). Each has its own HNSW index using vector_cosine_ops.

Supported file types#

The API's allow-list and the picker's accept attribute are close but not identical, and acceptance at upload does not guarantee extraction.

application/xml is on the API allow-list but missing from the file picker's accept list, so XML can be uploaded via the API but not chosen in the dialog. application/octet-stream is also accepted, which is how files whose MIME the browser can't guess still get through — they end up on the raw-decode fallback path.

The size cap is 10 MB (MAX_FILE_UPLOAD_BYTES), enforced in the browser and again by the upload-URL route.

How retrieval works at chat time#

When file_search runs, searchAgentFileChunks calls retrieveContext, which:

1. Embeds the query with the tenant's resolved embedder and picks the embedding table by the query vector's dimension.
2. Runs a cosine search (ORDER BY cosine distance, LIMIT topK) over that one table, joined to files on agentId and scoped by tenantId. Reported similarity is 1 - distance.
3. If — and only if — that returns zero rows, falls back to Postgres full-text search: plainto_tsquery('english', …) ranked by ts_rank, run across all four tables and merged.

Matches come back to the model as File: <name> / Snippet: blocks.

Re-embedding after a provider change#

Embeddings from different models are not comparable, and here the mismatch is structural: the query vector's dimension chooses the table. Switch a workspace from OpenAI (1536-dim) to an Ollama config (768-dim) and every query searches embeddings while all the file chunks still sit in embeddings_1536. Vector search returns nothing, the keyword fallback fires on every question, and the agent quietly degrades to plain full-text matching.

The Knowledge tab detects this on load. GET /api/agents/{id}/files resolves the provider embedding would actually use — the same way ingestion does, honouring an embed task assignment and falling back to the platform provider — and returns it as currentEmbeddingProvider. If any indexed file's embeddingProvider differs from it, the tab shows an amber banner — "LLM provider changed — file embeddings are stale and may return poor results" — with a Re-embed files button.

That button calls POST /api/agents/{id}/reembed, which re-runs the full ingest for every file currently at status indexed and reports { reembedded, total, errors }.

Troubleshooting#

Every upload fails with "Failed to fetch"#

The browser PUTs directly to the bucket, so the bucket must answer the CORS preflight — the application cannot supply those headers on its own responses. A bucket with no policy fails every upload with an opaque Failed to fetch and nothing in the UI explains why.

Apply it with scripts/apply-bucket-cors.sh, which grants GET, HEAD, and PUT for the origins you pass:

BUCKET=your-bucket ORIGINS="https://your-app.example,https://www.your-app.example" \
  scripts/apply-bucket-cors.sh

Run it whenever a bucket is created, renamed, or migrated. It is deliberately not automated: the deploy service account holds roles/storage.objectAdmin, which does not include storage.buckets.update. The deploy workflow does verify the policy on every deploy and fails loudly if the preflight isn't answered. See docs/deployment.md for the full policy shape.

A file is stuck in failed#

Read processingError from GET /api/agents/{id}/files (it's truncated to 500 characters). The messages the pipeline actually produces:

The first five come back as 422; a thrown exception returns 500. Both mark the row failed. Fix the cause, then delete and re-upload the file — POST /files/ingest upserts by (agentId, fileKey) and resets the row to pending.

The agent quotes a document that no longer exists#

Deleting from the Knowledge tab removes the embeddings, the files rows, the key on the agent, and the stored object in one transaction. If a file was removed some other way (a direct bucket delete, for instance) its chunks survive, because retrieval joins on files.agentId with no status or existence filter. Re-attach and delete it through the UI to clear it properly.

Related#