Expose only completed files as RAG evidence
File upload and search availability are different states. Only a search copy that finished parsing, embedding, and storage—and has processed = TRUE—may become evidence for a user answer.
File upload and search availability are different states. Only a search copy that finished parsing, embedding, and storage—and has processed = TRUE—may become evidence for a user answer.
Query boundary
Both vector similarity and trigram search must check the completion predicate inside the file join. Latest-version selection must not mix a processing new edition with a ready old one. Tables whose document-chunk or QA-pair search-copy link is nullable need partial indexes that match their actual OR paths.
... JOIN vector_files f ON f.id = e.vector_file_id
WHERE f.store_id = $1
AND f.processed = TRUE;
Operations and UX
Show processing and retryability in Admin instead of hiding failed files as an empty result. Preserve owner scope, source, model, and dimension so re-embedding and reconciliation are possible. A completion partial index can shrink the hot path, but verify it with staging EXPLAIN (ANALYZE, BUFFERS) before production.
Related course: AI providers and RAG boundaries
Terms in this content
More in data
All in this category →Related posts
pgvector and RAG
The trend of combining large language model responses with external knowledge has settled under the name RAG (Retrieval-Augmented Generation). At its core sits vector similarity search, and PostgreSQL's pgvector extensi…
RAG and Search — Chunking · Embedding · Reranking · Hybrid
RAG (Retrieval-Augmented Generation) is an approach to supplement an LLM's limited training cutoff and memory limits with external knowledge. By separating retrieval and generation, you can handle new material without r…