Document storage and embeddings

Document uploads need object storage, and AI document search needs an embedding provider. Both are platform-level settings applied by whoever operates the UnitOps deployment.

Day-to-day document management is in Documents.

How the pipeline works

1. Upload - an admin creates a document and uploads a file to cloud storage (Google Cloud Storage or Amazon S3).
2. Index - the API downloads the file, extracts the text (pdf, docx, txt, csv), and splits it into chunks of about 500 words with 50 words of overlap.
3. Store - each chunk is embedded and stored in Postgres with pgvector.
4. Search - a query is embedded and compared by cosine similarity; the top matches are returned with a score.
5. Generate - in chat, the retrieved chunks are added to the model prompt so answers cite your own content.

1. Configure storage

Set DOCUMENT_STORAGE_CONFIG as a JSON string in the API environment.

Google Cloud Storage:

{
  "provider": "gcs",
  "bucket": "unitops-customer-documents",
  "prefix": "documents",
  "gcs": {
    "use_workload_identity": true
  }
}

Amazon S3:

{
  "provider": "s3",
  "bucket": "unitops-customer-documents",
  "prefix": "documents",
  "s3": {
    "access_key_id": "AKIA...",
    "secret_access_key": "secret",
    "region": "us-east-1",
    "endpoint_url": "https://s3.amazonaws.com"
  }
}

Prefer workload identity or an IAM role over long-lived keys.

2. Configure the embedding provider

Set EMBEDDING_PROVIDER_CONFIG as a JSON string. Supported providers: openai, google, voyage.

{
  "default_provider": "openai",
  "default_model": null,
  "openai": {
    "api_key": "sk-..."
  }
}

{
  "default_provider": "google",
  "google": {
    "api_key": "AIza..."
  }
}

{
  "default_provider": "voyage",
  "voyage": {
    "api_key": "pa-..."
  }
}

Provider	Models
OpenAI	text-embedding-3-small (1536d), text-embedding-3-large (3072d)
Google	text-embedding-005 (768d), text-embedding-004, text-multilingual-embedding-002
Voyage	voyage-3 (1024d), voyage-3-lite (512d), voyage-code-3 (1024d)

Set default_model to override the provider default. Changing the model changes the vector dimension, so re-index existing documents afterwards.

Restart the API after changing either variable.

3. Verify

1. Open Admin -> Documents and upload a small PDF.
2. Click Index; the response should report a chunk count.
3. Expand AI Document Search and ask a question the document answers.

If the panel says the embedding provider is not configured, EMBEDDING_PROVIDER_CONFIG is missing or the API was not restarted.

API reference

Endpoint	Purpose
GET /api/v1/documents/upload-constraints	File type and size limits
GET /api/v1/documents	List documents
POST /api/v1/documents	Create the document record
POST /api/v1/documents/{id}/upload	Upload the file
GET /api/v1/documents/{id}/download	Signed download URL
POST /api/v1/documents/{id}/index	Index for RAG (optional ?model=)
DELETE /api/v1/documents/{id}/index	Remove the index
POST /api/v1/documents/search	Search indexed chunks
GET /api/v1/documents/embedding-config	Report the active provider

curl -X POST https://api.example.com/api/v1/documents/search \
  -H "Content-Type: application/json" \
  -d '{"query":"How do I reset the fryer timer?","top_k":5,"document_ids":null,"model":null}'

Security

Documents and embeddings carry the organization identifier and are protected by Postgres row-level security, so retrieval in the UI, in chat, and through the API is always limited to the caller's own tenant.