Kwizmo Local API β€” Developer Manual

Server base URL (example): https://localhost:5000  Β·  27 endpoints across accounts, domains, files, website crawling, search, AI answers, and administration.

Overview

Kwizmo Local is a self-hosted RAG (retrieval-augmented generation) knowledge base. Documents and crawled web pages are chunked, embedded, and stored per-domain in Qdrant. Once populated, a domain can be searched directly (/search) or used to generate an AI-written answer grounded in the retrieved context (/query_answer), using either OpenAI or Anthropic as the model provider.

This manual documents every HTTP endpoint exposed by app.py, in the order you're likely to need them: register β†’ login β†’ create a domain by uploading a file or crawling a site β†’ search/ask β†’ manage users and review the audit log.

Authentication & session model

Authentication is a two-step flow: register once, then log in to obtain a session API key.

  1. Create an account with POST /register.
  2. Authenticate with POST /login to receive an api_key.
  3. Send that key on every protected request using the header:
kwizmo_login_header: <api_key>

βœ… Multiple concurrent sessions are supported

Each successful POST /login creates its own independent session and returns its own api_key. Multiple users β€” or the same user from multiple browsers/tabs/devices β€” can be logged in at the same time; logging in does not invalidate any other active session. POST /logout ends only the session tied to the specific api_key used to call it, leaving every other session untouched. There is no fixed cap on the number of concurrent sessions.

Roles & activation

  • The first user ever registered automatically becomes headadmin with status: 1 (active) β€” no approval needed.
  • Every subsequent registration is created with role admin and status: 0 (inactive) β€” a headadmin must activate the account (via POST /saveUser) before that user can log in or use any protected endpoint.
  • An inactive user attempting most actions receives a quiet denial (often an empty string body or {"status":"error","message":"Inactive user"}, depending on the endpoint) rather than an HTTP error β€” check the response body, not just the status code.
  • headadmin implicitly has access to every domain, regardless of its own domains field.
Domain naming β€” read this before integrating

Two different conventions are used across endpoints

This is the single most common integration mistake. Endpoints fall into two groups:

  • Auto-prefixing endpoints β€” /ingest and /crawl take a short, unprefixed domain name (e.g. docs) and silently prefix it with the logged-in username (e.g. alice_docs) if it isn't already prefixed. These also auto-create the domain/collection on first use.
  • Exact-name endpoints β€” /search, /files, /collection_stats, /delete_collection, and /delete_file / /delete_files_bulk expect the full, already-prefixed domain name exactly as returned by GET /domains (e.g. alice_docs, not docs). They do not add a prefix for you.

When in doubt, call GET /domains first to get the exact names to use, or use GET /domain_exists before creating something new.

Account & session
POST /register No auth required

Creates a new user account.

Request (JSON)
{
  "email": "alice@example.com",
  "username": "alice",
  "password": "secret123"
}
Response (success)
{
  "status": "ok",
  "message": "User registered successfully",
  "loc": "..."
}
Response (username or email already taken)
HTTP 400
{ "detail": "User could not be registered" }
Notes
  • The very first account ever registered becomes headadmin, active immediately.
  • All later accounts are created as admin, status: 0 (inactive) until a headadmin activates them.
  • Recorded in the audit log as register (status ok or denied, with a reason such as "username taken" or "email taken").
POST /login No auth required

Authenticates a user and issues a new session api_key. Each login creates its own independent session β€” it does not invalidate any other session, including a previous session for the same user (e.g. logged in on another device) or any other user's active session. See Authentication & session model.

Request (JSON)
{
  "username": "alice",
  "password": "secret123"
}
Response (headadmin)
{
  "status": "ok",
  "api_key": "0123abcd...",
  "users": [ /* full users.json content */ ],
  "role": "headadmin"
}
Response (regular user)
{
  "status": "ok",
  "api_key": "abcd0123...",
  "users": [ /* only this user's own record */ ],
  "role": "admin"
}
Response (inactive user)
""  (empty string body)
Response (wrong credentials)
{ "status": "error", "message": "Invalid credentials" }
Notes
  • headadmin receives every user's record in users; other roles receive an array containing only their own record (used by the dashboard's Users tab to show a self-service password-change view β€” see /saveUser).
  • Recorded in the audit log as login (ok / denied, with reason "inactive user" or "invalid credentials").
GET /whoami Requires api_key

Verifies that a previously-issued api_key is still valid and returns the associated username/role β€” without requiring the password again. Used by the dashboard to restore the logged-in view after a browser refresh (F5 / Ctrl+F5) instead of dropping back to the login screen.

Request
GET /whoami
kwizmo_login_header: <api_key>
Response (valid session, regular user)
{
  "status": "ok",
  "username": "alice",
  "role": "admin",
  "users": [ /* only this user's own record */ ]
}
Response (session no longer valid)
{ "status": "error", "message": "Session no longer valid" }
POST /logout Requires api_key

Invalidates only the calling session's api_key server-side, so a stored key (e.g. kept in a browser's sessionStorage) can no longer be used afterward. Other sessions β€” including other active logins for the same user, or any other user's session β€” are unaffected.

Request
POST /logout
kwizmo_login_header: <api_key>
Response
{ "status": "ok" }
Notes
  • Recorded in the audit log as logout.
Dashboard & manual
GET / No auth required

Serves the dashboard's dashboard.html as-is. This is the same web UI a browser sees when visiting the server's root URL; it authenticates separately via /login from within the page.

GET /manual No auth required

Serves this developer manual (kwizmo_api_developer_manual_html.html) directly from disk. Linked from the dashboard's "πŸ“˜ API Developer Manual" button.

Domains
GET /domains Requires api_key

Returns domains (Qdrant collections) visible to the current user. headadmin sees every collection on the server; other roles see only the domains listed in their own domains field.

Request
GET /domains
kwizmo_login_header: <api_key>
Response
{ "domains": ["alice_docs", "bob_files"] }
GET /domain_exists Requires api_key

Checks whether a domain already exists, applying the same <username>_<domain> prefixing used by /ingest and /crawl. Since those two endpoints silently auto-create a domain on first use, this lets a client (such as the dashboard) confirm with the user before that happens, rather than a new domain appearing unannounced.

Request
GET /domain_exists?domain=ajpes
kwizmo_login_header: <api_key>
Response
{
  "status": "ok",
  "domain": "alice_ajpes",
  "exists": false
}
GET /collection_stats Requires api_key

Returns raw Qdrant collection metadata for a domain: vector count, indexing status, distance metric, HNSW/optimizer configuration, and so on. Powers the dashboard's "Vector DB Config" tab.

Exact-name endpoint β€” pass the full domain name as returned by /domains (e.g. alice_docs), not the short name. See Domain naming.

Request
GET /collection_stats?domain=alice_docs
kwizmo_login_header: <api_key>
Response
{
  "status": "green",
  "vectors_count": 1204,
  "points_count": 1204,
  "config": { "params": { "vectors": { "size": 768, "distance": "Cosine" } }, ... },
  ...
}
POST /delete_collection Owner only

Permanently deletes an entire domain (the whole Qdrant collection, including all indexed files/pages within it). Irreversible.

Request (form)
domain=alice_docs
Response
{ "status": "ok", "domain": "alice_docs" }
Notes
  • The caller must own the domain (its name must start with <their_username>_); otherwise the request is silently denied (empty response).
  • Recorded in the audit log as delete_collection.
Files
POST /ingest Requires api_key

Uploads a file to be chunked, embedded, and stored in a domain. Chunking strategy, chunk size, and overlap are chosen automatically based on file type β€” structured formats (code, markdown) are split on natural boundaries (functions, headers) rather than raw word count wherever possible. See Chunking strategy by file type below for full details.

Auto-prefixing & auto-creating endpoint β€” domain can be a short name; it will be prefixed with the logged-in username if not already prefixed, and the domain is created automatically if it doesn't exist yet. See /domain_exists to confirm first.

Supported file types

.pdf, .docx, .doc (best-effort β€” see Chunking strategy by file type), .txt, .md/.markdown, .csv/.tsv, and source code (.py .js .jsx .ts .tsx .java .c .h .cpp .hpp .cs .go .rb .php .rs .swift .kt .sql .sh). Anything else returns "Unsupported file type".

Request (multipart/form-data)
file: (binary file)
domain: docs
Example curl
curl -X POST "https://localhost:5000/ingest" \
  -H "kwizmo_login_header: <api_key>" \
  -F "domain=docs" \
  -F "file=@manual.pdf"
Response (success)
{
  "status": "ok",
  "domain": "alice_docs",
  "file": "manual.pdf",
  "chunks_indexed": 42
}
Response (errors)
{"status":"error","message":"Empty file"}
{"status":"error","message":"File produced no chunks"}
{"status":"error","message":"Unsupported file type"}
{"status":"error","message":"Could not extract readable text from this .doc file. ..."}
{"status":"error","message":"Failed to save to the vector database: ..."}

⏱️ Large documents and Qdrant write timeouts

Indexing writes chunks to Qdrant in batches of QDRANT_UPSERT_BATCH_SIZE points per request (default 64), rather than one single request for the whole document, so a large file (hundreds of chunks) doesn't produce one oversized, slow request. The Qdrant client's own request timeout is also raised beyond the underlying library's very short 5-second default, to QDRANT_TIMEOUT_SECONDS (default 60s). Both are configurable via environment variables if you still see timeouts on very large documents or a slow/loaded Qdrant instance. This applies the same way to documents indexed via /crawl.

Chunk metadata by file type
TypeChunk metadata recorded
PDFsource, page (1-indexed), chunk_index
DOCXsource, paragraph (1-indexed), chunk_index
DOC (legacy)source, paragraph (1-indexed, based on extracted text runs β€” not identical to the original paragraph structure), chunk_index
TXTsource, paragraph (1-indexed, split on newlines), chunk_index
Markdownsource, heading (nearest enclosing header text), chunk_index
Source codesource, language, function_name (function/class/method name when identifiable), chunk_index
CSV/TSVsource, chunk_index (header row repeated in every chunk)
Notes
  • Recorded in the audit log as ingest_file, including chunks_indexed on success.
GET /files Requires api_key

Lists the distinct files/URLs currently indexed in a domain, sorted case-insensitively.

Exact-name endpoint β€” pass the full domain name (e.g. alice_docs). See Domain naming.

Request
GET /files?domain=alice_docs
kwizmo_login_header: <api_key>
Response
{ "files": ["manual.pdf", "https://example.com/pricing", "notes.txt"] }
Notes
  • Entries can be plain filenames (from /ingest) or full URLs (from /crawl) β€” both are stored in the same file payload field.
  • The domain is auto-created if it doesn't exist yet (so this returns an empty list rather than an error for a brand-new name).
POST /delete_file Owner only

Deletes every indexed chunk belonging to one file or URL from a domain.

Request (form)
domain=alice_docs
filename=manual.pdf
Response
{
  "status": "ok",
  "domain": "alice_docs",
  "file": "manual.pdf",
  "deleted_points": 12
}
Notes
  • Recorded in the audit log as delete_file.
POST /delete_files_bulk Owner only

Deletes every indexed chunk belonging to any number of files/URLs in one call β€” the collection is scanned once regardless of batch size, which matters for domains with thousands of documents. Powers the dashboard's "delete filtered files" action.

Request (form)
domain=alice_docs
filenames=["manual.pdf","draft-v1.pdf","draft-v2.pdf"]

"filenames" is a JSON-encoded array of filenames/URLs, exactly as they appear in GET /files.

Response
{
  "status": "ok",
  "domain": "alice_docs",
  "deleted_files": ["draft-v1.pdf", "draft-v2.pdf", "manual.pdf"],
  "deleted_points": 47,
  "not_found": []
}
Notes
  • not_found lists any requested filenames that didn't match a point in the domain (e.g. already deleted).
  • Recorded in the audit log as delete_files_bulk, including the list of files actually deleted.
Website crawling

How crawling fits together

Typical flow: optionally call /crawl_discover to preview which pages (and linked documents β€” see below) a site contains, then call /crawl to start fetching and indexing them into a domain in the background, polling /crawl_progress to follow along and get the final results. Every /crawl call β€” even a one-off with no recurring schedule β€” creates a persisted crawl job that can be listed (/crawl_jobs), re-run on demand (/run_crawl_job_now), or removed (/delete_crawl_job). If a crawl or discovery fails with a network-level error, use /test_connectivity to diagnose exactly where it's failing (DNS, TCP connect, or the HTTP/TLS layer) before assuming it's an application bug.

POST /test_connectivity Requires api_key

Diagnoses network reachability to a URL, staged the same way curl -v would report it: DNS resolution, a raw TCP connect, then the full HTTP request (including the TLS handshake for https:// URLs) β€” each timed independently, plus an informational robots.txt check. Answers "why can't the crawler reach this site?" (DNS failure vs. network/firewall block vs. a slow-but-reachable server vs. the site simply returning an HTTP error) without needing to shell into the container to run curl/ping/nslookup by hand.

Request (JSON)
{ "url": "https://example.com" }
Response β€” reachable
{
  "status": "ok",
  "result": {
    "url": "https://example.com",
    "hostname": "example.com",
    "port": 443,
    "dns": { "status": "ok", "duration_ms": 12.4, "resolved_ip": "93.184.216.34", "error": null },
    "tcp_connect": { "status": "ok", "duration_ms": 45.1, "error": null },
    "http_request": { "status": "ok", "duration_ms": 210.7, "status_code": 200, "content_type": "text/html; charset=UTF-8", "content_length": 1256, "error": null },
    "robots_txt": { "status": "checked", "allowed": true, "error": null },
    "overall": "reachable",
    "summary": "Reachable. DNS: 12.4ms, TCP connect: 45.1ms, full HTTP request: 210.7ms. HTTP 200."
  }
}
Response β€” connection timeout (the most common real-world failure)
{
  "status": "ok",
  "result": {
    "url": "https://www.example-slow-site.com",
    "hostname": "www.example-slow-site.com",
    "port": 443,
    "dns": { "status": "ok", "duration_ms": 18.2, "resolved_ip": "203.0.113.5", "error": null },
    "tcp_connect": { "status": "timeout", "duration_ms": 15003.4, "error": "Connection to www.example-slow-site.com:443 timed out after 15s" },
    "http_request": { "status": "not_attempted", "duration_ms": null, "status_code": null, "content_type": null, "content_length": null, "error": null },
    "robots_txt": { "status": "not_attempted", "allowed": null, "error": null },
    "overall": "connect_timeout",
    "summary": "DNS resolved 'www.example-slow-site.com' to 203.0.113.5 in 18.2ms, but connecting to www.example-slow-site.com:443 timed out after 15s. This points to a network path problem (firewall, routing, or the remote host/service not accepting connections) rather than a DNS or application-level issue."
  }
}
overall values
ValueMeaning
reachableDNS, TCP connect, and the HTTP request all succeeded with a 2xx/3xx response.
http_errorConnected fine, but the server returned a non-2xx/3xx HTTP status β€” not a network problem.
dns_failureThe hostname could not be resolved at all β€” likely a typo, or this server has no working DNS.
connect_timeoutDNS resolved fine, but the TCP connection to the resolved IP timed out β€” a network path/firewall/routing problem, or the remote service isn't accepting connections on that port.
connect_failedDNS resolved fine, but the TCP connection failed immediately (e.g. connection refused) rather than timing out.
tls_errorTCP connected, but the TLS/SSL handshake failed (e.g. certificate problem).
http_timeoutTCP connected, but the full HTTP request (including TLS) didn't complete in time β€” the server is slow/unresponsive at the HTTP layer specifically.
http_failedTCP connected, but the HTTP request failed for some other reason.
invalid_urlurl doesn't start with http:///https://, or no hostname could be parsed from it.
Notes
  • This is a synchronous call, not a background job like /crawl β€” a single URL check is fast enough not to need progress polling. Worst case it takes roughly up to CRAWL_TIMEOUT_SECONDS (default 15s) per stage attempted.
  • The robots_txt check is informational only, run after the reachability stages regardless of their outcome (unless the URL is invalid) β€” it doesn't affect overall.
  • Uses the same CRAWL_TIMEOUT_SECONDS environment variable as page/document fetching for consistency with actual crawl behavior.
  • Recorded in the audit log as test_connectivity, including the resulting overall status.
POST /crawl_discover Requires api_key

Crawls outward from a starting URL (breadth-first, same site only) and finds every page URL reachable from it, without indexing anything. Used to let the user pick which pages to include before committing to a full crawl.

πŸ“„ Linked documents are included too

If a crawled page links to a file the server knows how to index β€” PDF, DOCX, Markdown, CSV/TSV, or source code (see Chunking strategy by file type) β€” that file's URL is included directly in the results, without being fetched as an HTML page or followed for further links. It's treated as a leaf document, the same as if it had been uploaded via /ingest. Links to files with unrecognized extensions, or on a different domain than the page being crawled, are not included.

A document is normally recognized from its URL's file extension. If a linked URL has no recognizable extension (e.g. a query-string download link), it's still detected as a document if fetching it returns a recognized document Content-Type instead of HTML β€” so links like /download?file=report.pdf are still picked up correctly.

⏱️ Runs in the background β€” this returns immediately

This endpoint does not wait for discovery to finish. It starts the site scan on a background thread and returns right away with a discover_id. Poll GET /discover_progress?discover_id=<id> to see live progress (which page is currently being checked, how many pages found so far) and to know when it's finished and retrieve the final page list. This matters for larger sites, which can take a while to fully scan β€” the dashboard uses this to show a live "Scanning site… 12 pages found so far" indicator instead of one static "please wait" message.

Request (JSON)
FieldTypeDefaultNotes
urlstringβ€”Required. The page to start discovery from.
max_pagesint5000Stop discovery once this many pages/documents have been found. Hard-capped server-side at MAX_ALLOWED_CRAWL_PAGES (default 5000) regardless of what's requested.
max_depthint2How many link-hops away from url to follow. Hard-capped server-side at MAX_ALLOWED_CRAWL_DEPTH (default 10).
{ "url": "https://example.com/docs", "max_pages": 200, "max_depth": 3 }
Response β€” discovery started
{
  "status": "ok",
  "discover_id": "3b8e91a0-...",
  "start_url": "https://example.com/docs",
  "max_pages": 200,
  "max_depth": 3
}

The returned max_pages/max_depth reflect the values actually applied (after server-side clamping), which may be lower than what was requested. Use discover_id with /discover_progress to follow along and get the final page list β€” this response does not include the page list itself. The final list from /discover_progress mixes HTML page URLs and document URLs together; a client can distinguish them by checking the URL's file extension against the list in Chunking strategy by file type, the same way the dashboard does.

Response (immediate validation errors)
{"status":"error","message":"URL must start with http:// or https://"}
{"status":"error","message":"This URL is disallowed by the site's robots.txt"}
{"status":"error","message":"max_pages must be at least 1"}
{"status":"error","message":"max_depth must be 0 or greater"}

⚠️ If a site has more pages than max_pages, only the first ones found are returned

Discovery stops as soon as max_pages is reached β€” it does not keep scanning to find the true total. /discover_progress's response includes "hit_page_limit": true whenever this happens, so a client can tell the user the results are likely incomplete rather than silently implying the whole site was found. Raise max_pages (up to MAX_ALLOWED_CRAWL_PAGES) and re-run discovery to find more.

Notes
  • Default limits are 5000 pages and a link depth of 2 from the start URL β€” both are request parameters, configurable from the dashboard's crawl UI as well as directly via the API, not fixed. The maximum either can be set to (MAX_ALLOWED_CRAWL_PAGES / MAX_ALLOWED_CRAWL_DEPTH) is itself configurable via environment variables (both default to matching the request defaults), to prevent an accidental request from launching an extremely long-running crawl against a huge site.
  • robots.txt is respected β€” disallowed pages and documents are never queued or returned. If a site's robots.txt itself can't be retrieved cleanly (missing, errors out, times out, or returns anything other than 200 OK), that fails open β€” crawling proceeds as if no robots.txt exists β€” rather than being treated as "disallow everything," which previously caused otherwise-normal sites to fail discovery/crawling with no clear error. Fetching robots.txt uses its own short, non-retried timeout (default 5s, configurable via ROBOTS_TXT_TIMEOUT_SECONDS) rather than the same timeout/retry settings as page content β€” this keeps otherwise-fast endpoints like this one and /test_connectivity from being blocked for a long time by a single slow or unreachable robots.txt. The result is cached per site after the first check.
  • A short delay is applied between page fetches during discovery, to avoid hammering the target site.
  • Each page fetch (here and in /crawl) automatically retries once after a connection-level failure (timeout, DNS error, connection refused/reset) before giving up β€” HTTP error responses like 404/500 are never retried, since those are a definitive answer from the server rather than a network glitch. Both the connect/read timeout (default 15s) and retry count (default 1) are configurable via the CRAWL_TIMEOUT_SECONDS and CRAWL_FETCH_RETRIES environment variables if your network has consistently higher latency.
GET /discover_progress Requires api_key

Polled while a POST /crawl_discover call is running in the background. Returns the page currently being checked and how many pages have been found so far. Once done is true, either the final pages list is populated, or status is "error" if the site couldn't be reached at all.

Request
GET /discover_progress?discover_id=3b8e91a0-...
kwizmo_login_header: <api_key>
Response β€” in progress
{
  "status": "ok",
  "done": false,
  "start_url": "https://example.com/docs",
  "current_url": "https://example.com/docs/getting-started",
  "found": 4,
  "pages": [],
  "max_pages": 50,
  "hit_page_limit": false
}
Response β€” finished, under the page limit
{
  "status": "ok",
  "done": true,
  "start_url": "https://example.com/docs",
  "current_url": null,
  "found": 12,
  "pages": [
    "https://example.com/docs",
    "https://example.com/docs/getting-started",
    "https://example.com/docs/faq"
  ],
  "max_pages": 50,
  "hit_page_limit": false
}
Response β€” finished, page limit reached (site may have more pages)
{
  "status": "ok",
  "done": true,
  "start_url": "https://example.com/docs",
  "current_url": null,
  "found": 5000,
  "pages": [ /* 5000 URLs */ ],
  "max_pages": 5000,
  "hit_page_limit": true
}

When hit_page_limit is true, discovery stopped because it hit max_pages, not because the whole site was exhausted β€” there may be more pages on the site that were never reached. Re-run /crawl_discover with a higher max_pages to find more.

Response β€” finished with an error (site unreachable)
{
  "status": "error",
  "message": "Could not reach or crawl the given URL: Connection to example.com timed out. (connect timeout=15)",
  "done": true
}
Response β€” unknown or expired discover_id
{ "status": "error", "message": "Unknown or expired discover_id" }
Notes
  • Progress is held in memory only (not persisted to disk), the same as /crawl_progress β€” it's meant for live polling during discovery, not as a permanent record. A finished discovery's progress entry is kept for about 30 minutes then discarded.
  • Recommended polling interval: about once per second β€” frequent enough to feel live, without hammering the server.
POST /crawl Requires api_key

Crawls either a single URL or a specific list of pages, indexes their content into a domain, and persists a crawl job. If a page/document was already indexed before (matched by URL), its old content is removed and replaced β€” the same "recrawl" semantics used for scheduled re-runs.

πŸ“„ URLs pointing to documents are indexed as documents

If a URL in url or pages points to a PDF, DOCX/DOC, Markdown, CSV/TSV, or source code file (see Chunking strategy by file type), it is downloaded and chunked using the exact same logic as an uploaded file via /ingest β€” per-page/per-paragraph/per-function chunking, not the generic web-page text chunker. Its result entry includes "is_document": true and a title equal to the file's name (e.g. "whitepaper.pdf") rather than an HTML page title. Legacy .doc files use best-effort text extraction β€” see the callout in Chunking strategy by file type.

Documents are detected two ways: primarily from the URL's own file extension, and as a fallback, from the response's actual Content-Type header when the URL itself gives no hint β€” e.g. a query-string download link like /download?file=report.pdf or an extension-less REST-style URL like /documents/482/content. This fallback also covers the case of a genuine .docx/.doc file served with an unexpected or legacy Content-Type (e.g. application/msword on a .docx file). If a URL fetched as a page doesn't return HTML and its real Content-Type doesn't match any recognized document type either, it's reported as a failed page rather than silently indexed as something incorrect. Large documents are written to Qdrant in batches (see /ingest's note on QDRANT_UPSERT_BATCH_SIZE/QDRANT_TIMEOUT_SECONDS) so a big file's page result doesn't fail with a write timeout.

⏱️ Runs in the background β€” this returns immediately

This endpoint does not wait for the crawl to finish. It starts the crawl on a background thread and returns right away with a crawl_id. Poll GET /crawl_progress?crawl_id=<id> to see live progress (which page is currently being fetched, how many are done, results as they complete) and to know when it's finished and retrieve the final result set. This matters for large crawl_all_pages jobs, which can take a while β€” the dashboard uses this to show a live "Crawling page 4 of 50…" indicator instead of one static "please wait" message.

Auto-prefixing & auto-creating endpoint β€” like /ingest, domain can be a short name and the domain is created automatically if needed.

Request (JSON)
FieldTypeDefaultNotes
domainstringβ€”Required. Short or full domain name.
urlstringβ€”Required. The page to crawl, or the seed URL if crawl_all_pages is true.
crawl_all_pagesboolfalseIf true, crawls every URL in pages instead of just url.
pagesarray of stringsnullRequired when crawl_all_pages is true β€” typically the list returned by /crawl_discover, filtered to what the user selected.
recrawl_intervalstring"none"One of "none", "daily", "weekly", "biweekly". Schedules automatic future recrawls of this job.
recrawl_hourintrandom hour 0–5Hour of day (0–23, server local time) to run the recrawl. If omitted, a random hour between midnight and 6am is chosen and used consistently for that job going forward β€” not re-randomized on every restart. Any hour can be chosen; the 0–5 default is only a suggestion for off-peak timing.
recrawl_daysarray of stringsall 7 daysWhich days of the week to recrawl on, from ["mon","tue","wed","thu","fri","sat","sun"]. Defaults to every day (weekends included) if omitted or empty. Invalid/unrecognized values are ignored; if none remain valid, falls back to all 7 days.
max_pagesint50Only relevant when crawl_all_pages is true and re-discovery runs on a future recrawl (see below) β€” stored on the job and reused so scheduled recrawls use the same site-scan scope originally chosen. Hard-capped server-side at MAX_ALLOWED_CRAWL_PAGES.
max_depthint2Same as max_pages β€” stored on the job for future recrawl re-discovery. Hard-capped server-side at MAX_ALLOWED_CRAWL_DEPTH.

If a site has more pages than the discovery limit found while building the pages list (see /crawl_discover), only those discovered pages are crawled here β€” /crawl itself does not re-discover further pages on a fresh call. max_pages/max_depth matter for this call mainly so future scheduled recrawls of the resulting job re-discover with the same scope, rather than reverting to the defaults.

πŸŒ™ Suggested recrawl window: midnight–6am, but fully overridable

By default, a scheduled recrawl is suggested to run sometime between 00:00 and 05:59 (server local time) on all 7 days, so it doesn't compete with normal daytime usage. This is only a suggestion, not a restriction: set recrawl_hour to any hour 0–23, and/or recrawl_days to any subset of days β€” including business hours, weekdays only, or weekends only.

πŸ—“οΈ How "every two weeks" actually works

"biweekly" is scheduled as a weekly cron trigger (at the chosen hour/days), but every other occurrence is skipped based on how many whole weeks have elapsed since the job's created_at timestamp β€” so it runs every 2 weeks from when the job was created, not on shared calendar-week boundaries. If two biweekly jobs are created on different dates, their recrawl weeks won't necessarily line up with each other.

Example β€” single page, no schedule
{
  "domain": "docs",
  "url": "https://example.com/pricing",
  "crawl_all_pages": false,
  "recrawl_interval": "none"
}
Example β€” whole site, recrawl weekly at the suggested off-peak default
{
  "domain": "docs",
  "url": "https://example.com/docs",
  "crawl_all_pages": true,
  "pages": [
    "https://example.com/docs",
    "https://example.com/docs/getting-started"
  ],
  "recrawl_interval": "weekly"
}

Omitting recrawl_hour/recrawl_days here means: a random hour between midnight and 6am, every day of the week.

Example β€” daily recrawl, weekdays only, at 2pm
{
  "domain": "docs",
  "url": "https://example.com/docs/pricing",
  "crawl_all_pages": false,
  "recrawl_interval": "daily",
  "recrawl_hour": 14,
  "recrawl_days": ["mon", "tue", "wed", "thu", "fri"]
}
Response β€” crawl started
{
  "status": "ok",
  "crawl_id": "9f3a7c21-...",
  "domain": "alice_docs",
  "pages_to_crawl": 2
}

Use crawl_id with /crawl_progress to follow along and get the final results β€” this response does not include page-by-page results itself.

πŸ”€ Character encoding is detected from page content, not just headers

Crawled HTML is decoded using the page's own declared encoding (a <meta charset> tag, a byte-order mark, or content-based detection), rather than trusting only the HTTP Content-Type header's charset β€” many sites only declare their encoding via the HTML meta tag, and relying solely on the header previously caused non-ASCII characters (e.g. č Ε‘ ΕΎ Δ‘ and similar) to be mis-decoded into garbled text. Already-indexed pages crawled before this fix can be corrected by recrawling them (via /run_crawl_job_now or waiting for the next scheduled recrawl) β€” a recrawl always replaces a page's existing content rather than appending to it.

πŸ—‘οΈ Detecting removed pages

If a previously-indexed page now responds 404 Not Found or 410 Gone, its existing content is automatically deleted from the domain instead of being left stale β€” the crawl treats this as "page removed," not as a failure. Other errors (timeouts, DNS failures, 5xx server errors, robots.txt disallow) are treated as transient/unknown and do not delete existing content, since the page may simply be temporarily unreachable. Removed pages appear in /crawl_progress's results as {"url": "...", "removed": true, "reason": "...", "deleted_points": N}.

Notes
  • Every call creates a new crawl job, even with recrawl_interval: "none" β€” there is no "one-off, don't remember this" mode. Remove unwanted jobs via /delete_crawl_job.
  • A short delay is applied between each page fetch, and robots.txt is respected the same way as in /crawl_discover.
  • Recorded in the audit log as a job-level crawl summary entry (including a removed count), plus one crawl_page entry per page (success, removal, or failure).
  • See /run_crawl_job_now and recrawl behavior below for how crawl_all_pages jobs also detect pages removed from a site's navigation entirely, not just pages whose exact URL now 404s.
GET /crawl_progress Requires api_key

Polled while a POST /crawl call is running in the background. Returns which page is currently being fetched, how many pages have been processed so far, and the results collected up to that point. Once done is true, the response reflects the final outcome β€” the same information the old synchronous /crawl response used to return directly.

Request
GET /crawl_progress?crawl_id=9f3a7c21-...
kwizmo_login_header: <api_key>
Response β€” in progress
{
  "status": "ok",
  "done": false,
  "total": 12,
  "current_index": 4,
  "current_url": "https://example.com/docs/getting-started",
  "domain": "alice_docs",
  "results": [
    { "url": "https://example.com/docs", "title": "Docs", "chunks_indexed": 6 },
    { "url": "https://example.com/docs/faq", "title": "FAQ", "chunks_indexed": 3 },
    { "url": "https://example.com/docs/old-page", "removed": true, "reason": "HTTP 404", "deleted_points": 5 },
    { "url": "https://example.com/whitepaper.pdf", "title": "whitepaper.pdf", "chunks_indexed": 9, "is_document": true }
  ],
  "job_id": null,
  "pages_crawled": null
}
Response β€” finished
{
  "status": "ok",
  "done": true,
  "total": 12,
  "current_index": 12,
  "current_url": null,
  "domain": "alice_docs",
  "results": [ /* full list of per-page results */ ],
  "job_id": "5a1c2e3f-...",
  "pages_crawled": 12
}
Response β€” unknown or expired crawl_id
{ "status": "error", "message": "Unknown or expired crawl_id" }
Notes
  • Progress is held in memory only (not persisted to disk) β€” it is meant for live polling during a crawl, not as a permanent record. A finished crawl's progress entry is kept for about 30 minutes then discarded; the crawl's actual results remain available afterward via /crawl_jobs (as the job's last_result) and via /audit_logs.
  • A server restart while a crawl is in progress loses that crawl's live progress tracking (you'll get "Unknown or expired crawl_id" on the next poll), though the crawl job itself, once it completes, is still persisted normally.
  • Recommended polling interval: about once per second β€” frequent enough to feel live, without hammering the server.
GET /crawl_jobs Requires api_key

Lists crawl jobs visible to the current user (jobs they created, or jobs belonging to a domain they have access to), optionally filtered to one domain.

Request
GET /crawl_jobs?domain=alice_docs
kwizmo_login_header: <api_key>

domain is optional β€” omit it to list jobs across all domains visible to the caller.

Response
{
  "jobs": [
    {
      "id": "5a1c2e3f-...",
      "domain": "alice_docs",
      "start_url": "https://example.com/docs",
      "crawl_all_pages": true,
      "pages": ["https://example.com/docs", "..."],
      "recrawl_interval": "weekly",
      "recrawl_hour": 3,
      "recrawl_days": ["mon", "tue", "wed", "thu", "fri", "sat", "sun"],
      "max_pages": 50,
      "max_depth": 2,
      "created_by": "alice",
      "created_at": "2026-07-01T12:00:00Z",
      "last_run": "2026-07-08T12:00:00Z",
      "last_result": [ /* per-page results from the most recent run */ ]
    }
  ]
}

recrawl_hour reflects the hour actually resolved at job creation (either what was requested, or the randomly suggested off-peak default) β€” it does not change on subsequent server restarts. max_pages/max_depth are the site-scan limits this job was created with, and are reused whenever a scheduled recrawl re-discovers the site's current pages.

POST /run_crawl_job_now Owner only

Triggers an immediate run of an existing crawl job, regardless of its configured recrawl interval. Uses the same recrawl semantics as a scheduled run (existing page content is removed and replaced with a freshly crawled version) β€” this is also how removed pages get cleaned up (see below).

Request (form)
job_id=5a1c2e3f-...
Response
{
  "status": "ok",
  "job_id": "5a1c2e3f-...",
  "message": "Recrawl started in the background"
}

πŸ—‘οΈ How a recrawl detects and removes deleted pages

Every recrawl (scheduled or manually triggered here) checks for pages that no longer exist, in two ways:

  • A known page now 404s or 410s β€” its indexed content is deleted from the domain, whether the job covers a single page or crawl_all_pages.
  • For crawl_all_pages jobs specifically, the site is re-discovered from the job's start_url before recrawling. Any page that was previously part of the job but is no longer reachable through the site's links β€” even if the URL itself doesn't error β€” is treated as removed and its content is deleted too. Any newly discovered page is picked up and crawled automatically. The job's stored pages list is then updated to match what was actually found, so future recrawls track the site's current structure rather than a stale snapshot from when the job was first created.

Other errors (timeouts, DNS failures, 5xx responses, a temporarily unreachable site during re-discovery) are treated as transient and do not delete anything β€” content is only removed on a definitive "this page is gone" signal. Specifically: if re-discovery for a crawl_all_pages job can't reach the site at all (e.g. the whole site times out), that is not treated as "every page vanished" β€” the recrawl falls back to the job's previously known page list for that run rather than risking mass deletion of an entire domain due to a transient outage.

Notes
  • Runs asynchronously in the background β€” this endpoint returns immediately rather than waiting for every page to finish, so it stays fast even for large jobs.
  • Progress and outcome are recorded in the audit log the same way as an automatic scheduled recrawl (crawl_page per page β€” including removals, plus a summary recrawl entry with a removed count), alongside a run_crawl_job_now entry marking that it was manually triggered.
  • Poll GET /crawl_jobs after a short delay to see the updated last_run timestamp, results, and (for crawl_all_pages jobs) the refreshed pages list.
POST /delete_crawl_job Owner only

Stops and removes a crawl job, cancelling any future scheduled recrawls. Pages already indexed by that job remain in the domain β€” this only removes the job's scheduling record, not previously-crawled content.

Request (form)
job_id=5a1c2e3f-...
Response
{ "status": "ok", "job_id": "5a1c2e3f-..." }
Response (job not found)
{ "status": "error", "message": "Job not found" }
Notes
  • Recorded in the audit log as delete_crawl_job.
Users & audit (admin)
POST /saveUser Headadmin only

Bulk-saves the full user list: activation status, role, assigned domains, and password resets.

⚠️ This replaces the entire user list

The request body is written to users.json as-is, in full β€” it is not a partial update. Any existing user omitted from the array you send will be permanently removed. Always start from the full list returned by POST /login (headadmin response) or reconstruct it completely before saving.

Request (JSON array)
[
  {
    "email": "bob@example.com",
    "username": "bob",
    "password": "$2b$12$...",
    "role": "admin",
    "status": 1,
    "domains": "alice_docs"
  }
]

Each entry needs email, username, password, role, status (0 or 1), and domains (comma-separated string).

Response
{ "status": "ok", "message": "Users saved successfully!" }
Response (not headadmin, or inactive)
{ "status": "failed", "message": "Users not saved successfully!" }
Notes
  • If a submitted password value isn't already a bcrypt hash, it's treated as a new plain-text password and hashed before saving β€” so the dashboard can leave existing hashes untouched while only sending a new value for users whose password is being reset.
  • Recorded in the audit log as save_users, including a per-user diff of what changed (password reset, status, role, domains).
GET /audit_logs Headadmin only

Returns structured audit log entries recording who did what across the whole system β€” logins, registrations, uploads/deletions, website crawls (including each individual page crawled or recrawled), searches, AI-provider queries, and user-management changes.

Query parameters
ParamTypeRequiredNotes
limitintNo (default 200, max 2000)Maximum entries returned, most recent first.
usernamestringNoExact match filter on the acting user.
actionstringNoExact match filter on the action name β€” see Audit action types.
Example request
GET /audit_logs?limit=100&username=alice&action=crawl_page
kwizmo_login_header: <api_key>
Response
{
  "status": "ok",
  "count": 2,
  "logs": [
    {
      "timestamp": "2026-07-09T14:02:11.203Z",
      "user": "alice",
      "action": "crawl_page",
      "status": "ok",
      "details": {
        "domain": "alice_docs",
        "url": "https://example.com/pricing",
        "chunks_indexed": 4,
        "job_id": "5a1c..."
      }
    },
    {
      "timestamp": "2026-07-09T14:01:58.771Z",
      "user": "alice",
      "action": "login",
      "status": "ok",
      "details": { "role": "admin" }
    }
  ]
}
Response (non-headadmin)
{ "status": "error", "message": "Only headadmin can view audit logs", "logs": [] }
Notes
  • Entries are stored as JSON Lines in /data/audit.log inside the container, independent of the free-text application log (/data/app.log).
  • API keys (OpenAI, Anthropic, and the session kwizmo_login_header key) are never written to audit entries.
Chunking strategy by file type

POST /ingest (and, separately, crawled web pages via POST /crawl) choose a chunking strategy and a chunk size/overlap based on content type, rather than applying one fixed word-count window to everything. The guiding idea: denser content (code, tables) uses smaller windows since more meaning is packed per word, looser prose uses larger windows, and formats with natural structure (headers, function/class boundaries) are split on that structure instead of an arbitrary word count.

Content typeStrategyChunk sizeOverlapExtra metadata
PDF / DOCX / DOCWord-count, per page/paragraph600 words80 wordspage or paragraph
Plain text (.txt)Word-count, per paragraph900 words150 wordsparagraph
Crawled web pagesWord-count700 words100 wordstitle
Markdown (.md)Split by header (#, ##, ###...); oversized sections sub-split by word count500 words (sub-split threshold)60 words (sub-split only)heading
Source codeSplit by function/class/method boundary; oversized blocks sub-split by word count220 words (sub-split threshold)20 words (sub-split only)language, function_name
CSV / TSVRow groups, header row repeated in every chunk20 rows/chunknoneβ€”

πŸ“„ Legacy Word (.doc) support is best-effort

.docx (the modern ZIP/XML-based format) is parsed properly and completely via python-docx. Legacy binary .doc files (Word 97–2003 format) are a completely different, much more complex binary structure with no equivalent full parser available here β€” instead, the server reads the document's internal text stream directly and heuristically extracts readable text runs. This recovers most of the plain text, headings, table cell contents, and list items in typical .doc files, but:

  • Reading order for tables and multi-column layouts is not guaranteed to exactly match the original visual layout.
  • Some formatting-heavy or unusually-encoded documents may extract incompletely.
  • If no readable text can be recovered at all (e.g. the file is corrupted, password-protected, or not really a .doc file despite its name/extension), /ingest and /crawl report a clear error rather than silently indexing nothing or garbled text.

If a .doc file doesn't extract well, the most reliable fix is to re-save it as .docx (in Word, LibreOffice, or Google Docs) and re-upload/re-crawl it β€” .docx gets the full, reliable parser.

Why structure-aware splitting for code and markdown

A fixed word-count window ignores syntax and section boundaries β€” it can cut a function in half or split a heading from the paragraph it introduces. For code and markdown, the server first splits on natural boundaries (a function/class definition, a header line) so each chunk is a complete logical unit whenever possible. A block that's still too large after that (e.g. one very long function) is sub-split by word count using a small window and light overlap, purely as a fallback β€” this is why those two rows above show a "sub-split threshold" rather than every chunk being exactly that size.

Recognized source code extensions

.py (Python), .js/.jsx (JavaScript), .ts/.tsx (TypeScript), .java (Java), .c/.h (C), .cpp/.hpp (C++), .cs (C#), .go (Go), .rb (Ruby), .php (PHP), .rs (Rust), .swift (Swift), .kt (Kotlin), .sql (SQL), .sh (Bash).

Notes
  • The code splitter is a lightweight, dependency-free heuristic (regex + indentation patterns for common function/class/method syntax across languages) β€” not a full parser. It errs on the side of leaving an unrecognized block attached to the previous one rather than mis-splitting mid-statement.
  • Leading content before the first recognized function/class (imports, header comments) is kept attached to the first chunk rather than discarded or split off on its own.
  • For markdown, content before the first header is similarly kept attached to the first section's chunk.
  • function_name and heading (and language for code) are surfaced in /search and /searchDomains results alongside the existing page/paragraph/chunk_index fields, and accepted as optional fields on /query_answer's results items.
Error handling & general caveats
  • Inconsistent "not allowed" responses by design β€” depending on the endpoint, a denied or inactive-user request returns an empty string (""), an empty response, or a JSON object like {"status":"error","message":"..."}. Always check for a non-"ok"/non-truthy status rather than assuming a particular shape.
  • HTTP status codes are mostly 200 β€” most business-logic failures (invalid credentials, inactive user, domain not found) are returned as 200 OK with an error payload, not a 4xx/5xx status. The main exception is POST /register on a duplicate username/email, which raises an actual HTTP 400.
  • Domain naming has two conventions β€” see Domain naming above; this is the most common source of confusing 404s from Qdrant (e.g. Collection `alice_docs` doesn't exist!) if the wrong convention is used for a given endpoint.
  • /saveUser replaces, not merges β€” see the warning under /saveUser.
Audit log action types

Recognized values for the action field, returned by /audit_logs:

ActionMeaning
registerNew user registration (success or denial, e.g. duplicate username/email).
loginLogin attempt, successful or denied (inactive user / invalid credentials).
logoutExplicit logout, invalidating the session's API key server-side.
ingest_fileFile upload/ingestion into a domain, including chunk count on success.
delete_fileDeletion of a single file's points from a domain.
delete_files_bulkBulk deletion of multiple files/URLs from a domain in one call.
delete_collectionDeletion of an entire domain (Qdrant collection).
crawlSummary entry for a manually-triggered crawl job (page count, succeeded/failed, recrawl schedule).
crawl_pageOne entry per page crawled, recrawled, or detected as removed (404/410, or no longer linked from the site), on both manual crawls and scheduled recrawls.
recrawlSummary entry for a scheduled recrawl run, attributed to a system (scheduled recrawl, created by <username>) actor.
run_crawl_job_nowManual, on-demand trigger of an existing crawl job's recrawl, outside its normal schedule.
delete_crawl_jobRemoval of a scheduled crawl job.
searchA call to /search (single-domain search).
query_answerA call to /query_answer, recording provider, model, domain and query β€” never the API key used.
save_usersHeadadmin changes to user role, status, domains, or password, via /saveUser. Includes a per-user diff of what changed.
view_audit_logsRecorded only when a non-headadmin user is denied access to /audit_logs.
test_connectivityA call to /test_connectivity, recording the target URL and the resulting overall status.
Quick integration checklist
  1. Register the first account via POST /register β€” it becomes headadmin automatically.
  2. Log in with POST /login to get an api_key; send it as the kwizmo_login_header header on every subsequent call.
  3. (Optional) Call GET /whoami to restore a session after a page refresh, instead of storing the password.
  4. Create a domain by uploading a file (POST /ingest) or crawling a site (POST /crawl); optionally check GET /domain_exists first if you want to confirm with the user before a new domain is created.
  5. List what's indexed with GET /files, and clean up with /delete_file, /delete_files_bulk, or /delete_collection as needed β€” remember these three use the full, prefixed domain name.
  6. Search with POST /search (one domain) or POST /searchDomains (all of the user's domains, combined).
  7. Use POST /query_answer with an OpenAI or Anthropic key (set provider accordingly) to generate an answer grounded in the search results.
  8. If logged in as headadmin, manage other users via POST /saveUser (remember: it replaces the whole list) and review activity via GET /audit_logs.
↑ Back to top