Kwizmo Local™ API β Developer Manual
Server base URL (example): https://localhost:5000 Β· 27 endpoints across accounts, domains, files, website crawling, search, AI answers, and administration.
Kwizmo Local is a self-hosted RAG (retrieval-augmented generation) knowledge base. Documents and crawled web pages are chunked, embedded, and stored per-domain in Qdrant. Once populated, a domain can be searched directly (/search) or used to generate an AI-written answer grounded in the retrieved context (/query_answer), using either OpenAI or Anthropic as the model provider.
This manual documents every HTTP endpoint exposed by app.py, in the order you're likely to need them: register β login β create a domain by uploading a file or crawling a site β search/ask β manage users and review the audit log.
Authentication is a two-step flow: register once, then log in to obtain a session API key.
- Create an account with
POST /register. - Authenticate with
POST /loginto receive anapi_key. - Send that key on every protected request using the header:
kwizmo_login_header: <api_key>
β Multiple concurrent sessions are supported
Each successful POST /login creates its own independent session and returns its own api_key. Multiple users β or the same user from multiple browsers/tabs/devices β can be logged in at the same time; logging in does not invalidate any other active session. POST /logout ends only the session tied to the specific api_key used to call it, leaving every other session untouched. There is no fixed cap on the number of concurrent sessions.
Roles & activation
- The first user ever registered automatically becomes
headadminwithstatus: 1(active) β no approval needed. - Every subsequent registration is created with role
adminandstatus: 0(inactive) β a headadmin must activate the account (viaPOST /saveUser) before that user can log in or use any protected endpoint. - An inactive user attempting most actions receives a quiet denial (often an empty string body or
{"status":"error","message":"Inactive user"}, depending on the endpoint) rather than an HTTP error β check the response body, not just the status code. headadminimplicitly has access to every domain, regardless of its owndomainsfield.
Two different conventions are used across endpoints
This is the single most common integration mistake. Endpoints fall into two groups:
- Auto-prefixing endpoints β
/ingestand/crawltake a short, unprefixed domain name (e.g.docs) and silently prefix it with the logged-in username (e.g.alice_docs) if it isn't already prefixed. These also auto-create the domain/collection on first use. - Exact-name endpoints β
/search,/files,/collection_stats,/delete_collection, and/delete_file//delete_files_bulkexpect the full, already-prefixed domain name exactly as returned byGET /domains(e.g.alice_docs, notdocs). They do not add a prefix for you.
When in doubt, call GET /domains first to get the exact names to use, or use GET /domain_exists before creating something new.
Creates a new user account.
Request (JSON){
"email": "alice@example.com",
"username": "alice",
"password": "secret123"
}
Response (success)
{
"status": "ok",
"message": "User registered successfully",
"loc": "..."
}
Response (username or email already taken)
HTTP 400
{ "detail": "User could not be registered" }
Notes
- The very first account ever registered becomes
headadmin, active immediately. - All later accounts are created as
admin,status: 0(inactive) until a headadmin activates them. - Recorded in the audit log as
register(statusokordenied, with a reason such as"username taken"or"email taken").
Authenticates a user and issues a new session api_key. Each login creates its own independent session β it does not invalidate any other session, including a previous session for the same user (e.g. logged in on another device) or any other user's active session. See Authentication & session model.
{
"username": "alice",
"password": "secret123"
}
Response (headadmin)
{
"status": "ok",
"api_key": "0123abcd...",
"users": [ /* full users.json content */ ],
"role": "headadmin"
}
Response (regular user)
{
"status": "ok",
"api_key": "abcd0123...",
"users": [ /* only this user's own record */ ],
"role": "admin"
}
Response (inactive user)
"" (empty string body)
Response (wrong credentials)
{ "status": "error", "message": "Invalid credentials" }
Notes
headadminreceives every user's record inusers; other roles receive an array containing only their own record (used by the dashboard's Users tab to show a self-service password-change view β see /saveUser).- Recorded in the audit log as
login(ok/denied, with reason"inactive user"or"invalid credentials").
Verifies that a previously-issued api_key is still valid and returns the associated username/role β without requiring the password again. Used by the dashboard to restore the logged-in view after a browser refresh (F5 / Ctrl+F5) instead of dropping back to the login screen.
GET /whoami kwizmo_login_header: <api_key>Response (valid session, regular user)
{
"status": "ok",
"username": "alice",
"role": "admin",
"users": [ /* only this user's own record */ ]
}
Response (session no longer valid)
{ "status": "error", "message": "Session no longer valid" }
Invalidates only the calling session's api_key server-side, so a stored key (e.g. kept in a browser's sessionStorage) can no longer be used afterward. Other sessions β including other active logins for the same user, or any other user's session β are unaffected.
POST /logout kwizmo_login_header: <api_key>Response
{ "status": "ok" }
Notes
- Recorded in the audit log as
logout.
Serves the dashboard's dashboard.html as-is. This is the same web UI a browser sees when visiting the server's root URL; it authenticates separately via /login from within the page.
Serves this developer manual (kwizmo_api_developer_manual_html.html) directly from disk. Linked from the dashboard's "π API Developer Manual" button.
Returns domains (Qdrant collections) visible to the current user. headadmin sees every collection on the server; other roles see only the domains listed in their own domains field.
GET /domains kwizmo_login_header: <api_key>Response
{ "domains": ["alice_docs", "bob_files"] }
Checks whether a domain already exists, applying the same <username>_<domain> prefixing used by /ingest and /crawl. Since those two endpoints silently auto-create a domain on first use, this lets a client (such as the dashboard) confirm with the user before that happens, rather than a new domain appearing unannounced.
GET /domain_exists?domain=ajpes kwizmo_login_header: <api_key>Response
{
"status": "ok",
"domain": "alice_ajpes",
"exists": false
}
Returns raw Qdrant collection metadata for a domain: vector count, indexing status, distance metric, HNSW/optimizer configuration, and so on. Powers the dashboard's "Vector DB Config" tab.
Exact-name endpoint β pass the full domain name as returned by /domains (e.g. alice_docs), not the short name. See Domain naming.
GET /collection_stats?domain=alice_docs kwizmo_login_header: <api_key>Response
{
"status": "green",
"vectors_count": 1204,
"points_count": 1204,
"config": { "params": { "vectors": { "size": 768, "distance": "Cosine" } }, ... },
...
}
Permanently deletes an entire domain (the whole Qdrant collection, including all indexed files/pages within it). Irreversible.
Request (form)domain=alice_docsResponse
{ "status": "ok", "domain": "alice_docs" }
Notes
- The caller must own the domain (its name must start with
<their_username>_); otherwise the request is silently denied (empty response). - Recorded in the audit log as
delete_collection.
Uploads a file to be chunked, embedded, and stored in a domain. Chunking strategy, chunk size, and overlap are chosen automatically based on file type β structured formats (code, markdown) are split on natural boundaries (functions, headers) rather than raw word count wherever possible. See Chunking strategy by file type below for full details.
Auto-prefixing & auto-creating endpoint β domain can be a short name; it will be prefixed with the logged-in username if not already prefixed, and the domain is created automatically if it doesn't exist yet. See /domain_exists to confirm first.
.pdf, .docx, .doc (best-effort β see Chunking strategy by file type), .txt, .md/.markdown, .csv/.tsv, and source code (.py .js .jsx .ts .tsx .java .c .h .cpp .hpp .cs .go .rb .php .rs .swift .kt .sql .sh). Anything else returns "Unsupported file type".
file: (binary file) domain: docsExample curl
curl -X POST "https://localhost:5000/ingest" \ -H "kwizmo_login_header: <api_key>" \ -F "domain=docs" \ -F "file=@manual.pdf"Response (success)
{
"status": "ok",
"domain": "alice_docs",
"file": "manual.pdf",
"chunks_indexed": 42
}
Response (errors)
{"status":"error","message":"Empty file"}
{"status":"error","message":"File produced no chunks"}
{"status":"error","message":"Unsupported file type"}
{"status":"error","message":"Could not extract readable text from this .doc file. ..."}
{"status":"error","message":"Failed to save to the vector database: ..."}
β±οΈ Large documents and Qdrant write timeouts
Indexing writes chunks to Qdrant in batches of QDRANT_UPSERT_BATCH_SIZE points per request (default 64), rather than one single request for the whole document, so a large file (hundreds of chunks) doesn't produce one oversized, slow request. The Qdrant client's own request timeout is also raised beyond the underlying library's very short 5-second default, to QDRANT_TIMEOUT_SECONDS (default 60s). Both are configurable via environment variables if you still see timeouts on very large documents or a slow/loaded Qdrant instance. This applies the same way to documents indexed via /crawl.
| Type | Chunk metadata recorded |
|---|---|
source, page (1-indexed), chunk_index | |
| DOCX | source, paragraph (1-indexed), chunk_index |
| DOC (legacy) | source, paragraph (1-indexed, based on extracted text runs β not identical to the original paragraph structure), chunk_index |
| TXT | source, paragraph (1-indexed, split on newlines), chunk_index |
| Markdown | source, heading (nearest enclosing header text), chunk_index |
| Source code | source, language, function_name (function/class/method name when identifiable), chunk_index |
| CSV/TSV | source, chunk_index (header row repeated in every chunk) |
- Recorded in the audit log as
ingest_file, includingchunks_indexedon success.
Lists the distinct files/URLs currently indexed in a domain, sorted case-insensitively.
Exact-name endpoint β pass the full domain name (e.g. alice_docs). See Domain naming.
GET /files?domain=alice_docs kwizmo_login_header: <api_key>Response
{ "files": ["manual.pdf", "https://example.com/pricing", "notes.txt"] }
Notes
- Entries can be plain filenames (from
/ingest) or full URLs (from/crawl) β both are stored in the samefilepayload field. - The domain is auto-created if it doesn't exist yet (so this returns an empty list rather than an error for a brand-new name).
Deletes every indexed chunk belonging to one file or URL from a domain.
Request (form)domain=alice_docs filename=manual.pdfResponse
{
"status": "ok",
"domain": "alice_docs",
"file": "manual.pdf",
"deleted_points": 12
}
Notes
- Recorded in the audit log as
delete_file.
Deletes every indexed chunk belonging to any number of files/URLs in one call β the collection is scanned once regardless of batch size, which matters for domains with thousands of documents. Powers the dashboard's "delete filtered files" action.
Request (form)domain=alice_docs filenames=["manual.pdf","draft-v1.pdf","draft-v2.pdf"]
"filenames" is a JSON-encoded array of filenames/URLs, exactly as they appear in GET /files.
{
"status": "ok",
"domain": "alice_docs",
"deleted_files": ["draft-v1.pdf", "draft-v2.pdf", "manual.pdf"],
"deleted_points": 47,
"not_found": []
}
Notes
not_foundlists any requested filenames that didn't match a point in the domain (e.g. already deleted).- Recorded in the audit log as
delete_files_bulk, including the list of files actually deleted.
How crawling fits together
Typical flow: optionally call /crawl_discover to preview which pages (and linked documents β see below) a site contains, then call /crawl to start fetching and indexing them into a domain in the background, polling /crawl_progress to follow along and get the final results. Every /crawl call β even a one-off with no recurring schedule β creates a persisted crawl job that can be listed (/crawl_jobs), re-run on demand (/run_crawl_job_now), or removed (/delete_crawl_job). If a crawl or discovery fails with a network-level error, use /test_connectivity to diagnose exactly where it's failing (DNS, TCP connect, or the HTTP/TLS layer) before assuming it's an application bug.
Diagnoses network reachability to a URL, staged the same way curl -v would report it: DNS resolution, a raw TCP connect, then the full HTTP request (including the TLS handshake for https:// URLs) β each timed independently, plus an informational robots.txt check. Answers "why can't the crawler reach this site?" (DNS failure vs. network/firewall block vs. a slow-but-reachable server vs. the site simply returning an HTTP error) without needing to shell into the container to run curl/ping/nslookup by hand.
{ "url": "https://example.com" }
Response β reachable
{
"status": "ok",
"result": {
"url": "https://example.com",
"hostname": "example.com",
"port": 443,
"dns": { "status": "ok", "duration_ms": 12.4, "resolved_ip": "93.184.216.34", "error": null },
"tcp_connect": { "status": "ok", "duration_ms": 45.1, "error": null },
"http_request": { "status": "ok", "duration_ms": 210.7, "status_code": 200, "content_type": "text/html; charset=UTF-8", "content_length": 1256, "error": null },
"robots_txt": { "status": "checked", "allowed": true, "error": null },
"overall": "reachable",
"summary": "Reachable. DNS: 12.4ms, TCP connect: 45.1ms, full HTTP request: 210.7ms. HTTP 200."
}
}
Response β connection timeout (the most common real-world failure)
{
"status": "ok",
"result": {
"url": "https://www.example-slow-site.com",
"hostname": "www.example-slow-site.com",
"port": 443,
"dns": { "status": "ok", "duration_ms": 18.2, "resolved_ip": "203.0.113.5", "error": null },
"tcp_connect": { "status": "timeout", "duration_ms": 15003.4, "error": "Connection to www.example-slow-site.com:443 timed out after 15s" },
"http_request": { "status": "not_attempted", "duration_ms": null, "status_code": null, "content_type": null, "content_length": null, "error": null },
"robots_txt": { "status": "not_attempted", "allowed": null, "error": null },
"overall": "connect_timeout",
"summary": "DNS resolved 'www.example-slow-site.com' to 203.0.113.5 in 18.2ms, but connecting to www.example-slow-site.com:443 timed out after 15s. This points to a network path problem (firewall, routing, or the remote host/service not accepting connections) rather than a DNS or application-level issue."
}
}
overall values
| Value | Meaning |
|---|---|
reachable | DNS, TCP connect, and the HTTP request all succeeded with a 2xx/3xx response. |
http_error | Connected fine, but the server returned a non-2xx/3xx HTTP status β not a network problem. |
dns_failure | The hostname could not be resolved at all β likely a typo, or this server has no working DNS. |
connect_timeout | DNS resolved fine, but the TCP connection to the resolved IP timed out β a network path/firewall/routing problem, or the remote service isn't accepting connections on that port. |
connect_failed | DNS resolved fine, but the TCP connection failed immediately (e.g. connection refused) rather than timing out. |
tls_error | TCP connected, but the TLS/SSL handshake failed (e.g. certificate problem). |
http_timeout | TCP connected, but the full HTTP request (including TLS) didn't complete in time β the server is slow/unresponsive at the HTTP layer specifically. |
http_failed | TCP connected, but the HTTP request failed for some other reason. |
invalid_url | url doesn't start with http:///https://, or no hostname could be parsed from it. |
- This is a synchronous call, not a background job like
/crawlβ a single URL check is fast enough not to need progress polling. Worst case it takes roughly up toCRAWL_TIMEOUT_SECONDS(default 15s) per stage attempted. - The
robots_txtcheck is informational only, run after the reachability stages regardless of their outcome (unless the URL is invalid) β it doesn't affectoverall. - Uses the same
CRAWL_TIMEOUT_SECONDSenvironment variable as page/document fetching for consistency with actual crawl behavior. - Recorded in the audit log as
test_connectivity, including the resultingoverallstatus.
Crawls outward from a starting URL (breadth-first, same site only) and finds every page URL reachable from it, without indexing anything. Used to let the user pick which pages to include before committing to a full crawl.
π Linked documents are included too
If a crawled page links to a file the server knows how to index β PDF, DOCX, Markdown, CSV/TSV, or source code (see Chunking strategy by file type) β that file's URL is included directly in the results, without being fetched as an HTML page or followed for further links. It's treated as a leaf document, the same as if it had been uploaded via /ingest. Links to files with unrecognized extensions, or on a different domain than the page being crawled, are not included.
A document is normally recognized from its URL's file extension. If a linked URL has no recognizable extension (e.g. a query-string download link), it's still detected as a document if fetching it returns a recognized document Content-Type instead of HTML β so links like /download?file=report.pdf are still picked up correctly.
β±οΈ Runs in the background β this returns immediately
This endpoint does not wait for discovery to finish. It starts the site scan on a background thread and returns right away with a discover_id. Poll GET /discover_progress?discover_id=<id> to see live progress (which page is currently being checked, how many pages found so far) and to know when it's finished and retrieve the final page list. This matters for larger sites, which can take a while to fully scan β the dashboard uses this to show a live "Scanning siteβ¦ 12 pages found so far" indicator instead of one static "please wait" message.
| Field | Type | Default | Notes |
|---|---|---|---|
url | string | β | Required. The page to start discovery from. |
max_pages | int | 5000 | Stop discovery once this many pages/documents have been found. Hard-capped server-side at MAX_ALLOWED_CRAWL_PAGES (default 5000) regardless of what's requested. |
max_depth | int | 2 | How many link-hops away from url to follow. Hard-capped server-side at MAX_ALLOWED_CRAWL_DEPTH (default 10). |
{ "url": "https://example.com/docs", "max_pages": 200, "max_depth": 3 }
Response β discovery started
{
"status": "ok",
"discover_id": "3b8e91a0-...",
"start_url": "https://example.com/docs",
"max_pages": 200,
"max_depth": 3
}
The returned max_pages/max_depth reflect the values actually applied (after server-side clamping), which may be lower than what was requested. Use discover_id with /discover_progress to follow along and get the final page list β this response does not include the page list itself. The final list from /discover_progress mixes HTML page URLs and document URLs together; a client can distinguish them by checking the URL's file extension against the list in Chunking strategy by file type, the same way the dashboard does.
{"status":"error","message":"URL must start with http:// or https://"}
{"status":"error","message":"This URL is disallowed by the site's robots.txt"}
{"status":"error","message":"max_pages must be at least 1"}
{"status":"error","message":"max_depth must be 0 or greater"}
β οΈ If a site has more pages than max_pages, only the first ones found are returned
Discovery stops as soon as max_pages is reached β it does not keep scanning to find the true total. /discover_progress's response includes "hit_page_limit": true whenever this happens, so a client can tell the user the results are likely incomplete rather than silently implying the whole site was found. Raise max_pages (up to MAX_ALLOWED_CRAWL_PAGES) and re-run discovery to find more.
- Default limits are 5000 pages and a link depth of 2 from the start URL β both are request parameters, configurable from the dashboard's crawl UI as well as directly via the API, not fixed. The maximum either can be set to (
MAX_ALLOWED_CRAWL_PAGES/MAX_ALLOWED_CRAWL_DEPTH) is itself configurable via environment variables (both default to matching the request defaults), to prevent an accidental request from launching an extremely long-running crawl against a huge site. robots.txtis respected β disallowed pages and documents are never queued or returned. If a site'srobots.txtitself can't be retrieved cleanly (missing, errors out, times out, or returns anything other than200 OK), that fails open β crawling proceeds as if norobots.txtexists β rather than being treated as "disallow everything," which previously caused otherwise-normal sites to fail discovery/crawling with no clear error. Fetchingrobots.txtuses its own short, non-retried timeout (default 5s, configurable viaROBOTS_TXT_TIMEOUT_SECONDS) rather than the same timeout/retry settings as page content β this keeps otherwise-fast endpoints like this one and /test_connectivity from being blocked for a long time by a single slow or unreachablerobots.txt. The result is cached per site after the first check.- A short delay is applied between page fetches during discovery, to avoid hammering the target site.
- Each page fetch (here and in /crawl) automatically retries once after a connection-level failure (timeout, DNS error, connection refused/reset) before giving up β HTTP error responses like 404/500 are never retried, since those are a definitive answer from the server rather than a network glitch. Both the connect/read timeout (default 15s) and retry count (default 1) are configurable via the
CRAWL_TIMEOUT_SECONDSandCRAWL_FETCH_RETRIESenvironment variables if your network has consistently higher latency.
Polled while a POST /crawl_discover call is running in the background. Returns the page currently being checked and how many pages have been found so far. Once done is true, either the final pages list is populated, or status is "error" if the site couldn't be reached at all.
GET /discover_progress?discover_id=3b8e91a0-... kwizmo_login_header: <api_key>Response β in progress
{
"status": "ok",
"done": false,
"start_url": "https://example.com/docs",
"current_url": "https://example.com/docs/getting-started",
"found": 4,
"pages": [],
"max_pages": 50,
"hit_page_limit": false
}
Response β finished, under the page limit
{
"status": "ok",
"done": true,
"start_url": "https://example.com/docs",
"current_url": null,
"found": 12,
"pages": [
"https://example.com/docs",
"https://example.com/docs/getting-started",
"https://example.com/docs/faq"
],
"max_pages": 50,
"hit_page_limit": false
}
Response β finished, page limit reached (site may have more pages)
{
"status": "ok",
"done": true,
"start_url": "https://example.com/docs",
"current_url": null,
"found": 5000,
"pages": [ /* 5000 URLs */ ],
"max_pages": 5000,
"hit_page_limit": true
}
When hit_page_limit is true, discovery stopped because it hit max_pages, not because the whole site was exhausted β there may be more pages on the site that were never reached. Re-run /crawl_discover with a higher max_pages to find more.
{
"status": "error",
"message": "Could not reach or crawl the given URL: Connection to example.com timed out. (connect timeout=15)",
"done": true
}
Response β unknown or expired discover_id
{ "status": "error", "message": "Unknown or expired discover_id" }
Notes
- Progress is held in memory only (not persisted to disk), the same as /crawl_progress β it's meant for live polling during discovery, not as a permanent record. A finished discovery's progress entry is kept for about 30 minutes then discarded.
- Recommended polling interval: about once per second β frequent enough to feel live, without hammering the server.
Crawls either a single URL or a specific list of pages, indexes their content into a domain, and persists a crawl job. If a page/document was already indexed before (matched by URL), its old content is removed and replaced β the same "recrawl" semantics used for scheduled re-runs.
π URLs pointing to documents are indexed as documents
If a URL in url or pages points to a PDF, DOCX/DOC, Markdown, CSV/TSV, or source code file (see Chunking strategy by file type), it is downloaded and chunked using the exact same logic as an uploaded file via /ingest β per-page/per-paragraph/per-function chunking, not the generic web-page text chunker. Its result entry includes "is_document": true and a title equal to the file's name (e.g. "whitepaper.pdf") rather than an HTML page title. Legacy .doc files use best-effort text extraction β see the callout in Chunking strategy by file type.
Documents are detected two ways: primarily from the URL's own file extension, and as a fallback, from the response's actual Content-Type header when the URL itself gives no hint β e.g. a query-string download link like /download?file=report.pdf or an extension-less REST-style URL like /documents/482/content. This fallback also covers the case of a genuine .docx/.doc file served with an unexpected or legacy Content-Type (e.g. application/msword on a .docx file). If a URL fetched as a page doesn't return HTML and its real Content-Type doesn't match any recognized document type either, it's reported as a failed page rather than silently indexed as something incorrect. Large documents are written to Qdrant in batches (see /ingest's note on QDRANT_UPSERT_BATCH_SIZE/QDRANT_TIMEOUT_SECONDS) so a big file's page result doesn't fail with a write timeout.
β±οΈ Runs in the background β this returns immediately
This endpoint does not wait for the crawl to finish. It starts the crawl on a background thread and returns right away with a crawl_id. Poll GET /crawl_progress?crawl_id=<id> to see live progress (which page is currently being fetched, how many are done, results as they complete) and to know when it's finished and retrieve the final result set. This matters for large crawl_all_pages jobs, which can take a while β the dashboard uses this to show a live "Crawling page 4 of 50β¦" indicator instead of one static "please wait" message.
Auto-prefixing & auto-creating endpoint β like /ingest, domain can be a short name and the domain is created automatically if needed.
| Field | Type | Default | Notes |
|---|---|---|---|
domain | string | β | Required. Short or full domain name. |
url | string | β | Required. The page to crawl, or the seed URL if crawl_all_pages is true. |
crawl_all_pages | bool | false | If true, crawls every URL in pages instead of just url. |
pages | array of strings | null | Required when crawl_all_pages is true β typically the list returned by /crawl_discover, filtered to what the user selected. |
recrawl_interval | string | "none" | One of "none", "daily", "weekly", "biweekly". Schedules automatic future recrawls of this job. |
recrawl_hour | int | random hour 0β5 | Hour of day (0β23, server local time) to run the recrawl. If omitted, a random hour between midnight and 6am is chosen and used consistently for that job going forward β not re-randomized on every restart. Any hour can be chosen; the 0β5 default is only a suggestion for off-peak timing. |
recrawl_days | array of strings | all 7 days | Which days of the week to recrawl on, from ["mon","tue","wed","thu","fri","sat","sun"]. Defaults to every day (weekends included) if omitted or empty. Invalid/unrecognized values are ignored; if none remain valid, falls back to all 7 days. |
max_pages | int | 50 | Only relevant when crawl_all_pages is true and re-discovery runs on a future recrawl (see below) β stored on the job and reused so scheduled recrawls use the same site-scan scope originally chosen. Hard-capped server-side at MAX_ALLOWED_CRAWL_PAGES. |
max_depth | int | 2 | Same as max_pages β stored on the job for future recrawl re-discovery. Hard-capped server-side at MAX_ALLOWED_CRAWL_DEPTH. |
If a site has more pages than the discovery limit found while building the pages list (see /crawl_discover), only those discovered pages are crawled here β /crawl itself does not re-discover further pages on a fresh call. max_pages/max_depth matter for this call mainly so future scheduled recrawls of the resulting job re-discover with the same scope, rather than reverting to the defaults.
π Suggested recrawl window: midnightβ6am, but fully overridable
By default, a scheduled recrawl is suggested to run sometime between 00:00 and 05:59 (server local time) on all 7 days, so it doesn't compete with normal daytime usage. This is only a suggestion, not a restriction: set recrawl_hour to any hour 0β23, and/or recrawl_days to any subset of days β including business hours, weekdays only, or weekends only.
ποΈ How "every two weeks" actually works
"biweekly" is scheduled as a weekly cron trigger (at the chosen hour/days), but every other occurrence is skipped based on how many whole weeks have elapsed since the job's created_at timestamp β so it runs every 2 weeks from when the job was created, not on shared calendar-week boundaries. If two biweekly jobs are created on different dates, their recrawl weeks won't necessarily line up with each other.
{
"domain": "docs",
"url": "https://example.com/pricing",
"crawl_all_pages": false,
"recrawl_interval": "none"
}
Example β whole site, recrawl weekly at the suggested off-peak default
{
"domain": "docs",
"url": "https://example.com/docs",
"crawl_all_pages": true,
"pages": [
"https://example.com/docs",
"https://example.com/docs/getting-started"
],
"recrawl_interval": "weekly"
}
Omitting recrawl_hour/recrawl_days here means: a random hour between midnight and 6am, every day of the week.
{
"domain": "docs",
"url": "https://example.com/docs/pricing",
"crawl_all_pages": false,
"recrawl_interval": "daily",
"recrawl_hour": 14,
"recrawl_days": ["mon", "tue", "wed", "thu", "fri"]
}
Response β crawl started
{
"status": "ok",
"crawl_id": "9f3a7c21-...",
"domain": "alice_docs",
"pages_to_crawl": 2
}
Use crawl_id with /crawl_progress to follow along and get the final results β this response does not include page-by-page results itself.
π€ Character encoding is detected from page content, not just headers
Crawled HTML is decoded using the page's own declared encoding (a <meta charset> tag, a byte-order mark, or content-based detection), rather than trusting only the HTTP Content-Type header's charset β many sites only declare their encoding via the HTML meta tag, and relying solely on the header previously caused non-ASCII characters (e.g. Δ Ε‘ ΕΎ Δ and similar) to be mis-decoded into garbled text. Already-indexed pages crawled before this fix can be corrected by recrawling them (via /run_crawl_job_now or waiting for the next scheduled recrawl) β a recrawl always replaces a page's existing content rather than appending to it.
ποΈ Detecting removed pages
If a previously-indexed page now responds 404 Not Found or 410 Gone, its existing content is automatically deleted from the domain instead of being left stale β the crawl treats this as "page removed," not as a failure. Other errors (timeouts, DNS failures, 5xx server errors, robots.txt disallow) are treated as transient/unknown and do not delete existing content, since the page may simply be temporarily unreachable. Removed pages appear in /crawl_progress's results as {"url": "...", "removed": true, "reason": "...", "deleted_points": N}.
- Every call creates a new crawl job, even with
recrawl_interval: "none"β there is no "one-off, don't remember this" mode. Remove unwanted jobs via/delete_crawl_job. - A short delay is applied between each page fetch, and
robots.txtis respected the same way as in/crawl_discover. - Recorded in the audit log as a job-level
crawlsummary entry (including aremovedcount), plus onecrawl_pageentry per page (success, removal, or failure). - See /run_crawl_job_now and recrawl behavior below for how
crawl_all_pagesjobs also detect pages removed from a site's navigation entirely, not just pages whose exact URL now 404s.
Polled while a POST /crawl call is running in the background. Returns which page is currently being fetched, how many pages have been processed so far, and the results collected up to that point. Once done is true, the response reflects the final outcome β the same information the old synchronous /crawl response used to return directly.
GET /crawl_progress?crawl_id=9f3a7c21-... kwizmo_login_header: <api_key>Response β in progress
{
"status": "ok",
"done": false,
"total": 12,
"current_index": 4,
"current_url": "https://example.com/docs/getting-started",
"domain": "alice_docs",
"results": [
{ "url": "https://example.com/docs", "title": "Docs", "chunks_indexed": 6 },
{ "url": "https://example.com/docs/faq", "title": "FAQ", "chunks_indexed": 3 },
{ "url": "https://example.com/docs/old-page", "removed": true, "reason": "HTTP 404", "deleted_points": 5 },
{ "url": "https://example.com/whitepaper.pdf", "title": "whitepaper.pdf", "chunks_indexed": 9, "is_document": true }
],
"job_id": null,
"pages_crawled": null
}
Response β finished
{
"status": "ok",
"done": true,
"total": 12,
"current_index": 12,
"current_url": null,
"domain": "alice_docs",
"results": [ /* full list of per-page results */ ],
"job_id": "5a1c2e3f-...",
"pages_crawled": 12
}
Response β unknown or expired crawl_id
{ "status": "error", "message": "Unknown or expired crawl_id" }
Notes
- Progress is held in memory only (not persisted to disk) β it is meant for live polling during a crawl, not as a permanent record. A finished crawl's progress entry is kept for about 30 minutes then discarded; the crawl's actual results remain available afterward via /crawl_jobs (as the job's
last_result) and via /audit_logs. - A server restart while a crawl is in progress loses that crawl's live progress tracking (you'll get "Unknown or expired crawl_id" on the next poll), though the crawl job itself, once it completes, is still persisted normally.
- Recommended polling interval: about once per second β frequent enough to feel live, without hammering the server.
Lists crawl jobs visible to the current user (jobs they created, or jobs belonging to a domain they have access to), optionally filtered to one domain.
RequestGET /crawl_jobs?domain=alice_docs kwizmo_login_header: <api_key>
domain is optional β omit it to list jobs across all domains visible to the caller.
{
"jobs": [
{
"id": "5a1c2e3f-...",
"domain": "alice_docs",
"start_url": "https://example.com/docs",
"crawl_all_pages": true,
"pages": ["https://example.com/docs", "..."],
"recrawl_interval": "weekly",
"recrawl_hour": 3,
"recrawl_days": ["mon", "tue", "wed", "thu", "fri", "sat", "sun"],
"max_pages": 50,
"max_depth": 2,
"created_by": "alice",
"created_at": "2026-07-01T12:00:00Z",
"last_run": "2026-07-08T12:00:00Z",
"last_result": [ /* per-page results from the most recent run */ ]
}
]
}
recrawl_hour reflects the hour actually resolved at job creation (either what was requested, or the randomly suggested off-peak default) β it does not change on subsequent server restarts. max_pages/max_depth are the site-scan limits this job was created with, and are reused whenever a scheduled recrawl re-discovers the site's current pages.
Triggers an immediate run of an existing crawl job, regardless of its configured recrawl interval. Uses the same recrawl semantics as a scheduled run (existing page content is removed and replaced with a freshly crawled version) β this is also how removed pages get cleaned up (see below).
Request (form)job_id=5a1c2e3f-...Response
{
"status": "ok",
"job_id": "5a1c2e3f-...",
"message": "Recrawl started in the background"
}
ποΈ How a recrawl detects and removes deleted pages
Every recrawl (scheduled or manually triggered here) checks for pages that no longer exist, in two ways:
- A known page now 404s or 410s β its indexed content is deleted from the domain, whether the job covers a single page or
crawl_all_pages. - For
crawl_all_pagesjobs specifically, the site is re-discovered from the job'sstart_urlbefore recrawling. Any page that was previously part of the job but is no longer reachable through the site's links β even if the URL itself doesn't error β is treated as removed and its content is deleted too. Any newly discovered page is picked up and crawled automatically. The job's storedpageslist is then updated to match what was actually found, so future recrawls track the site's current structure rather than a stale snapshot from when the job was first created.
Other errors (timeouts, DNS failures, 5xx responses, a temporarily unreachable site during re-discovery) are treated as transient and do not delete anything β content is only removed on a definitive "this page is gone" signal. Specifically: if re-discovery for a crawl_all_pages job can't reach the site at all (e.g. the whole site times out), that is not treated as "every page vanished" β the recrawl falls back to the job's previously known page list for that run rather than risking mass deletion of an entire domain due to a transient outage.
- Runs asynchronously in the background β this endpoint returns immediately rather than waiting for every page to finish, so it stays fast even for large jobs.
- Progress and outcome are recorded in the audit log the same way as an automatic scheduled recrawl (
crawl_pageper page β including removals, plus a summaryrecrawlentry with aremovedcount), alongside arun_crawl_job_nowentry marking that it was manually triggered. - Poll
GET /crawl_jobsafter a short delay to see the updatedlast_runtimestamp, results, and (forcrawl_all_pagesjobs) the refreshedpageslist.
Stops and removes a crawl job, cancelling any future scheduled recrawls. Pages already indexed by that job remain in the domain β this only removes the job's scheduling record, not previously-crawled content.
Request (form)job_id=5a1c2e3f-...Response
{ "status": "ok", "job_id": "5a1c2e3f-..." }
Response (job not found)
{ "status": "error", "message": "Job not found" }
Notes
- Recorded in the audit log as
delete_crawl_job.
Searches a single domain using the sentence-transformer embeddings and returns the top-k most similar chunks from Qdrant, with their similarity scores and source metadata.
Exact-name endpoint β pass the full domain name (e.g. alice_docs). See Domain naming.
query=How do I configure X? domain=alice_docs top_k=5Response
{
"domain": "alice_docs",
"query": "How do I configure X?",
"results": [
{
"score": 0.8421,
"text": "...",
"source": "manual.pdf",
"page": 2,
"paragraph": null,
"chunk_index": 3,
"id": "b7e2..."
}
]
}
Notes
- Returns
""(empty string) if the user is inactive or doesn't have access to the requested domain. - Recorded in the audit log as
search, including the query text and result count.
Searches every domain assigned to the current user in one call and returns a single combined, re-ranked result set.
Request (form)query=How do I configure X? top_k=5Response
{
"domain": "alice_docs,alice_notes",
"query": "How do I configure X?",
"results": [ /* top_k results, combined and re-sorted by score across all domains */ ]
}
β οΈ top_k applies to the final combined list, not per domain
Each assigned domain is searched for up to top_k results, all results are merged and sorted by score, and then the list is truncated to top_k overall. With many domains, results from a weaker-matching domain can be crowded out entirely by a strong-matching one.
- The user must have at least one domain listed in their own
domainsfield β this reads that field directly, so aheadadminaccount (whose owndomainsis typically empty, since it has implicit access to everything) will get"No domains assigned for this user"here even though it can access every domain via other endpoints. domainin the response is the raw comma-separated string from the user's profile, not a list.
Takes a query plus a set of retrieved chunks (typically the results from /search or /searchDomains) and asks an AI provider β OpenAI or Anthropic (Claude) β to write an answer grounded only in that context.
{
"domain": "alice_docs",
"query": "How to configure X?",
"provider": "openai",
"model": "gpt-4o-mini",
"openAIKey": "sk-...",
"creativity": 0.2,
"results": [
{"score": 0.9, "text": "...", "source": "manual.pdf", "page": 2}
]
}
Request (JSON) β Anthropic
{
"domain": "alice_docs",
"query": "How to configure X?",
"provider": "anthropic",
"model": "claude-sonnet-4-5",
"anthropicKey": "sk-ant-...",
"creativity": 0.2,
"results": [
{"score": 0.9, "text": "...", "source": "manual.pdf", "page": 2}
]
}
Response
{
"domain": "alice_docs",
"query": "How to configure X?",
"answer": "...AI-generated answer..."
}
Fields
| Field | Type | Required | Notes |
|---|---|---|---|
provider | string | No (default "openai") | Either "openai" or "anthropic". |
model | string | Yes | OpenAI example: gpt-4o-mini. Anthropic example: claude-sonnet-4-5. |
openAIKey | string | Required if provider is "openai" | Your OpenAI API key. Ignored for Anthropic requests. |
anthropicKey | string | Required if provider is "anthropic" | Your Anthropic API key. Ignored for OpenAI requests. |
creativity | float | Yes | Passed through as temperature. For Anthropic this is clamped server-side to Anthropic's valid 0β1 range even if a higher value (up to OpenAI's 0β2 scale) is sent. |
results | array | Yes | The context chunks to answer from β each needs at least score and text. Optional fields page, paragraph, function_name, heading, language are accepted but not required. Sorted by score descending server-side before being joined into the prompt. |
- If
provideris omitted, defaults to"openai"for backward compatibility. - Missing the required key for the chosen provider, or an unrecognized
providervalue, returns a short message inanswerand does not call any AI provider. - The system prompt instructs the model to answer using only the provided context, for both providers.
- Anthropic responses are generated with
max_tokens: 4096server-side. - Recorded in the audit log as
query_answer(provider, model, domain, query) β API keys are never written to the audit log.
Bulk-saves the full user list: activation status, role, assigned domains, and password resets.
β οΈ This replaces the entire user list
The request body is written to users.json as-is, in full β it is not a partial update. Any existing user omitted from the array you send will be permanently removed. Always start from the full list returned by POST /login (headadmin response) or reconstruct it completely before saving.
[
{
"email": "bob@example.com",
"username": "bob",
"password": "$2b$12$...",
"role": "admin",
"status": 1,
"domains": "alice_docs"
}
]
Each entry needs email, username, password, role, status (0 or 1), and domains (comma-separated string).
{ "status": "ok", "message": "Users saved successfully!" }
Response (not headadmin, or inactive)
{ "status": "failed", "message": "Users not saved successfully!" }
Notes
- If a submitted
passwordvalue isn't already a bcrypt hash, it's treated as a new plain-text password and hashed before saving β so the dashboard can leave existing hashes untouched while only sending a new value for users whose password is being reset. - Recorded in the audit log as
save_users, including a per-user diff of what changed (password reset, status, role, domains).
Returns structured audit log entries recording who did what across the whole system β logins, registrations, uploads/deletions, website crawls (including each individual page crawled or recrawled), searches, AI-provider queries, and user-management changes.
Query parameters| Param | Type | Required | Notes |
|---|---|---|---|
limit | int | No (default 200, max 2000) | Maximum entries returned, most recent first. |
username | string | No | Exact match filter on the acting user. |
action | string | No | Exact match filter on the action name β see Audit action types. |
GET /audit_logs?limit=100&username=alice&action=crawl_page kwizmo_login_header: <api_key>Response
{
"status": "ok",
"count": 2,
"logs": [
{
"timestamp": "2026-07-09T14:02:11.203Z",
"user": "alice",
"action": "crawl_page",
"status": "ok",
"details": {
"domain": "alice_docs",
"url": "https://example.com/pricing",
"chunks_indexed": 4,
"job_id": "5a1c..."
}
},
{
"timestamp": "2026-07-09T14:01:58.771Z",
"user": "alice",
"action": "login",
"status": "ok",
"details": { "role": "admin" }
}
]
}
Response (non-headadmin)
{ "status": "error", "message": "Only headadmin can view audit logs", "logs": [] }
Notes
- Entries are stored as JSON Lines in
/data/audit.loginside the container, independent of the free-text application log (/data/app.log). - API keys (OpenAI, Anthropic, and the session
kwizmo_login_headerkey) are never written to audit entries.
POST /ingest (and, separately, crawled web pages via POST /crawl) choose a chunking strategy and a chunk size/overlap based on content type, rather than applying one fixed word-count window to everything. The guiding idea: denser content (code, tables) uses smaller windows since more meaning is packed per word, looser prose uses larger windows, and formats with natural structure (headers, function/class boundaries) are split on that structure instead of an arbitrary word count.
| Content type | Strategy | Chunk size | Overlap | Extra metadata |
|---|---|---|---|---|
| PDF / DOCX / DOC | Word-count, per page/paragraph | 600 words | 80 words | page or paragraph |
| Plain text (.txt) | Word-count, per paragraph | 900 words | 150 words | paragraph |
| Crawled web pages | Word-count | 700 words | 100 words | title |
| Markdown (.md) | Split by header (#, ##, ###...); oversized sections sub-split by word count | 500 words (sub-split threshold) | 60 words (sub-split only) | heading |
| Source code | Split by function/class/method boundary; oversized blocks sub-split by word count | 220 words (sub-split threshold) | 20 words (sub-split only) | language, function_name |
| CSV / TSV | Row groups, header row repeated in every chunk | 20 rows/chunk | none | β |
π Legacy Word (.doc) support is best-effort
.docx (the modern ZIP/XML-based format) is parsed properly and completely via python-docx. Legacy binary .doc files (Word 97β2003 format) are a completely different, much more complex binary structure with no equivalent full parser available here β instead, the server reads the document's internal text stream directly and heuristically extracts readable text runs. This recovers most of the plain text, headings, table cell contents, and list items in typical .doc files, but:
- Reading order for tables and multi-column layouts is not guaranteed to exactly match the original visual layout.
- Some formatting-heavy or unusually-encoded documents may extract incompletely.
- If no readable text can be recovered at all (e.g. the file is corrupted, password-protected, or not really a
.docfile despite its name/extension),/ingestand/crawlreport a clear error rather than silently indexing nothing or garbled text.
If a .doc file doesn't extract well, the most reliable fix is to re-save it as .docx (in Word, LibreOffice, or Google Docs) and re-upload/re-crawl it β .docx gets the full, reliable parser.
Why structure-aware splitting for code and markdown
A fixed word-count window ignores syntax and section boundaries β it can cut a function in half or split a heading from the paragraph it introduces. For code and markdown, the server first splits on natural boundaries (a function/class definition, a header line) so each chunk is a complete logical unit whenever possible. A block that's still too large after that (e.g. one very long function) is sub-split by word count using a small window and light overlap, purely as a fallback β this is why those two rows above show a "sub-split threshold" rather than every chunk being exactly that size.
.py (Python), .js/.jsx (JavaScript), .ts/.tsx (TypeScript), .java (Java), .c/.h (C), .cpp/.hpp (C++), .cs (C#), .go (Go), .rb (Ruby), .php (PHP), .rs (Rust), .swift (Swift), .kt (Kotlin), .sql (SQL), .sh (Bash).
- The code splitter is a lightweight, dependency-free heuristic (regex + indentation patterns for common function/class/method syntax across languages) β not a full parser. It errs on the side of leaving an unrecognized block attached to the previous one rather than mis-splitting mid-statement.
- Leading content before the first recognized function/class (imports, header comments) is kept attached to the first chunk rather than discarded or split off on its own.
- For markdown, content before the first header is similarly kept attached to the first section's chunk.
function_nameandheading(andlanguagefor code) are surfaced in /search and /searchDomains results alongside the existingpage/paragraph/chunk_indexfields, and accepted as optional fields on /query_answer'sresultsitems.
- Inconsistent "not allowed" responses by design β depending on the endpoint, a denied or inactive-user request returns an empty string (
""), an empty response, or a JSON object like{"status":"error","message":"..."}. Always check for a non-"ok"/non-truthystatusrather than assuming a particular shape. - HTTP status codes are mostly 200 β most business-logic failures (invalid credentials, inactive user, domain not found) are returned as
200 OKwith an error payload, not a 4xx/5xx status. The main exception isPOST /registeron a duplicate username/email, which raises an actualHTTP 400. - Domain naming has two conventions β see Domain naming above; this is the most common source of confusing 404s from Qdrant (e.g.
Collection `alice_docs` doesn't exist!) if the wrong convention is used for a given endpoint. - /saveUser replaces, not merges β see the warning under /saveUser.
Recognized values for the action field, returned by /audit_logs:
| Action | Meaning |
|---|---|
register | New user registration (success or denial, e.g. duplicate username/email). |
login | Login attempt, successful or denied (inactive user / invalid credentials). |
logout | Explicit logout, invalidating the session's API key server-side. |
ingest_file | File upload/ingestion into a domain, including chunk count on success. |
delete_file | Deletion of a single file's points from a domain. |
delete_files_bulk | Bulk deletion of multiple files/URLs from a domain in one call. |
delete_collection | Deletion of an entire domain (Qdrant collection). |
crawl | Summary entry for a manually-triggered crawl job (page count, succeeded/failed, recrawl schedule). |
crawl_page | One entry per page crawled, recrawled, or detected as removed (404/410, or no longer linked from the site), on both manual crawls and scheduled recrawls. |
recrawl | Summary entry for a scheduled recrawl run, attributed to a system (scheduled recrawl, created by <username>) actor. |
run_crawl_job_now | Manual, on-demand trigger of an existing crawl job's recrawl, outside its normal schedule. |
delete_crawl_job | Removal of a scheduled crawl job. |
search | A call to /search (single-domain search). |
query_answer | A call to /query_answer, recording provider, model, domain and query β never the API key used. |
save_users | Headadmin changes to user role, status, domains, or password, via /saveUser. Includes a per-user diff of what changed. |
view_audit_logs | Recorded only when a non-headadmin user is denied access to /audit_logs. |
test_connectivity | A call to /test_connectivity, recording the target URL and the resulting overall status. |
- Register the first account via
POST /registerβ it becomesheadadminautomatically. - Log in with
POST /loginto get anapi_key; send it as thekwizmo_login_headerheader on every subsequent call. - (Optional) Call
GET /whoamito restore a session after a page refresh, instead of storing the password. - Create a domain by uploading a file (
POST /ingest) or crawling a site (POST /crawl); optionally checkGET /domain_existsfirst if you want to confirm with the user before a new domain is created. - List what's indexed with
GET /files, and clean up with/delete_file,/delete_files_bulk, or/delete_collectionas needed β remember these three use the full, prefixed domain name. - Search with
POST /search(one domain) orPOST /searchDomains(all of the user's domains, combined). - Use
POST /query_answerwith an OpenAI or Anthropic key (setprovideraccordingly) to generate an answer grounded in the search results. - If logged in as headadmin, manage other users via
POST /saveUser(remember: it replaces the whole list) and review activity viaGET /audit_logs.