cwvaServer-py: Architecture Reference

Derived from the Python retooling project session, May–June 2026

Assumptions

These premises were accepted during the session and underpin the architecture. If any is wrong, the decisions built on it may need revisiting.

• The production domain is visualartsdna.org; the server is the authoritative host for all RDF content queries and HTML rendering
• All TTL data files are the single source of truth; no relational database is used
• The SPARQL browser is low-traffic (measured at ~0.001% of production hits) and does not require isolation to a separate process
• GCP bucket is the canonical store for images and documents; local disk is a cache populated at load time
• RDFLib's SPARQL UPDATE engine cannot reliably execute BIND(IRI(...) AS ?subject) patterns — this is a known gap vs. Apache Jena
• Policy.upd is the single authoritative source for the ontology scheme top-class list; Python parses that list rather than hardcoding it
• Punning is used to place OWL classes into skos:ConceptScheme instances; ontology and thesaurus schemes are differentiated by the:tag the:forOntology vs the:tag the:forThesaurus, not by RDF type
• Ontology concept schemes are built at runtime by policy.py walking the rdfs:subClassOf hierarchy; thesaurus concept schemes exist in static TTL files
• The development environment is WSL2 on Windows; production target is GCP Debian

Technology Stack

Server Architecture

Single-Server Design

The original Groovy system used two servers (main on port 80, function on port 8080). The Python port consolidates to a single server. The function server's role in production was solely to handle SPARQL browser queries in an isolated process with a limited memory footprint. That risk is mitigated in Python by three controls applied to the built-in SPARQL browser endpoint:

• Query timeout — 10-second wall-clock limit via concurrent.futures thread wrapper
• Row cap — LIMIT 10000 appended to any incoming query lacking an explicit LIMIT
• Per-IP rate limiting — 10 requests per 10-minute window, same pattern as the agent endpoint

The function server architecture is preserved in Policy.upd as documentation and for portability back to Jena, but is not deployed.

Process Model

Single Uvicorn process, async throughout. Blocking operations (RDFLib queries, GCP fetches, DBMgr reload) are dispatched via asyncio.to_thread() to avoid blocking the event loop.

Data Stores (in-memory RDFLib graphs)

Load Sequence

1. GCP bucket sync — TTL files downloaded to local folders
2. Discrete stores loaded from TTL folders
3. Stores merged → data; schema loaded separately
4. owlrl RDFS closure applied → rdfs store
5. Policy executed: license/copyright appended per work; skos:Collection instances created; ontology concept schemes built by walking rdfs:subClassOf hierarchy from top-level classes (list parsed from Policy.upd)
6. SHACL validation run against rdfs store (pyshacl, advanced=True)
7. Load lock file released

Refresh

/refresh endpoint triggers a full DBMgr rebuild (re-sync from GCP + full RDFLib reload). The rebuild runs in a thread to keep the server responsive. Protected by a load lock file to prevent concurrent reloads. Also available via /cmd?cmd=refresh with token authentication.

Routing Structure

Two-tier routing inherited from the Groovy servlet structure, flattened into FastAPI route handlers in servlet.py and servlet_base.py.

Primary Routes (servlet.py)

Base Routes (servlet_base.py)

Policy Gate

servletPolicy.rson is loaded at startup. For the main server the policy is effectively open (".*" pattern matches all paths). The gate is implemented faithfully but does not block any requests in normal operation.

Key Implementation Decisions

Browser page rendering — JSON-LD intermediate serialization from the Groovy JsonLd2Html is replaced with direct RDFLib graph traversal in rdf2html.py. The property/value dict is built directly from the graph; no intermediate serialization. Avoids the JSON-LD 1.0 vs 1.1 migration issue entirely.

SPARQL query engine — all operational queries (gallery, browser page, explore, vocab tree) run against the in-process rdfs RDFLib graph. No connection to an external triple store for operational queries.

URI rehosting — schema: and vad: URIs containing http://visualartsdna.org are rewritten to cfg["host"] at query/render time, enabling local dev and production to use the same TTL files.

On-demand GCP image fetch — images and documents are not bulk-downloaded at startup. Individual files are fetched from the GCP bucket on first request and cached locally. Thumbnails are generated at fetch time via Pillow. Stale local files are evicted on /refresh.

Ontology graph scheme disambiguation — the:tag the:forOntology / the:tag the:forThesaurus on skos:ConceptScheme instances is the correct and only mechanism for separating the two select lists on the Explore page. Both scheme types are skos:ConceptScheme; RDF type is not a reliable discriminator.

RDFLib ORDER BY on dateTime — str(?dt) cast is required for correct chronological sorting. Without the cast, timezone-aware and timezone-naive xsd:dateTime literals sort incorrectly.

WSL2 networking — all generated HTML uses root-relative URLs (/images/..., /dist/...). Absolute URLs using localhost or the WSL2 internal IP break when the server is accessed by IP from another device on the LAN.

const redeclaration — gallery JavaScript must use let for variables declared in loops or conditional blocks. const redeclaration across script inclusion points causes a SyntaxError in strict-mode browsers.

Logging — normal operational output → stdout. Error and exception output → stderr. Both streams are captured to separate log files at process startup.

Deployment Environment

Development: WSL2 (Ubuntu) on Windows. Server runs on a WSL2-internal IP (172.29.x.x); LAN access from other devices requires a Windows port proxy (netsh portproxy). The WSL2 IP changes on each restart, so the proxy must be refreshed each session. The Groovy function server (retained for comparison) occupies port 8080; the Python server runs on a different port (e.g. 8081) in dev.

Production target: GCP Debian instance. Single server process, single port. The agent server (cwva_agent, separate process) runs on the same instance on port 8090; agentUrl in the rson config points to localhost:8090. No port proxy or firewall complexity — co-located processes communicate on loopback.

Content repositories:

• cwvaServer-py — server code (GitHub, MIT)
• cwvaMetacontent — ontology model and vocabulary TTL files (separate repo)
• cwvaContent — user content TTL files; images and documents excluded via .gitignore

Module Map (Groovy → Python)