Technical Graphs - Hawkesbury Council AI

System shape

Not a generic chatbot wrapped around a search box

Corpus engineering

The system ingests council business papers, minutes, attachments, policies, planning instruments, web-crawled pages, consultation sites, data-portal pages, demographic profiles, and legacy PDFs. Source metadata is preserved so answers can point back to meetings, documents, URLs, dates, and corpus families.

RAG retrieval

Documents are converted into markdown or structured records, chunked, embedded, and stored in Chroma. At question time, the backend plans which corpuses are relevant, retrieves candidate evidence, reranks the material, and gives the model a constrained evidence pack rather than asking it to rely on memory.

Custom executors

Repeated civic questions are routed to deterministic code paths where possible: motion catalog queries, topic histories, vote comparisons, attendance statistics, conflict declarations, financial tables, planning controls, media-release lookups, and other structured extractors.

LLM synthesis

The LLM is used where language understanding and synthesis matter most: interpreting ambiguous questions, combining retrieved evidence, explaining caveats, and producing readable answers. It sits on top of a retrieval and executor layer rather than replacing it.

Hybrid answer path

Custom executors first, LLM synthesis where it adds value

This data-flow view shows how the system avoids treating every query as a blank natural-language problem. Structured civic questions are routed through deterministic executors, open-ended questions use semantic retrieval, and the LLM receives a compact evidence pack for the final explanation.

Data flow diagram showing user questions routed through FastAPI, query planning, custom executors, semantic RAG, OpenAI synthesis, and cited answer rendering. — The main efficiency gain is architectural: the application narrows the problem before invoking the model. The LLM is still central, but it works from curated facts, passages, and citations rather than broad raw context.

Interactive artifacts

Two graph views of the project

These previews are screenshots. Select either image to open the full interactive graph on this same host.

Interactive graph of the codebase structure and communities

Code graph Codebase structure and module communities Graphify scanned the repository and clustered source files by import, reference, and dependency relationships. The generic community names were relabelled into human-readable functional areas.

Interactive semantic graph of document themes and document families

Semantic document atlas Themes, subthemes, sources, and document families A custom Graphify-style atlas groups corpus records into canonical document families, derives civic themes and subthemes with the LLM, and reveals more detail as the viewer zooms in.

Admin workflow

A complete RAG production pipeline, not a one-off upload

The admin interface coordinates the local evidence factory behind the chatbot: source discovery, document download, segmentation, Markdown conversion, structured sidecar generation, vector embedding, and deployment sync all sit in one workflow.

Admin Council Web Crawler page showing synced meeting-year folders and downloaded Council documents. — Council Web Crawler: tracks yearly meeting documents and downloads new or changed source files.

Admin Semantic Chunker page showing Markdown files selected for vector embedding. — Semantic Chunker: segments converted Markdown into context-aware chunks for vector embeddings.

Admin LLM and database management page showing vector database statistics, local models, and cloud model options. — LLM and DB Management: monitors vector-store size, local models, cloud models, and token costs.

The workflow starts by scraping configured source websites, including Council meeting repositories, Council web corpuses, consultation and data sites, and Councillor Zamprogno's site. New and changed files are downloaded, catalogued, and checked against prior metadata so repeat runs can skip unchanged material.

Meeting papers are then split into item-level records: agendas and minutes are segmented, attachments are distributed to their parent items, page-number provenance is correlated, and PDF material is converted into Markdown. Those Markdown files are chunked and embedded into sharded Chroma vector stores.

In parallel, preprocessing compiles JSON sidecars and global indexes for factual executors: votes, attendance, conflicts, keywords, financial tables, rates, capital works, road-network subtotals, procurement, grants, and other recurring civic statistics. The cloud synchronisation dashboard compares the local workstation with the Oracle deployment, then syncs the built frontend, FastAPI code, Chroma shards, backend data files, and corpus sidecars needed for public serving.

Vertical preprocessing workflow showing admin stages from source scraping through item folders, Markdown conversion, embeddings, sidecar and JSON compilation, validation, cloud synchronisation, and public chatbot serving. — The preparation path is deliberately staged: scrape first, preserve page and item provenance, convert to searchable Markdown, build vectors and structured JSON, then sync the tested local build to the cloud.

Retrieval strategy

How a question becomes an answer

1

Parse intent

The backend classifies whether the user is asking for semantic search, a dated meeting item, a topic history, a person or vote comparison, a planning-control lookup, a financial figure, or another structured task.

2

Select evidence

The public chat defaults to automatic source selection. It can activate meetings, policy documents, Councillor Zamprogno website material, and Hawkesbury Council web corpuses depending on the query.

3

Route to tools

If a structured executor is a better fit than plain vector search, the dispatcher sends the query to that executor and returns a compact, citation-ready evidence set.

4

Synthesise with citations

Retrieved passages and executor outputs are assembled into a prompt for the LLM. The answer is rendered with visible citations, activated-corpus disclosure, and guardrails against unsupported claims.

Tooling

What each layer contributes

FastAPI backend Authentication, chat APIs, streaming responses, corpus routing, executor dispatch, and admin workflows.

Vite frontend Public chat interface, saved sessions, source controls, account overlay, and this technical graph page.

Chroma vector stores Embedded document chunks for semantic retrieval across council meetings, policy material, and web corpuses.

OpenAI models Question interpretation, answer synthesis, graph taxonomy generation, and human-readable cluster labelling.

Graphify and graph builders Repository structure visualisation plus a bespoke semantic atlas for document themes and corpus relationships.

Custom executors Deterministic answers for repeatable civic-information tasks where raw vector search is too noisy or expensive.

Why the hybrid approach matters

Lower token costs, better reliability

A pure RAG chatbot can retrieve plausible passages, but it often spends tokens rediscovering structure the application already knows: meetings have dates, agenda items have identifiers, votes have named participants, and policies have stable document families. Encoding that knowledge in custom executors reduces the amount of context sent to the model and improves answer repeatability.

The LLM remains essential, but it is used as a reasoning and explanation layer. The lower layers narrow the question, find evidence, preserve source metadata, and produce compact intermediate results. That creates a more interrogable system: colleagues can inspect the corpus, the route, the executor output, the citations, and now the visual graphs that explain the project structure.

Cybersecurity

Standard protections built into the system

The public chatbot and admin dashboard use separate access controls, and the cloud deployment blocks local-only maintenance workflows that should remain on the workstation.

Authentication and sessions

Separate public-user and administrator authentication flows.
Google sign-in plus email registration with confirmation links.
BCrypt password hashing and tokenised password-reset flows for email accounts.
Signed JWT sessions with role and token-type checks before protected actions.

Access control

Admin APIs and the admin chat endpoint require admin JWTs.
Cloud mode blocks local ingestion, crawler, database, LLM, and pipeline maintenance endpoints.
Configurable CORS origins restrict browser API access to approved frontend hosts.
Secrets, environment files, local databases, logs, virtual environments, and cache folders are excluded from cloud code sync targets.

Abuse and cost controls

IP and user rate limits on login, registration, password reset, Google auth, and public chat.
Rolling public-chat quotas by query count, with optional token and estimated-cost quotas.
Maximum query length enforcement before expensive retrieval or model calls.
Administrator controls for custom user quotas and account suspension.

Auditability

Security audit events record authentication failures, rate limits, blocked access, quota blocks, and admin actions.
Audit metadata includes request path, method, IP address, user agent, actor type, user ID, and status code where available.
Admin dashboards expose user activity, question history, quota state, cloud usage, and moderation controls.
Source citations, page links, executor methodology notes, and structured sidecars help trace answer provenance.