Technical demonstrator
HAIKU, the Hawkesbury Artificial Intelligence Knowledge Utility, is a citation-first civic research system that combines local document engineering, hybrid search, validated analytics, and OpenAI reasoning to answer questions about Hawkesbury Council records while keeping the supporting evidence visible.
Video overview
Councillor Nathan Zamprogno introduces the Hawkesbury Artificial Intelligence Knowledge Utility before the technical diagrams below unpack its retrieval, data-processing, and answer-generation architecture.
System shape
The system ingests council business papers, minutes, attachments, policies, planning instruments, web-crawled pages, consultation sites, data-portal pages, demographic profiles, and legacy PDFs. Source metadata is preserved so answers can point back to meetings, documents, URLs, dates, and corpus families.
Documents are converted into Markdown or structured records, chunked, and embedded in Chroma. At question time, semantic matches are combined with exact-word FTS5/BM25 matches, fused and reranked, then reduced to a citation-ready evidence pack rather than asking the model to rely on memory.
Repeatable civic questions are routed to deterministic code paths where possible: CouncilStats comparisons, motion and vote analysis, meeting histories, attendance, finances, planning controls and guidance, consultation feedback, capital projects, rates, roads, grants, and other structured records.
The selected OpenAI model is used where language understanding and synthesis matter most: selecting likely sources, interpreting ambiguous questions, resolving constrained planning or statistical terms, and explaining the evidence. Hard-coded validation and retrieval still govern what reaches the answer.
Hybrid answer path
This data-flow view shows how the system avoids treating every query as a blank natural-language problem. Structured civic questions are routed through deterministic executors, open-ended questions use fused semantic and lexical retrieval, and the LLM receives a compact evidence pack for the final explanation.
The planner combines syntax cues such as rank, list all, by year, voted with, can I build, documents mention, or a street address with automatic corpus selection and constrained semantic nominations. Results are then checked against what the question actually asked for before they are accepted.
Interactive artifacts
These previews are screenshots. Select either image to open the full interactive graph on this same host.
Code graph
Codebase structure and module communities
The refreshed Graphify scan maps 5,359 public-safe code and documentation nodes, 13,836 relationships, and 243
communities. The largest communities are relabelled into human-readable functional areas.
Semantic document atlas
Themes, subthemes, sources, and document families
The refreshed atlas groups 21,113 corpus records into 13,969 canonical document families across eight
sources, 28 civic themes, and 136 subthemes, revealing more detail as the viewer zooms in.
Admin workflow
The admin interface coordinates the local evidence factory behind the chatbot: source discovery, document download, segmentation, Markdown conversion, structured sidecar generation, vector and lexical indexing, citation previews, quality checks, and deployment sync all sit in one workflow.
The workflow starts by scraping configured source websites, including Council meeting repositories, Council web corpuses, consultation and data sites, and Councillor Zamprogno's site. New and changed files are downloaded, catalogued, and checked against prior metadata so repeat runs can skip unchanged material.
Meeting papers are then split into item-level records: agendas and minutes are segmented, attachments are distributed to their parent items, page-number provenance is correlated, and PDF material is converted into Markdown. Those Markdown files are chunked and embedded into sharded Chroma vector stores.
In parallel, preprocessing compiles JSON sidecars and global indexes for factual executors: votes, attendance, conflicts, keywords, financial tables, rates, capital works, road-network subtotals, procurement, grants, CouncilStats, and other recurring civic statistics. A generated SQLite FTS5 index mirrors the searchable Chroma chunks, while preview workflows create safe first-page or website snapshots for citations. Cloud synchronisation then compares and transfers the tested serving assets to the Oracle deployment.
Retrieval strategy
The backend resolves dates and Council terms, classifies the request, selects likely corpuses, and may nominate a compatible executor with typed, grounded slots. Lexical route matches retain priority.
The public chat defaults to automatic source selection. Ordinary search runs Chroma semantic retrieval and SQLite FTS5/BM25 retrieval in parallel, then combines their ranked results with reciprocal-rank fusion.
If a deterministic executor is a better fit, the dispatcher runs it and validates the result against the question's obligations. Useful but incomplete results can be supplemented with focused retrieval.
Retrieved passages and executor outputs are assembled into a prompt for the LLM. The answer is rendered with visible citations, grouped meeting-item sources, optional previews, activated-corpus disclosure, and guardrails against unsupported claims.
Tooling
Why the hybrid approach matters
A retrieval-only chatbot can find plausible passages, but it often spends tokens rediscovering structure the application already knows: meetings have dates, agenda items have identifiers, votes have named participants, and policies have stable document families. Encoding that knowledge in custom executors reduces the amount of context sent to the model and improves answer repeatability.
The LLM remains essential, but it is used as a reasoning and explanation layer. The lower layers narrow the question, fuse semantic and exact-word evidence, validate structured results, preserve source metadata, and produce compact intermediate results. That creates a more interrogable system: people can inspect the corpus, route, executor output, citations, previews, and the visual graphs that explain the project structure.
Cybersecurity
The HAIKU public chatbot and admin dashboard use separate access controls, and the cloud deployment blocks local-only maintenance workflows that should remain on the workstation.
Project scale audit
Counts separate application code, semantic document records, vector-search chunks, and structured extractor records so the project size is visible at more than one level. This audit was refreshed on 18 July 2026 alongside both interactive graphs.
Council meeting coverage runs from 10 February 1981 through the latest available 2026 Council and HLPP material. The published 1981-2006 archive adds 558 meeting folders to the searchable evidence base.
Counted with tools/count_project_lines.py. Excludes generated frontend bundles, corpuses,
virtual environments, vector databases, logs, and runtime data.
Physical disk usage for backend/database/chroma_shards, excluding legacy fallback Chroma
folders outside the serving shard layout.
The public atlas renders 1,800 representative document families while counting restricted titles without exposing them as individual nodes.
The 1981-2006 contribution is counted from meeting folders in Distributed_Corpus_1981-2006.
Pre-2007 records are document-level; structured voting and attendance detail is strongest from 2007 onward.