Technical demonstrator
HAIKU, the Hawkesbury Artificial Intelligence Knowledge Utility, is a citation-first civic research system that combines local document engineering, hybrid search, validated analytics, and OpenAI reasoning to answer questions about Hawkesbury Council records while keeping the supporting evidence visible.
Video overview
Councillor Nathan Zamprogno introduces the Hawkesbury Artificial Intelligence Knowledge Utility before the technical diagrams below unpack its retrieval, data-processing, and answer-generation architecture.
System shape
The system ingests council business papers, minutes, attachments, policies, planning instruments, web-crawled pages, consultation sites, data-portal pages, demographic profiles, legacy PDFs, and recorded meeting debate. Source metadata is preserved so answers can point back to meetings, documents, exact speeches, URLs, dates, and corpus families.
Documents are converted into Markdown or structured records, chunked, and embedded in Chroma. At question time, semantic matches are combined with exact-word FTS5/BM25 matches, fused and reranked, then reduced to a citation-ready evidence pack rather than asking the model to rely on memory.
Repeatable civic questions are routed to deterministic code paths where possible: CouncilStats comparisons, motion and vote analysis, meeting histories, attendance, finances, planning controls and guidance, consultation feedback, chamber participation, exact speech catalogues, capital projects, rates, roads, grants, and other structured records.
The selected OpenAI model is used where language understanding and synthesis matter most: selecting likely sources, interpreting ambiguous questions, resolving constrained planning or statistical terms, and explaining the evidence. Hard-coded validation and retrieval still govern what reaches the answer.
Hybrid answer path
This data-flow view shows how the system avoids treating every query as a blank natural-language problem. Structured civic questions are routed through deterministic executors, open-ended questions use fused semantic and lexical retrieval across documents and relevant Hansard speech, and the LLM receives a compact evidence pack for the final explanation.
The planner combines syntax cues such as rank, list all, by year, voted with, can I build, documents mention, or a street address with automatic corpus selection and constrained semantic nominations. Results are then checked against what the question actually asked for before they are accepted.
Interactive artifacts
These previews open three complementary interactive views: the software itself, the broad evidence library, and the themes discussed in recorded Council meetings.
Code graph
Codebase structure and module communities
The refreshed Graphify scan maps 13,222 public-safe code and documentation nodes, 34,773 relationships,
and 495 communities. Search and community filters make the dense dependency map practical to inspect.
Semantic document atlas
Themes, subthemes, sources, and document families
The existing atlas groups 94,468 corpus records into 16,238 canonical document families across ten
sources, 28 civic themes, and 136 subthemes, revealing more detail as the viewer zooms in.
Hansard semantic map
What Council discussed, grouped by language and context
A read-only analysis of the active release maps 1,919 agenda topics into 24 language clusters. Its 465
nodes and 945 links expose representative meetings, speakers, themes and exact Hansard deep links.
The code map is generated with Graphify. The two semantic maps use HAIKU's Python graph builders, scikit-learn analysis, corpus metadata and the vis-network browser renderer; the broad document atlas remains unchanged in this refresh.
Admin workflow
The admin interface coordinates the local evidence factory behind the chatbot: source discovery, document download, segmentation, Markdown conversion, structured sidecar generation, vector and lexical indexing, citation previews, quality checks, and deployment sync all sit in one workflow.
The workflow starts by scraping configured source websites, including Council meeting repositories, Council web corpuses, consultation and data sites, and Councillor Zamprogno's site. New and changed files are downloaded, catalogued, and checked against prior metadata so repeat runs can skip unchanged material.
Meeting papers are then split into item-level records: agendas and minutes are segmented, attachments are distributed to their parent items, page-number provenance is correlated, and PDF material is converted into Markdown. Those Markdown files are chunked and embedded into sharded Chroma vector stores.
In parallel, preprocessing compiles JSON sidecars and global indexes for factual executors: votes, attendance, conflicts, keywords, financial tables, rates, capital works, road-network subtotals, procurement, grants, CouncilStats, and other recurring civic statistics. A generated SQLite FTS5 index mirrors the searchable Chroma chunks, while preview workflows create safe first-page or website snapshots for citations. Cloud synchronisation then compares and transfers the tested serving assets to the Oracle deployment.
Recorded meeting workflow
Meeting audio follows its own evidence pipeline. The media archive is linked to stable meeting and item IDs; glossary-assisted transcription preserves the authoritative words; local timing, diarisation and voice comparison add reviewable speaker evidence; and only a complete, validated release reaches the public Hansard. Expensive stages are durable and resumable, while later text or identity corrections use overlays so the source recording never has to be altered.
Meeting Hansard
HAIKU treats speaker-labelled transcripts as a fifth, first-class evidence corpus. It preserves the difference between what was said, what staff recommended, and what Council formally resolved, while making exact contributions discoverable from the chatbot and the public Hansard.
Meeting recordings are archived with yt-dlp and FFmpeg, transcribed in resumable chunks, timed with MLX Whisper, separated into turns by pyannote Community-1, and compared with quality-filtered WeSpeaker voice profiles. Unknown speakers remain explicit instead of being forced to a name.
SoundCloud item tracks inherit agenda-item links directly. Whole webcasts are divided into broad, reviewable chapters using the complete transcript and correlated minutes. Public Forum topics can link to a later formal item while retaining their own proceeding, speaker, chronology, and evidentiary role.
A validated release precompiles authoritative SQLite, compressed JSONL, SQLite FTS5, year-sharded Chroma vectors, statistics, static pages, and content-addressed AAC speech clips. The interactive reader hydrates current identity and citation data by stable speech ID, while static pages provide the crawlable search-engine surface.
Exact item questions can combine official papers and decisions, formal Councillor and staff debate, and separately labelled Public Forum contributions. Participation rankings come from deterministic statistics; broad topic and speaker questions use hybrid lexical/vector retrieval with deep links to the exact speech.
Retrieval strategy
The backend resolves dates and Council terms, classifies the request, selects likely corpuses, and may nominate a compatible executor with typed, grounded slots. Lexical route matches retain priority.
The public chat defaults to automatic source selection. Ordinary search runs Chroma semantic retrieval and SQLite FTS5/BM25 retrieval in parallel across official documents and, when relevant, the validated Chamber Debate release, then combines their ranked results with reciprocal-rank fusion.
If a deterministic executor is a better fit, the dispatcher runs it and validates the result against the question's obligations. Useful but incomplete results can be supplemented with focused retrieval.
Retrieved passages and executor outputs are assembled into a prompt for the LLM. The answer is rendered with visible citations, grouped meeting-item sources, optional previews, activated-corpus disclosure, and guardrails against unsupported claims.
Tooling
Why the hybrid approach matters
A retrieval-only chatbot can find plausible passages, but it often spends tokens rediscovering structure the application already knows: meetings have dates, agenda items have identifiers, votes have named participants, and policies have stable document families. Encoding that knowledge in custom executors reduces the amount of context sent to the model and improves answer repeatability.
The LLM remains essential, but it is used as a reasoning and explanation layer. The lower layers narrow the question, fuse semantic and exact-word evidence, validate structured results, preserve source metadata, and produce compact intermediate results. That creates a more interrogable system: people can inspect the corpus, route, executor output, citations, previews, and the visual graphs that explain the project structure.
Cybersecurity
The HAIKU public chatbot and admin dashboard use separate access controls, and the cloud deployment blocks local-only maintenance workflows that should remain on the workstation.
Project scale audit
Counts separate application code, semantic document records, vector-search chunks, and structured extractor records so the project size is visible at more than one level. This audit was refreshed on 31 August 2026; the Code graph and Hansard map were regenerated while the broad Semantic Document Atlas was left unchanged.
Council meeting coverage runs from 10 February 1981 through the latest available 2026 Council, HLPP, and committee material. The published 1981-2006 archive contributes 558 meeting folders to the searchable evidence base.
Counted with tools/count_project_lines.py. Excludes generated frontend bundles, corpuses,
virtual environments, vector databases, logs, and runtime data.
Physical disk usage for backend/database/chroma_shards, excluding legacy fallback Chroma
folders outside the serving shard layout.
The public atlas renders 1,800 representative document families while counting restricted titles without exposing them as individual nodes.
The 1981-2006 contribution is counted from meeting folders in Distributed_Corpus_1981-2006.
Pre-2007 records are document-level; structured voting and attendance detail is strongest from 2007 onward.
Committee agendas and minutes remain under the same Council-meetings search selection.
These are the deliberately unchanged document-atlas snapshot counts, not the active retrieval-release inventory above. The atlas retains an earlier, broader speech-card representation for visual comparison.