Why This Project Exists
Delivery questions are annoying to answer because the truth is spread across systems. Commit and pull-request activity lives in GitHub. Tasks and due dates live in a task tracker. The reasoning — why a decision was made, what the stated risks are — lives in documents nobody rereads. Answering "how is this project going?" means joining three sources by hand.
DeliveryLens does that joining ahead of time. A scheduled job copies GitHub and task-tracker data into a local Postgres database, a separate ingest indexes PDFs and pull-request bodies into a searchable corpus, and questions are answered entirely from those local copies. The model never calls GitHub or the tracker directly. It only calls tools, and every tool reads the mirror.
The Rule: The Model Writes The Text, Never The Number
Every tool result goes to two places. One copy goes to the model, which writes the prose. The other goes to a view builder that renders charts and tables straight from that same JSON, with no model in the path.
flowchart LR
T["Tool result JSON"] --> M["AI model"] --> P["Prose on the screen"]
T --> V["View builder (fixed code)"] --> C["Charts and tables on the screen"]
So when COUNT(*) FILTER (WHERE is_overdue) returns 7, the 7 on screen came from
the query. The model never retypes it. The system prompt also forbids markdown tables,
because the real figures are already rendered above the text.
This is what makes a small model good enough. It has to pick the right tool and explain what the result means. It never has to be trusted with arithmetic.
Architecture
Data flows one way: external services into the database, database into the answer. Two reasons that matters. One question can need several database lookups, and doing that against live APIs would be slow and would burn through GitHub's hourly rate limit. And a tool result is a fact read out of a table, not something the model can misremember.
flowchart TD
GH["GitHub API"]
CU["Task tracker API"]
PDF["PDFs and PR bodies"]
SYNC["Sync services: watermarked, branch-aware"]
ING["Corpus ingest: chunk then embed"]
DB[("Postgres: Neon + pgvector")]
GH --> SYNC --> DB
CU --> SYNC
PDF --> ING --> DB
Q["Question"] --> AGENT["Agent loop"]
AGENT -->|"picks a tool"| TOOLS["16 typed tools"]
TOOLS --> INS["Insight layer: SQL aggregates"]
TOOLS --> RET["Hybrid retrieval: vector + full-text, fused, reranked"]
INS --> DB
RET --> DB
TOOLS -->|"result JSON"| LLM["Model"]
TOOLS -->|"same result JSON"| VIEW["View builder (no model)"]
LLM -->|"text"| UI["Answer on screen"]
VIEW -->|"charts, tables"| UI
How A Question Turns Into An Answer
A status question runs about three rounds and two tool calls, roughly six seconds end to end. The model is told to always resolve a name before measuring anything, because people ask about "the retail thing" and that has to map to a real project row first.
sequenceDiagram
participant D as User
participant API as /api/v1/chat
participant A as Agent loop
participant T as Tools
participant DB as Neon
participant M as AI model
D->>API: "How is this project going?"
API->>API: Check session and allowlist
API->>A: Start the agent loop
A->>M: Question, plus 16 tool definitions
M-->>A: Call resolve_entity
A->>T: Run the tool
T->>DB: SELECT
DB-->>T: Rows
T-->>A: Result
A-->>D: Event: working out what you mean
A->>M: Tool result
M-->>A: Call get_project_status
A->>T: Run the tool
T-->>A: Result plus chart views
A-->>D: Charts and tables
A->>M: Tool result
M-->>A: Text
A-->>D: Text, one part at a time
The status query joins tracker tasks with GitHub commits and pull requests and returns one
structured result plus a provenance block: when each source last synced, and how
many accounts are still unmatched. The client never sends conversation history back — just
the new message and a conversation id. The server reads earlier turns from the database, so
refreshing resumes the thread and a client cannot rewrite what was already said.
Engineering Highlights
Problem: the two sources describe the world at different scales. The financial documents describe a company with thousands of customers; the operational data describes a handful of projects and a small team. "How many people do we have?" therefore has two different correct answers depending on which source you ask.
Implementation: counts and rates come from SQL; prose comes from the corpus. The assistant always states which source it used, and it never adds a document figure to a measured one. Mixing them would produce a number that is arithmetically fine and semantically meaningless.
Problem: an aggregate without its denominator invites a wrong reading. A zero is the worst case: "no GitHub account linked" and "did not write any code" look identical, but only one of them is a performance signal.
Implementation: get_overdue_tasks also
reports how many tasks have no due date and so cannot be judged either way.
rank_developers returns its formula, its time window, who was excluded, and a
note on any row where one data source contributed nothing. The caveat travels with the
number instead of living in documentation nobody reads.
Problem: a project manager will never have commits. Scoring everyone on one combined number puts every non-engineering role at the bottom by construction — that is a fact about the job, not about the person.
Implementation: code-shipping roles and non-code roles are ranked separately, and the charts are drawn per group rather than combined, since combining them would imply a comparison the data cannot support. Anyone with zero activity in the window is listed separately instead of ranked, because no data is not the same as bad performance.
Problem: plain vector search misses exact terms — a similarity search can skip straight past the passage that names a specific metric. Keyword search misses paraphrasing. Either one alone loses real answers.
Implementation: a pgvector/HNSW vector search and Postgres full-text search run together, their rankings are fused by reciprocal rank, and the combined list goes to a Cohere cross-encoder that reads the query and each passage together and scores them. The per-document cap is applied after reranking, so it never reorders results — it only stops one long document from taking every slot.
flowchart LR
Q["Question"]
V["Vector search: top 30"]
K["Keyword search: top 30"]
F["Fuse by reciprocal rank"]
R["Cohere cross-encoder rerank"]
C["Cap 2 per document"]
OUT["Passages shown"]
Q --> V --> F
Q --> K --> F
F -->|"pool of 4x limit"| R --> C --> OUT
Problem: the reranker is a third-party API call on the critical path of every document question. If it is the single point of failure, a quota problem takes out retrieval entirely.
Implementation: if the Cohere call times out or
errors, retrieval falls back to selecting a diverse set from the vectors it already has —
no extra API call, just arithmetic, which is what matters when quota is the limiting
factor. The result records which path ran (cross-encoder or
diversity-fallback) and the note passed to the model changes accordingly, so
a weaker answer gets flagged rather than shipped silently.
Problem: GitHub handles and tracker accounts follow no shared naming pattern. An automatic wrong link would silently attribute one person's work to someone else, and nothing downstream would catch it.
Implementation: the identity service scores likely
pairs and surfaces the busiest unmatched accounts first, but it only ever
suggests. A human confirms. Bot accounts are filtered on both handle and display
name, since some bots commit under a normal-looking login but a [bot]-tagged
name.
Problem: the auth provider allows open sign-up, so a successful login says nothing about whether you should see the data.
Implementation: a separate allowlist decides access, and an empty allowlist locks everyone out rather than letting everyone in by mistake. That check deliberately does not depend on the auth library or on the framework, so the preflight script can test it directly by feeding in bad addresses and confirming they are rejected.
The Agent Loop
flowchart TD
S["Start: system prompt plus question"] --> R["Send to the model"]
R --> C{"Did the model ask for tools?"}
C -->|No| T["Send the text. Stop."]
C -->|Yes| E["Run each tool"]
E --> V["Build the charts from the result"]
V --> A["Add the result to the conversation"]
A --> L{"Round limit reached?"}
L -->|No| R
L -->|Yes| X["Report the limit. Stop."]
Three Bugs Worth Writing Up
One provider streams a single tool call in fragments, each chunk tagged with a position index. Another sends multiple complete calls that all report the same position, distinguished only by an id. Keying purely by position concatenated two calls into one broken JSON string, which the provider then rejected with an unhelpful 400. The fix keys by call id when one is present, falling back to the position index only for a continuation of a call already in progress — which covers both providers' behaviour.
A lot of work lived on personal branches that were never merged, so syncing only the default branch missed most of it. The first attempt walked the full history of every branch on every sync, which made the rate-limit problem worse rather than better. The actual fix: sync the default branch first so its history is already stored, then skip any branch whose latest commit is already in the database.
get_overdue_tasks filtered its results by project but computed the "how many
tasks have due dates" total against the whole workspace — so "6 overdue out of 35 with due
dates, out of 51 total" mixed one project's overdue count with the entire organisation's
task count. Every number was correct on its own; together they were misleading.
Separately, renameConversation and deleteConversation checked
ownership correctly in their SQL, then decided success with result.length > 0.
TypeORM returns [rows, affectedCount] for updates and deletes, so that check
was always true regardless of who owned the row. Reading the code would not have caught
either bug. Tests that asked "does the denominator match the filter?" and "can another
user delete this?" found both immediately.
Tradeoffs And Honest Limits
Answers are only as fresh as the last sync. That is the deliberate cost of not calling the
live APIs at question time, and it is why every result carries a provenance block
saying when each source was last copied — a stale answer should announce itself rather than
look current.
The system is also bounded by the quality of the source data. With no time tracking in the tracker, it cannot report effort or capacity, and it says so instead of estimating. Incomplete due dates mean some tasks genuinely cannot be judged late or early. A 0% review rate is a signal about a team's habits, not about the quality of their work, and the assistant is careful not to present it as the latter.
What This Demonstrates
The interesting work here was not wiring up an LLM — it was deciding what the LLM is not allowed to do. Keeping every number on a path the model cannot touch, making each aggregate carry its own denominator, refusing to rank incomparable roles together, and giving the reranker a degraded path that announces itself: those are the decisions that make the output trustworthy enough to act on. Most of the rules in the system prompt exist because of a specific way an earlier answer went wrong.