About Projects Experience Contact
Back to Projects

AI systems case study / built during an AI engineering internship

DeliveryLens

An AI assistant for engineering delivery questions — project status, who is blocked, how the team is tracking. The model writes the answer, but it never carries a number: every measurement comes out of SQL, and the charts are rendered from the same JSON with no model involved.

16 typed tools, every measurement from SQL
0 numbers written by the language model
~6s 3 rounds, 2 tool calls for a status question
2 retrieval paths fused, then reranked

Why This Project Exists

Delivery questions are annoying to answer because the truth is spread across systems. Commit and pull-request activity lives in GitHub. Tasks and due dates live in a task tracker. The reasoning — why a decision was made, what the stated risks are — lives in documents nobody rereads. Answering "how is this project going?" means joining three sources by hand.

DeliveryLens does that joining ahead of time. A scheduled job copies GitHub and task-tracker data into a local Postgres database, a separate ingest indexes PDFs and pull-request bodies into a searchable corpus, and questions are answered entirely from those local copies. The model never calls GitHub or the tracker directly. It only calls tools, and every tool reads the mirror.

The constraint that shaped everything: the model behind this is a small, cheap one, chosen mainly for its rate limits. Early on it mis-transcribed a year while summarising a tool result — the underlying data was right, the model's retelling of it was not. That single failure produced the main rule in the codebase.

The Rule: The Model Writes The Text, Never The Number

Every tool result goes to two places. One copy goes to the model, which writes the prose. The other goes to a view builder that renders charts and tables straight from that same JSON, with no model in the path.

flowchart LR
    T["Tool result JSON"] --> M["AI model"] --> P["Prose on the screen"]
    T --> V["View builder (fixed code)"] --> C["Charts and tables on the screen"]
                    

So when COUNT(*) FILTER (WHERE is_overdue) returns 7, the 7 on screen came from the query. The model never retypes it. The system prompt also forbids markdown tables, because the real figures are already rendered above the text.

This is what makes a small model good enough. It has to pick the right tool and explain what the result means. It never has to be trusted with arithmetic.

Architecture

Data flows one way: external services into the database, database into the answer. Two reasons that matters. One question can need several database lookups, and doing that against live APIs would be slow and would burn through GitHub's hourly rate limit. And a tool result is a fact read out of a table, not something the model can misremember.

flowchart TD
    GH["GitHub API"]
    CU["Task tracker API"]
    PDF["PDFs and PR bodies"]
    SYNC["Sync services: watermarked, branch-aware"]
    ING["Corpus ingest: chunk then embed"]
    DB[("Postgres: Neon + pgvector")]

    GH --> SYNC --> DB
    CU --> SYNC
    PDF --> ING --> DB

    Q["Question"] --> AGENT["Agent loop"]
    AGENT -->|"picks a tool"| TOOLS["16 typed tools"]
    TOOLS --> INS["Insight layer: SQL aggregates"]
    TOOLS --> RET["Hybrid retrieval: vector + full-text, fused, reranked"]
    INS --> DB
    RET --> DB

    TOOLS -->|"result JSON"| LLM["Model"]
    TOOLS -->|"same result JSON"| VIEW["View builder (no model)"]
    LLM -->|"text"| UI["Answer on screen"]
    VIEW -->|"charts, tables"| UI
                    

How A Question Turns Into An Answer

A status question runs about three rounds and two tool calls, roughly six seconds end to end. The model is told to always resolve a name before measuring anything, because people ask about "the retail thing" and that has to map to a real project row first.

sequenceDiagram
    participant D as User
    participant API as /api/v1/chat
    participant A as Agent loop
    participant T as Tools
    participant DB as Neon
    participant M as AI model

    D->>API: "How is this project going?"
    API->>API: Check session and allowlist
    API->>A: Start the agent loop
    A->>M: Question, plus 16 tool definitions
    M-->>A: Call resolve_entity
    A->>T: Run the tool
    T->>DB: SELECT
    DB-->>T: Rows
    T-->>A: Result
    A-->>D: Event: working out what you mean
    A->>M: Tool result
    M-->>A: Call get_project_status
    A->>T: Run the tool
    T-->>A: Result plus chart views
    A-->>D: Charts and tables
    A->>M: Tool result
    M-->>A: Text
    A-->>D: Text, one part at a time
                    

The status query joins tracker tasks with GitHub commits and pull requests and returns one structured result plus a provenance block: when each source last synced, and how many accounts are still unmatched. The client never sends conversation history back — just the new message and a conversation id. The server reads earlier turns from the database, so refreshing resumes the thread and a client cannot rewrite what was already said.

Engineering Highlights

1. Measurements and documents are kept apart on purpose

Problem: the two sources describe the world at different scales. The financial documents describe a company with thousands of customers; the operational data describes a handful of projects and a small team. "How many people do we have?" therefore has two different correct answers depending on which source you ask.

Implementation: counts and rates come from SQL; prose comes from the corpus. The assistant always states which source it used, and it never adds a document figure to a measured one. Mixing them would produce a number that is arithmetically fine and semantically meaningless.

2. Every result carries its own caveats

Problem: an aggregate without its denominator invites a wrong reading. A zero is the worst case: "no GitHub account linked" and "did not write any code" look identical, but only one of them is a performance signal.

Implementation: get_overdue_tasks also reports how many tasks have no due date and so cannot be judged either way. rank_developers returns its formula, its time window, who was excluded, and a note on any row where one data source contributed nothing. The caveat travels with the number instead of living in documentation nobody reads.

3. Rankings stay inside their own groups

Problem: a project manager will never have commits. Scoring everyone on one combined number puts every non-engineering role at the bottom by construction — that is a fact about the job, not about the person.

Implementation: code-shipping roles and non-code roles are ranked separately, and the charts are drawn per group rather than combined, since combining them would imply a comparison the data cannot support. Anyone with zero activity in the window is listed separately instead of ranked, because no data is not the same as bad performance.

4. Retrieval and reranking are two separate steps

Problem: plain vector search misses exact terms — a similarity search can skip straight past the passage that names a specific metric. Keyword search misses paraphrasing. Either one alone loses real answers.

Implementation: a pgvector/HNSW vector search and Postgres full-text search run together, their rankings are fused by reciprocal rank, and the combined list goes to a Cohere cross-encoder that reads the query and each passage together and scores them. The per-document cap is applied after reranking, so it never reorders results — it only stops one long document from taking every slot.

flowchart LR
    Q["Question"]
    V["Vector search: top 30"]
    K["Keyword search: top 30"]
    F["Fuse by reciprocal rank"]
    R["Cohere cross-encoder rerank"]
    C["Cap 2 per document"]
    OUT["Passages shown"]
    Q --> V --> F
    Q --> K --> F
    F -->|"pool of 4x limit"| R --> C --> OUT
                    
5. The reranker can go down without breaking the answer

Problem: the reranker is a third-party API call on the critical path of every document question. If it is the single point of failure, a quota problem takes out retrieval entirely.

Implementation: if the Cohere call times out or errors, retrieval falls back to selecting a diverse set from the vectors it already has — no extra API call, just arithmetic, which is what matters when quota is the limiting factor. The result records which path ran (cross-encoder or diversity-fallback) and the note passed to the model changes accordingly, so a weaker answer gets flagged rather than shipped silently.

6. Identity links are confirmed by a person, never guessed

Problem: GitHub handles and tracker accounts follow no shared naming pattern. An automatic wrong link would silently attribute one person's work to someone else, and nothing downstream would catch it.

Implementation: the identity service scores likely pairs and surfaces the busiest unmatched accounts first, but it only ever suggests. A human confirms. Bot accounts are filtered on both handle and display name, since some bots commit under a normal-looking login but a [bot]-tagged name.

7. Authorization is separate from authentication, and fails closed

Problem: the auth provider allows open sign-up, so a successful login says nothing about whether you should see the data.

Implementation: a separate allowlist decides access, and an empty allowlist locks everyone out rather than letting everyone in by mistake. That check deliberately does not depend on the auth library or on the framework, so the preflight script can test it directly by feeding in bad addresses and confirming they are rejected.

The Agent Loop

flowchart TD
    S["Start: system prompt plus question"] --> R["Send to the model"]
    R --> C{"Did the model ask for tools?"}
    C -->|No| T["Send the text. Stop."]
    C -->|Yes| E["Run each tool"]
    E --> V["Build the charts from the result"]
    V --> A["Add the result to the conversation"]
    A --> L{"Round limit reached?"}
    L -->|No| R
    L -->|Yes| X["Report the limit. Stop."]
                    

Three Bugs Worth Writing Up

Parallel tool calls landing on the same stream position

One provider streams a single tool call in fragments, each chunk tagged with a position index. Another sends multiple complete calls that all report the same position, distinguished only by an id. Keying purely by position concatenated two calls into one broken JSON string, which the provider then rejected with an unhelpful 400. The fix keys by call id when one is present, falling back to the position index only for a continuation of a call already in progress — which covers both providers' behaviour.

A 65-branch repository exhausting the hourly rate limit

A lot of work lived on personal branches that were never merged, so syncing only the default branch missed most of it. The first attempt walked the full history of every branch on every sync, which made the rate-limit problem worse rather than better. The actual fix: sync the default branch first so its history is already stored, then skip any branch whose latest commit is already in the database.

Correct SQL, misleading answer

get_overdue_tasks filtered its results by project but computed the "how many tasks have due dates" total against the whole workspace — so "6 overdue out of 35 with due dates, out of 51 total" mixed one project's overdue count with the entire organisation's task count. Every number was correct on its own; together they were misleading.

Separately, renameConversation and deleteConversation checked ownership correctly in their SQL, then decided success with result.length > 0. TypeORM returns [rows, affectedCount] for updates and deletes, so that check was always true regardless of who owned the row. Reading the code would not have caught either bug. Tests that asked "does the denominator match the filter?" and "can another user delete this?" found both immediately.

Tradeoffs And Honest Limits

Answers are only as fresh as the last sync. That is the deliberate cost of not calling the live APIs at question time, and it is why every result carries a provenance block saying when each source was last copied — a stale answer should announce itself rather than look current.

The system is also bounded by the quality of the source data. With no time tracking in the tracker, it cannot report effort or capacity, and it says so instead of estimating. Incomplete due dates mean some tasks genuinely cannot be judged late or early. A 0% review rate is a signal about a team's habits, not about the quality of their work, and the assistant is careful not to present it as the latter.

What This Demonstrates

The interesting work here was not wiring up an LLM — it was deciding what the LLM is not allowed to do. Keeping every number on a path the model cannot touch, making each aggregate carry its own denominator, refusing to rank incomparable roles together, and giving the reranker a degraded path that announces itself: those are the decisions that make the output trustworthy enough to act on. Most of the rules in the system prompt exist because of a specific way an earlier answer went wrong.