When your AI confidently miscounts your risk register

Ask an AI assistant connected to your GRC platform “how many of our vulnerabilities are exploitable?” and you will get a number. It will be phrased with complete confidence. Until v3.21.1, there was a decent chance that number was wrong — and not wrong in a way anyone could see.

That bug is the reason we stopped adding tools to ca_mcp, our Model Context Protocol server, and rebuilt its architecture instead. This post covers what we found, what we changed, and the parts we think generalise to anyone exposing a real business system to an agent.

The rewrite also lifted a practical limit. Claude Desktop, Claude Code, LM Studio and OpenWebUI have been able to reach CISO Assistant for a while; ChatGPT and Microsoft Copilot Studio could not, because they run in the vendor’s cloud and cannot launch a local process. A Streamable HTTP transport closes that gap — so the platform is now reachable from the assistant your organisation has already standardised on, whichever one that is.

Worth being clear about what that does and does not mean. CISO Assistant is a GRC platform, not a chatbot, and none of this moves the product into a chat window. The web application remains where the work happens — the registers, the audits, the evidence trail, the workflows. MCP is an additional way in: a read path for the questions people ask in passing, and a write path for the bulk operations that are tedious in a form. The assistant is a capability on top of the platform, not a replacement for it.

Problem one: most of the product wasn’t reachable

The original server followed the obvious pattern: one hand-written tool per object type, per verb. get_risk_scenarios, get_assets, get_applied_controls, and so on. It works, it produces good tool descriptions, and it does not scale. CISO Assistant’s API exposes 76 collections. Roughly 45 of them had no tool at all.

The gaps were not exotic corners. The entire privacy app — processings, personal data, data subjects, data recipients, transfers, breaches, right requests — was unreachable. So was document management. So were questionnaires. If you had asked the assistant about your GDPR records, it could not answer, and the conversation would drift into something adjacent that it could do.

The fix was to stop writing a tool per object and add three generic ones — list_objects, get_object and count_objects — over a registry that maps an object type to its API path. A new type becomes reachable by adding one line. The dedicated tools stay where they earn their keep, with tailored columns for the well-trodden objects; the registry covers the long tail. The server now exposes 105 tools spanning risk, compliance, assets, TPRM, EBIOS RM, privacy, findings and evidences, document management, and the new threat-model and TTP catalogs.

One thing the registry deliberately does not cover: credential and session material. Workflow secrets, workflow tokens, sessions, IdP groups and global settings are excluded by construction, not by permission check. Generic access is a convenience for reading your GRC data, and there is no version of “the agent enumerates your secrets” that is a feature.

Problem two: the confidently wrong number

Now the interesting one.

Every list response an MCP server returns is bounded. It has to be — an unbounded register would blow through the model’s context window and cost a fortune doing it. Ours caps at 100 rows per call by default.

An agent asked “how many vulnerabilities do we have?” does the natural thing: it calls the list tool and counts what comes back. It sees 100 rows. It answers “100”. You have 1,592.

Nothing errors. Nothing looks truncated. The model isn’t hallucinating — it counted accurately, over the wrong set. And in GRC, a plausible wrong number is considerably worse than a refusal, because it ends up in a steering committee slide.

Two changes address it. count_objects asks the API for an exact server-side count and returns numbers only, never rows — for totals, breakdowns by status or severity, proportions, distributions. It costs almost nothing in context regardless of register size, and it is correct by construction. And every list response now states its own bounds:

Found 100 of 1592 vulnerabilities (rows 1-100; pass offset=100 for the next page)

A truncated answer can no longer be mistaken for a complete one — by the model, or by the person reading over its shoulder. Counts and percentages are computed over the whole set regardless of the display limits.

The general lesson: when a tool returns a bounded view of an unbounded set, the boundary has to be in the response. An agent cannot infer what it wasn’t shown, and it will not think to ask.

Problem three: more tools is not more capability

Expanding coverage cuts against something else. Every tool definition sits in the model’s context on every single turn, and past some surface size, selection accuracy starts to slide — models begin picking near-miss tools, especially smaller local ones.

Rather than argue about this from intuition, we built two harnesses and shipped them in the repo.

tools_selection_harness.py pulls the real tool definitions out of ca_mcp, converts them to function specs, and asks a local model fixed GRC questions — so the read-only surface and the full surface can be compared on identical prompts. Several prompts are deliberately near-miss prone, because those are where it actually fails: risk scenarios versus quantitative risk scenarios, “list the domains” against a tool named get_folders. There is a French tier too, since a tool description written in English and a question asked in French is exactly the seam where selection degrades.

multiturn_harness.py covers what the single-turn version can’t: each conversation opens with the heaviest read available, then keeps going, so later turns are answered with a context already full of tool output. It runs with response bounding on and off, which turns “response bloat degrades later turns” from a hypothesis into a measurement you can reproduce.

The resulting defaults — 100 rows per call, 200 items when a tool paginates internally, 20,000 characters per response — are configurable, but they are chosen rather than guessed. If you run a local model against your own instance, run the harness against it; the number that matters is the one from your setup, not ours.

Problem four: HTTP, and why stdio is still the default

The Streamable HTTP transport is the part of this release with the widest reach, and it exists for exactly one reason: the cloud-hosted clients cannot spawn a local process.

For every client that can — Claude Desktop, Claude Code, LM Studio, OpenWebUI, Cursor — stdio remains the default and the recommendation. Nothing listens on the network. Every API call leaves from your own machine, through firewall rules and proxies you already run. No credential crosses a network boundary between client and server. That is a materially smaller attack surface, and we would rather say so plainly than market the newer transport as the better one.

When you do enable HTTP, the defaults reflect what we wrote about MCP’s real attack surface earlier this year:

• Read-only by default. The HTTP endpoint exposes only the 48 read tools. Write tools require setting CA_MCP_READ_ONLY=false — a deliberate act, because an agent driven by a third-party orchestrator would then be able to modify your GRC data.
• Every caller brings their own token. The server holds no credential of its own. Each request carries the caller’s Personal Access Token and runs with exactly that user’s permissions and domain scope. A request without a token is rejected, not served under a shared identity. (There is an escape hatch for single-user deployments; it collapses every caller into one identity and one audit trail, and the docs say so.)
• Loopback bind, with Host validation. The server listens on 127.0.0.1 unless told otherwise, and CA_MCP_ALLOWED_HOSTS checks the Host header of every request. You put your own HTTPS reverse proxy in front, keeping TLS, certificates and access logging in infrastructure you already operate.

If you deploy with Docker Compose, the MCP server ships as an optional service behind a mcp profile, routed through your existing proxy with no new published port:

docker compose --profile mcp up -d

What this looks like in practice

The AI work that landed alongside this in v3.20.x is part of the same arc: spreadsheet import through the chat window, risk assessment import, richer page context so an in-window session knows which audit you are looking at, and maturity scores over MCP. Different surfaces, same underlying principle: the assistant should be able to reach your data, should say what it actually saw, and should never quietly answer from a partial view.

Setup takes about ten minutes — a Personal Access Token, a config file, one JSON block in your client. Full instructions, including the ChatGPT and Copilot Studio specifics, are in the MCP setup guide. The server is in cli/ in the community repository, Apache-licensed like the rest of it.

If you find a question where the assistant answers confidently and wrongly, that is a bug worth an issue. Those are the ones we most want to hear about.