An agent over a mailbox: what it takes, and where it breaks

An owner asks a reasonable question: can we have an assistant that answers questions about our shared inbox? Not search — answers. Which visa did we get for that Bali trip and how many days did it run? The information exists. It sits in a PDF attached to an email from a colleague, buried among eight signature images, in a thread whose subject line mentions neither visas nor days.

I built exactly that, on my own mailbox, in one evening. It works. This article is the whole build: the architecture, every configuration that matters, the seven things that broke, and the three limits I did not remove. If you follow it, you will have the same assistant. More usefully, you will know what it costs and where it stops.

The tools are n8n, the Anthropic API, and Telegram. Nothing else. No database, no vector index, no migration.

Why the obvious approach fails

The instinct is to build a pipeline: pull the whole archive out of the mail server, parse it, chunk it, embed it, store the vectors, then search that store semantically. It is the architecturally correct answer and it is what I started building. It is also a week of work before anything answers a single question, and every hour of it is spent on plumbing that has nothing to do with whether the assistant is useful.

The alternative is to give an agent the search that already exists — the mail provider’s own — and let the agent operate it. Instead of one perfect query against a perfect index, the agent runs six imperfect queries against an ordinary one, reads what comes back, and tries again with different words. It compensates for a weak tool with iteration, which is the thing agents are actually good at.

That trade is the whole design. You lose true semantic retrieval. You gain a working system the same day, and you find out within an hour which of your assumptions about the work were wrong. On my archive, the agent’s own query reformulation covered most of what an embedding index would have covered, and the cases where it did not were visible immediately rather than after a week.

The architecture

Seven components. Four of them are the agent’s tools.

Component	What it does
Telegram trigger	Receives the question. The messaging app is the interface — no UI to build, works from a phone, and access is controlled by one account check.
Agent	Claude Sonnet 5. Decides which tool to call, reads results, decides whether to call again, composes the answer.
Memory	Holds the last few exchanges so follow-up questions work without repeating context.
Tool: search	Runs a query against the mail provider’s search. The query string is written by the agent, not by the user.
Tool: read	Fetches one message in full by its identifier.
Tool: read attachment	A separate sub-workflow: downloads the message, selects the PDF among the attachments, extracts its text.
Tool: calculator	Arithmetic. Not optional — see below.

The loop is the part worth understanding. The agent receives the question, calls a tool, gets a result, and decides on the basis of that result whether it has enough to answer or needs another call. Nobody scripts the sequence. On a question about a trip, the agent might search for the country, get nothing, search for the city, get a thread, read the thread, notice an attachment, open the attachment, and only then answer. Six calls, none of them planned in advance by me.

This is also the honest reason to build the loop rather than buy a chatbot: you can watch it. Every tool call and every result is in the execution log. When the answer is wrong, the log tells you whether the agent searched badly, read the wrong thing, or reasoned poorly over good data — three different problems with three different fixes.

Access: the decision that costs the most later

The mail provider gives programs access through an application you register yourself. Two choices in that registration matter more than anything else in the build.

Audience. An application registered as external and left in testing status has authorisations that expire after seven days, taking the refresh token with them. The assistant will work all week and be dead on Monday, with no error that explains why. Registered as internal — available when the domain runs on a business account — there is no expiry and no verification review. This is a one-way choice: changing it later means a new project.

Scope. The mail integration built into the automation platform requests a broad permission set, including modification and deletion. The alternative is a generic credential with a single read-only scope, calling the mail API directly. The second path costs a few extra hours and removes an entire category of risk: the application cannot delete or send, regardless of what the agent decides.

I recommended the narrow path and then, under time pressure, built the first version on the broad one. That is worth stating plainly rather than hiding, because the reasoning on both sides is real. On my own mailbox, with no tool attached that can send or delete, the practical exposure is nil. On a client’s shared inbox it is not, and the answer to what can your agent do to our mail should be a list of one permission, verifiable on the consent screen in a second, rather than an assurance.

The seven failures

This is the useful part. Every one of these cost time, and none of them is in the documentation.

1. The search tool ignored the search. The agent reported, correctly and repeatedly, that its search tool returned the same fifteen recent messages no matter what it asked for. The search parameter existed in the tool but was not marked as model-controlled, so the agent’s carefully composed queries went nowhere. The tool ran, returned plausible data, and told nobody it was ignoring its input.

The general form of this failure: a tool that silently discards its arguments is worse than a tool that errors, because the agent will construct an explanation for the useless results and that explanation will sound reasonable. Read the agent’s complaints literally. Mine said the search was not filtering. It was right.

2. Formatting broke the delivery. The agent’s answers contained asterisks and underscores. The messaging platform tried to parse them as markup, hit an unterminated span, and rejected the message. The agent had answered correctly; the answer never arrived. Fixed by instructing the agent to write plain text and removing the parse mode.

3. Memory expected a session identifier that did not exist. The conversation memory component defaults to reading a session key produced by chat-style triggers. A messaging trigger produces no such field. The fix is to define the key explicitly — the chat identifier serves — but the default configuration fails with an error that names the missing field without saying where to get it.

4. Attachments are not text and cannot be passed to a model. This one is structural rather than incidental. A tool returns text to the agent. A PDF is bytes. Bridging that requires a separate workflow that downloads the message, picks the right file, extracts its text, and returns the text — invoked by the agent as a tool like any other. The agent never touches the file.

5. The PDF was the ninth attachment. The email had nine attachments: eight signature images and, last, the visa document. Code that takes the first attachment gets a logo. Hard-coding the ninth works once and fails on the next message. The extraction has to select by file type, not by position.

6. Selecting by type is harder than it looks. Splitting the attachments into separate items preserved their original names — attachment_0 through attachment_8 — rather than normalising them. A filter written against the expected name matched nothing, the workflow returned no rows, and the agent got back the workflow did not return a response. Two intermediate attempts failed before the filter read the file type from whatever the field happened to be called.

7. The sub-workflow was not published. The agent called its attachment tool and received workflow is not active and cannot be executed. It reported this accurately and fell back to the email body. Ten seconds to fix, once the message was read rather than skimmed.

Seven failures, and the pattern across them is one thing: the agent was right every time and the plumbing was wrong every time. Not once did the model hallucinate a visa. It said the search was not filtering, that the attachment tool was unavailable, that the duration was in a file it could not open. Every report was accurate. The debugging consisted of believing it.

Two constraints that do the real work

Never let the model do arithmetic. Asked how many days remain on a permit, a language model will produce a number, and the number will be confident and occasionally wrong. Two things fix this: give the agent the current date in its instructions — it does not know what day it is — and give it a calculator tool with an instruction to use it for anything involving dates. Then require it to state the input dates and their source before computing. The output stops being a number and becomes a derivation you can check.

This generalises past dates. Any quantity a client will act on should be computed by something deterministic and shown with its inputs. The model’s job is finding and extracting the inputs, which is the part it is good at.

Require the gap to be named. The instruction that produced the best behaviour in this build was the requirement to state, as part of every answer, what was not found and what document would be needed to answer properly. Twice the agent used it correctly: once when the attachment tool was broken, once when the email body gave the visa type but not the duration. Both times it stopped rather than guessing, and both times the gap it named was the actual bug.

An assistant that says I do not have this, and here is what would give it to me is more valuable than one that is right slightly more often, because you can act on the first and only trust the second.

What it does not do

Three limits remain. Naming them is not a caveat; it is what makes the rest credible.

No semantic search. The agent finds messages by words, and compensates by trying several formulations — synonyms, English and Russian, sender addresses, date ranges. Instructing it to try at least three phrasings before concluding that nothing exists closed most of the gap. It will still miss a thread that discusses the subject in vocabulary nobody would guess. Removing this limit means building the index I described at the start: extract, chunk, embed, store, retrieve. It is a day of work and it is the correct next step — after the simple version has shown you which questions people actually ask.

No scanned documents. Text-layer PDFs are read. A photograph of a document is not. The fix is small — pass the image to the model directly, which handles documents natively — but it was not needed for this build, so it was not done.

One mailbox, one person. A shared corporate inbox adds questions this build does not answer: who may ask what, how permissions map onto conversations, what is logged. Those are governance questions, not technical ones, and they belong to a different conversation than does the assistant work.

What this cost, and what it means

An evening. One API key, one bot, one mail credential, and about a dozen configuration screens. The running cost is cents per question.

The point is not that assistants over email are easy. It is that the distance between we should look into AI for the inbox and a working thing that answers with sources is one evening — and that the entire difficulty lies in plumbing, not intelligence. Every single failure above was a wiring problem. The model found the right email among years of archive, identified the right file among nine, read a visa document, extracted the type, number, issue date and duration, and separated what it knew from what it inferred.

Which is the argument this practice makes in general. The models are not the constraint. The constraint is that nothing in a company is structured for them to act on, and the work is the structuring — done in small, verifiable pieces, each one running before the next is started.

If you want to reproduce this: build the agent first and the infrastructure second. I did it the other way round and spent the first half of the evening on a data pipeline I did not use.