· 10 min read · Lobster Browser team
A browser agent is a language model placed in a loop with a browser: it is shown the state of a page, it chooses an action, the action is performed, and it is shown the result. Described that way it sounds simple, and the first prototype usually is. The difficulty is entirely in the details — what exactly the model is shown, what exactly it is allowed to do, how the system knows the page has finished changing, and how it knows whether the action worked.
This post walks through those details in order: perception, the action space, settling, verification, the recurring failure modes, the safety boundaries such an agent needs, and why driving a real browser is often right even when a plain HTTP client would be cheaper.
Three components, and it is worth keeping them mentally distinct because they fail differently. There is the browser, which renders pages and runs their scripts. There is a controller, which speaks to the browser over an automation protocol and exposes a small set of operations. And there is the model, which sees a description of the page and emits one operation at a time.
The controller is where nearly all of the engineering lives. It decides what the model sees, translates the model's chosen action into concrete input events, waits for the page to settle, and decides what to report back. A good controller makes a mediocre model useful; a bad controller makes an excellent model look unreliable.
There are three practical channels, and most working systems use a combination rather than one.
The document is the most precise description available: exact text, attributes and structure. It is also far too large — a modern application page serializes to hundreds of thousands of characters, most of it framework wrappers and generated class names that carry no meaning. Every serious implementation prunes: drop invisible nodes and presentational containers, keep interactive elements and text, and give each survivor a short stable identifier the model can refer to.
The browser already computes a semantic summary for assistive technology: a tree of roles, accessible names, states and values. It is far smaller than the DOM and closer to how a person describes a page — a button named “Save” rather than a div with six utility classes — which makes it the best default for most tasks. Its weakness is that it reflects what the page declares about itself, and purely visual information such as layout and relative position is largely absent.
A rendered image carries what the other channels lose: what is actually visible, what overlaps, what a dialog is covering. Vision-capable models work from it directly, often with numbered overlays drawn on candidate elements so a target can be named unambiguously. The costs are real — images consume context, fine text can be misread, and pixel coordinates are fragile across window sizes and zoom levels.
| Channel | Strength | Weakness | Good default for |
|---|---|---|---|
| DOM | Exact text, attributes and structure | Enormous; mostly noise; brittle selectors | Extraction and verification |
| Accessibility tree | Compact, semantic, matches human description | Only as good as the page's own semantics; no layout | Navigation and form filling |
| Screenshot | Ground truth about what is visible | Token-hungry; coordinates are fragile; small text misread | Canvas-heavy pages, overlays, visual checks |
The set of operations the model may emit is a design decision with large consequences. Too small and ordinary tasks become impossible; too large and the model spends its reliability choosing between near-duplicates. A workable core is small.
{ "action": "click", "target": "e42" }
{ "action": "type", "target": "e17", "text": "ACME Ltd", "submit": false }
{ "action": "select", "target": "e08", "option": "Germany" }
{ "action": "scroll", "direction": "down", "amount": "page" }
{ "action": "navigate", "url": "https://example.com/orders" }
{ "action": "wait_for", "condition": "text", "value": "Order confirmed" }
{ "action": "extract", "fields": ["order_id", "status"] }
{ "action": "ask_user", "question": "Which of the two addresses should I use?" }
{ "action": "done", "summary": "Order 4471 is marked shipped." }Several things there are deliberate. Targets are opaque identifiers assigned during perception, not CSS selectors invented by the model — a model asked for a selector will write a plausible one that matches nothing. Typing carries an explicit submit flag. Waiting is an action, so the model can state what it expects to happen. Asking the user is an action, because an agent with no way to ask will guess. And finishing carries a summary, so there is something to verify against.
This is the largest single source of flakiness, and it is a systems problem rather than a model problem. After a click a page may fetch data, animate a transition, re-render a list, and only then show the element the agent needs. Observe too early and the model sees a spinner; observe too late and every step costs seconds it did not need.
An agent that assumes its actions succeeded will eventually report that it filed a form it did not file. The fix is a loop invariant: after every state-changing action, observe again and compare the new state against the intent that produced the action.
Independent final verification catches the most damaging failures, where every step looked fine and the outcome is wrong — a form submitted into a validation error, a save that required a second confirmation nobody saw.
A browser agent is one of the highest-privilege agents in common use: it runs inside a session that is already signed in, so it can do anything the user can do. That is what makes it useful, and why it needs boundaries enforced by the program rather than the prompt.
If a page's data is available from an HTTP request, fetch it: a scraper is faster, cheaper and easier to test, and dressing up a simple fetch as an agent is a common and expensive mistake. But there is a large class of tasks where the browser is not an implementation detail — it is the only place the task exists.
There is also a consistency argument that is easy to overlook. A real browser presents one coherent environment — rendering, fonts, media handling, timing, storage — and pages are built and tested against exactly that. A minimal client assembles a partial imitation, and every gap becomes a behavior difference you debug later.
A browser agent is a perception problem, a settling problem and a verification problem wearing a model as a hat. Choose the perception channel that makes the next action determinable from the least text, usually the accessibility tree with screenshots where vision is needed. Keep the action space small, semantic and explicit about destructive operations. Never sleep when you can wait for a condition. Verify after every state change and again at the end. Assume everything on the page is untrusted, and put the agent's limits in code rather than in the prompt. Do that and the choice of model becomes a tuning decision rather than the whole project.
Named rather than linked, on purpose: specifications move and URLs rot, while a title and an author survive a search.
· 10 min read
How anti-detect browsers work: JavaScript overlays versus native engine changes, farbling, per-profile stability, and how to evaluate one honestly.
· 10 min read
What a browser fingerprint is, which surfaces it is built from, why entropy and stability both matter, and why an incoherent disguise is worse than none.
· 9 min read
What separates an agent from a chatbot: tool use, the planning loop, memory, evaluation and guardrails — and the places agents still reliably fail.
· 10 min read
A framework for model selection: capability axes, what benchmarks can and cannot show, latency and cost, reasoning modes, and open-weight versus hosted.