MCP::Server::Tool::Web

A web tools (search, fetch, crawl, grep) toolkit for MCP::Server.

Four tools — web_search, web_fetch, web_crawl and web_grep — under one rule that runs beneath all of them: a URL a model supplied is fetched only after the address of the socket that was actually opened has been checked, so a hostname that resolves onto the host machine's own network is refused whatever it is spelled like. See Security.

Synopsis

Plug it into a server you are already building:

use MCP::Server;
use MCP::Server::Tool::Web;

my $server = MCP::Server.new(:name<research>, :version<1.0>);

$server.plug: MCP::Server::Tool::Web.new;

$server.run;   # registers web_search, web_fetch, web_crawl, web_grep

Or assemble a server out of toolkit packs in one shot, with no glue code — the shape a host's JSON config takes:

MCP::Server.new(
    :name<research>,
    :tools[
        'Web' => {
            provider        => 'brave',
            'api-key-env'   => 'BRAVE_API_KEY',
            'max-bytes'     => 4 * 1024 * 1024,
            'total-timeout' => 90,
            'crawl-delay'   => 0.5,
            'allow-hosts'   => <wiki.corp docs.corp>,   # exact names only
        },
    ],
).run;

Or run it straight off the raku-mcp command line that ships with MCP::Server:

raku-mcp --tool=Web='{"crawl-delay":0.5,"max-bytes":4194304}'

raku-mcp --config=mcp.json

{
  "name": "research",
  "instructions": "Search and read the web. Prefer web_grep over web_fetch when you know what you are looking for.",
  "tools": {
    "Web": { "crawl-delay": 0.5, "allow-hosts": ["wiki.corp"] }
  }
}

Wiring that into Claude Code:

{
  "mcpServers": {
    "research": {
      "command": "raku-mcp",
      "args": ["--config=/home/me/mcp.json"],
      "env": { "BRAVE_API_KEY": "..." }
    }
  }
}

Description

web_search answers JSON so the model can pick a URL out of it; web_fetch, web_crawl and web_grep answer plain text, because a page — or a report about pages — is text. Nothing here needs a browser: pages are fetched over plain HTTP(S) and their markup, scripts and navigation are stripped by a hand-rolled extractor (MCP::Server::Tool::Web::Extract), not rendered.

web_search needs a search provider's API key to answer anything; web_fetch, web_crawl and web_grep need nothing beyond network access, and construct successfully even with no key configured anywhere — see [Search provider](#Search provider).

Tools

Parameter names are a contract, not a matter of taste: a policy layer in front of the server (such as MCP::Client::Policy) matches calls by them to decide what a call would reach, so url stays url and multi-word parameters stay kebab-case.

The Required column of every table below is the published JSON Schema's required array, and every "defaults to" in them is that property's JSON Schema default. The model reads both before it calls: a schema that required nothing would say web_fetch can be called with no url, and a schema that hid the defaults would leave the model unable to tell a web_grep of one page from a web_grep that crawls.

All four are annotated readOnlyHint => True and idempotentHint => True . Every one of them is a read — a search asks an engine a question, a fetch, a crawl and a grep ask origins for pages, and nothing in this pack ever submits anything — so a host may run a batch of them side by side, and may ask for one URL once however many slots wanted it (MCP::Server's execute-tool-calls does both). That is a claim about the handlers as much as the intent: everything a call touches is either its own (a budget per call, a crawl per call, a page's text) or carries a lock of its own — the transport's cache of vetted connections, the robots.txt cache, and the search provider's HTTP client.

web_search

The search engine, behind a seam. Answers JSON so the model can pick a URL out of it. Zero results is an empty list and a notice, not an error.

# web_search — JSON, so the model can pick a URL out of it
{"count":10,"provider":"brave","query":"raku grammars",
 "results":[{"snippet":"…","title":"Grammars","url":"https://docs.raku.org/…"}]}

# a query that matches nothing:
{"count":10,"notice":"No results for 'asdkfjhasdkfjh39847'. The engine matched nothing: try fewer or more common terms, or drop any quoting.",
 "provider":"brave","query":"asdkfjhasdkfjh39847","results":[]}

web_fetch

One page, as text: a # title line, the final URL when a redirect moved it, then the page with its markup, scripts and navigation removed. Exactly one parameter, deliberately: a windowing tool would invite the model to page through a document it should have grepped instead. Only http and https URLs can be fetched, redirects are followed (every hop re-checked — see Security), and a page bigger than max-bytes comes back cut short and says so on its last line.

# web_fetch — plain text, because a page is text
# Grammars
Retrieved from https://docs.raku.org/language/grammars

Grammars are a powerful tool used to destructure text …

# a page cut short by the byte cap:
[truncated: kept the first 2097152 characters because the server's byte cap of 2097152 bytes was reached; the rest of the page was not fetched]

web_crawl

Follows the links from a page and returns an index of what is there — never the pages themselves. Only links on the same origin as the starting page (same scheme, host and effective port) are followed; depth counts hops from the starting page; robots.txt is respected by default (see Security). See MCP::Server::Tool::Web::Crawl for the walk itself.

# web_crawl — an index, not the pages
Crawled 3 pages from https://docs.example.com/ (depth 1, same origin only).

https://docs.example.com/  200  4213 bytes  Documentation
https://docs.example.com/install  200  2288 bytes  Installing

[stopped after 20 pages; raise max-pages to go further]

Why an index rather than the pages: concatenating twenty pages into one tool result is a way to spend a context window without answering anything — what the model would actually receive is page one, half of page two, and a truncation marker. So a crawl reports what is there, one line per page, and the corpus stays reachable through web_grep (with a depth) and web_fetch.

web_grep

fs_grep for the web: search one page, or a whole site at a depth, and return only the matching lines as url:line:text. The pattern is literal text unless regex is set, in which case it is a Raku regex, not a PCRE one. This is far cheaper than web_fetch or web_crawl when you know what you are looking for.

# web_grep — only the lines that matched, exactly as fs_grep reports them
https://docs.example.com/install:42:  zef install Foo::Bar

[stopped at 50 matching lines; raise max-results to see more]

With the default depth of 0 only the URL given is searched; with a depth above 0 the same-origin links are followed as web_crawl does, and no more pages are fetched once max-results lines have matched — which is the whole reason web_grep is cheaper than crawling and then fetching.

Configuration

Every attribute below is a JSON config key — from-config validates against exactly this set, so a typo names the valid keys instead of doing nothing quietly:

MCP::Server::Tool::Web.from-config({ providr => 'brave' });
# Unknown config key(s) for MCP::Server::Tool::Web: providr.
# Valid keys: allow-cidrs, allow-hosts, allow-loopback, allow-private,
# api-key, api-key-env, connect-timeout, crawl-delay, headers-timeout,
# max-bytes, proxy, provider, respect-robots, safesearch, search-country,
# search-lang, timeout, total-timeout, user-agent

• max-bytes has a floor of 4096: below that a page cannot hold a document, and every fetch would report more truncation notice than page.

• timeout, total-timeout, connect-timeout and headers-timeout must all be greater than zero.

• crawl-delay cannot be negative.

• user-agent must not be empty — robots.txt groups are matched against its product token, and an origin cannot report who visited it if it is blank.

• allow-hosts entries are exact host names, not URLs and not patterns: a * or ? in one is a configuration error, refused at construction, because a pattern here would match nothing and look like it matched everything — see Security.

Two more seams exist and are deliberately not config keys, because a Callable or a live object cannot come out of JSON — they are is built private attributes, so .new can pass them and from-config cannot:

• search-provider — anything doing MCP::Server::Tool::Web::SearchProvider, used instead of building the named provider. This is how a test dispatches web_search without a network, and how a host wires up an engine this pack has never heard of. See [Search provider](#Search provider).

• fetcher — a prepared MCP::Server::Tool::Web::Fetcher. Given one, the pack uses it for single-page work and clones it (with the robots policy swapped in) for traversal, so the transport — and its cache of vetted connections — is shared.

Search provider

web_search does not talk to any particular engine directly: it asks whatever object composes MCP::Server::Tool::Web::SearchProvider, a role of exactly two methods —

method name(--> Str:D) { ... }                              # 'brave', 'my-provider', ...
method search(Str:D $query, Int:D $count --> List:D) { ... } # [ {title,url,snippet,age?}, ... ]

— which MCP::Server::Tool::Web composes a concrete implementation of via its provider config key. This distribution ships one: MCP::Server::Tool::Web::Provider::Brave, talking to the Brave Search API's web/search endpoint. A provider that throws is the whole story the model gets: its first line is written to teach, and MCP::Server::Tool::Web's web_search tool surfaces it verbatim as the tool's error text.

The API key is resolved at call time, inside .search, never at construction — so a pack with no BRAVE_API_KEY in its environment still constructs and serves web_fetch, web_crawl and web_grep perfectly well, and only web_search fails, the moment it is actually called, naming the environment variable and the two config keys that would fix it:

web_search needs a Brave Search API key. Set the environment variable
BRAVE_API_KEY, or point "api-key-env" at the variable that holds it, or put
an "api-key" in the web config.

Bring your own engine by passing an object doing MCP::Server::Tool::Web::SearchProvider as search-provider — the constructor accepts it in place of provider, since a live object cannot come out of JSON config.

The search provider is not guard-subject

The search provider deliberately does not go through the SSRF guard described in Security. The endpoint web_search calls is the operator's, named in configuration — not the model's. A SearXNG on the operator's own LAN, or a self-hosted Brave-compatible proxy, is a legitimate thing to point this at, so the search provider talks to its API directly. The guard exists for URLs an attacker — or a confused model — chose via web_fetch, web_crawl or web_grep; a config key set once, by whoever deploys this pack, is not that threat model. If a provider's endpoint is ever made settable from an untrusted source, guarding it becomes that call site's responsibility — the shipped provider intentionally does not.

Security

Every URL a model supplies — to web_fetch, web_crawl or web_grep — passes through the same floor before a byte is sent anywhere.

Connect, then verify the peer

A URL is checked for shape first (scheme, credentials, host spelling, port range — MCP::Server::Tool::Web::Url), then a TCP connection is opened, and only then is the address of the socket that was actually opened checked against policy (MCP::Server::Tool::Web::Guard, enforced by MCP::Server::Tool::Web::Transport). Nothing is written to that socket until the check passes — no request line, no Host header, not even a TLS ClientHello — so a refused connection is a handshake and an immediate reset, and the peer learns nothing about what was going to be asked for.

This kills DNS rebinding rather than narrowing it: the address that is checked is the address of the established connection, so there is no window between "resolve, check" and "connect" for an attacker with a short-TTL DNS zone to win. The usual "resolve, check, then connect" shape has exactly that window; this pack does not use that shape.

What is refused

Anything that is not classified public is refused unless a configuration key says otherwise — loopback, RFC 1918 private ranges, CGNAT, link-local addresses, cloud metadata endpoints, and the handful of other reserved and special-purpose ranges IANA has carved out (documentation ranges, benchmark ranges, multicast, and so on — see MCP::Server::Tool::Web::Addr for the full table and the RFC behind each one). An address that does not parse is refused too: the guard never treats "I could not tell" as "it is fine". The file:, javascript: and similar schemes are refused on shape, before any of this runs, and no allow-list entry can rescue a URL refused on shape — allow-lists widen which machines may be reached, and file:///etc/passwd is not a machine.

The escape hatches

• allow-private — allow every non-public class at once. The blunt instrument: one switch for "this machine may reach its own network".

• allow-loopback — allow loopback only. The narrow key for test rigs and local development servers, so enabling it does not open the whole LAN.

• allow-hosts — exact, lowercase host names, matched exactly. corp permits the host corp and never wiki.corp, and emphatically never evil.com.corp: suffix matching is where allow-list CVEs come from, so there is no wildcard syntax, and a pattern in the list is refused at construction rather than silently doing nothing.

• allow-cidrs — CIDR blocks, IPv4 and IPv6, matched against the peer's actual address (a v4 address written in its v6-mapped form still matches a v4 block).

Every refusal names the rule, the address class, the RFC behind it, the host, the address it resolved to, and the configuration key that would permit it — the audience is a model deciding what to try next, and "denied" with nothing else just makes it try the same thing spelled differently.

Redirects are re-checked, every hop

Redirects are followed by MCP::Server::Tool::Web::Fetcher itself, one hop at a time, rather than by the underlying HTTP client — so a Location: http://10.0.0.1/ or a Location: file:///etc/passwd is refused on the hop that produced it, with the same rule and the same message a first request would have gotten. A redirect that drops from https to http is refused outright, downgrade or not: the bytes that would come back are readable, and rewritable, by anything on the path.

Connection reuse is a security property here, not a risk

The HTTP client is persistent, and that is deliberate: a cached pipeline is an already vetted, still established socket. Rebinding a name cannot retarget an open connection, so reusing one costs nothing in safety and saves a re-check on every subsequent request. The consequence to respect the other way: one client belongs to one guard configuration, and a Transport is never shared between packs configured with different allow-lists.

Fetch truncation is head-only

When a page runs into max-bytes or the deadline, only the head is kept — never a head-and-tail splice. The tail of a truncated HTML document is closing markup: the extractor needs a parseable prefix to produce anything at all, and web_grep's line numbers have to mean something stable. (Compare MCP::Server::Tool::Shell, whose command output keeps head and tail, because the interesting part of a build log is usually at the end — a web page is a different shape of document, and gets a different rule.)

robots.txt

Per the pack's ruling: web_crawl, and web_grep with a depth above zero, respect robots.txt by default (respect-robots, default True); web_fetch and depth-zero web_grep never consult it, regardless of that setting. A page a person asked for by name is not crawling, and no robots.txt has ever meant "this URL may not be read once, deliberately, on a human's behalf" — traversal is a different act, and that one is asked for. Setting respect-robots to False turns it off for the traversing tools too. See MCP::Server::Tool::Web::Robots for the RFC 9309 details (group matching, longest-pattern-wins, Crawl-delay, and why a robots.txt that cannot be fetched at all means "allowed", not "deny everything").

Known limitations

• Static fetch only. These tools issue a plain HTTP(S) GET and parse the bytes that come back; there is no browser anywhere in the code path, so a single-page application whose content is assembled by client-side JavaScript renders as its empty shell. Sites that server-render their content, or that offer a JSON API alongside their UI, are unaffected.

• A harmless dependency-internal deprecation notice. The installed JSON::Fast 0.19 prints a Method perl (from Mu) deprecation warning to standard error at process exit when from-json is handed input that is not valid JSON (the path web_search and web_grep exercise when a provider or a page returns something unparseable). It is harmless, entirely internal to that dependency, and a candidate to report upstream; it does not affect the result either tool returns.

Examples

• examples/web-server.raku — a research server plugging this pack with sensible traversal limits, with the invocation line in the header.

See Also

• MCP::Server — the server framework, the MCP::Server::Toolkit role this pack implements, and the raku-mcp command.

• MCP::Server::Tool::Shell — a command-runner toolkit for the same framework, with its own gate rather than a sandbox.

Author

Matt Doughty

License

Artistic-2.0