MCP::Server::Tool::Web

A web tools (search, fetch, crawl, grep) toolkit for MCP::Server.

Four tools — web_search, web_fetch, web_crawl and web_grep — under one rule that runs beneath all of them: a URL a model supplied is fetched only after the address of the socket that was actually opened has been checked, so a hostname that resolves onto the host machine's own network is refused whatever it is spelled like. See Security.

Synopsis

Plug it into a server you are already building:

use MCP::Server;
use MCP::Server::Tool::Web;

my $server = MCP::Server.new(:name<research>, :version<1.0>);

$server.plug: MCP::Server::Tool::Web.new;

$server.run;   # registers web_search, web_fetch, web_crawl, web_grep

Or assemble a server out of toolkit packs in one shot, with no glue code — the shape a host's JSON config takes:

MCP::Server.new(
    :name<research>,
    :tools[
        'Web' => {
            provider        => 'brave',
            'api-key-env'   => 'BRAVE_API_KEY',
            'max-bytes'     => 4 * 1024 * 1024,
            'total-timeout' => 90,
            'crawl-delay'   => 0.5,
            'allow-hosts'   => <wiki.corp docs.corp>,   # exact names only
        },
    ],
).run;

Or run it straight off the raku-mcp command line that ships with MCP::Server:

raku-mcp --tool=Web='{"crawl-delay":0.5,"max-bytes":4194304}'

raku-mcp --config=mcp.json

{
  "name": "research",
  "instructions": "Search and read the web. Prefer web_grep over web_fetch when you know what you are looking for.",
  "tools": {
    "Web": { "crawl-delay": 0.5, "allow-hosts": ["wiki.corp"] }
  }
}

Wiring that into Claude Code:

{
  "mcpServers": {
    "research": {
      "command": "raku-mcp",
      "args": ["--config=/home/me/mcp.json"],
      "env": { "BRAVE_API_KEY": "..." }
    }
  }
}

Description

web_search answers JSON so the model can pick a URL out of it; web_fetch, web_crawl and web_grep answer plain text, because a page — or a report about pages — is text. Nothing here needs a browser: pages are fetched over plain HTTP(S) and their markup, scripts and navigation are stripped by a hand-rolled extractor (MCP::Server::Tool::Web::Extract), not rendered.

web_search needs a search provider's API key to answer anything; web_fetch, web_crawl and web_grep need nothing beyond network access, and construct successfully even with no key configured anywhere — see [Search provider](#Search provider).

Tools

Parameter names are a contract, not a matter of taste: a policy layer in front of the server (such as MCP::Client::Policy) matches calls by them to decide what a call would reach, so url stays url and multi-word parameters stay kebab-case.

web_search

The search engine, behind a seam. Answers JSON so the model can pick a URL out of it. Zero results is an empty list and a notice, not an error.

# web_search — JSON, so the model can pick a URL out of it
{"count":10,"provider":"brave","query":"raku grammars",
 "results":[{"snippet":"…","title":"Grammars","url":"https://docs.raku.org/…"}]}

# a query that matches nothing:
{"count":10,"notice":"No results for 'asdkfjhasdkfjh39847'. The engine matched nothing: try fewer or more common terms, or drop any quoting.",
 "provider":"brave","query":"asdkfjhasdkfjh39847","results":[]}

web_fetch

One page, as text: a # title line, the final URL when a redirect moved it, then the page with its markup, scripts and navigation removed. Exactly one parameter, deliberately: a windowing tool would invite the model to page through a document it should have grepped instead. Only http and https URLs can be fetched, redirects are followed (every hop re-checked — see Security), and a page bigger than max-bytes comes back cut short and says so on its last line.

# web_fetch — plain text, because a page is text
# Grammars
Retrieved from https://docs.raku.org/language/grammars

Grammars are a powerful tool used to destructure text …

# a page cut short by the byte cap:
[truncated: kept the first 2097152 characters because the server's byte cap of 2097152 bytes was reached; the rest of the page was not fetched]

web_crawl

Follows the links from a page and returns an index of what is there — never the pages themselves. Only links on the same origin as the starting page (same scheme, host and effective port) are followed; depth counts hops from the starting page; robots.txt is respected by default (see Security). See MCP::Server::Tool::Web::Crawl for the walk itself.

# web_crawl — an index, not the pages
Crawled 3 pages from https://docs.example.com/ (depth 1, same origin only).

https://docs.example.com/  200  4213 bytes  Documentation
https://docs.example.com/install  200  2288 bytes  Installing

[stopped after 20 pages; raise max-pages to go further]

Why an index rather than the pages: concatenating twenty pages into one tool result is a way to spend a context window without answering anything — what the model would actually receive is page one, half of page two, and a truncation marker. So a crawl reports what is there, one line per page, and the corpus stays reachable through web_grep (with a depth) and web_fetch.

web_grep

fs_grep for the web: search one page, or a whole site at a depth, and return only the matching lines as url:line:text. The pattern is literal text unless regex is set, in which case it is a Raku regex, not a PCRE one. This is far cheaper than web_fetch or web_crawl when you know what you are looking for.

# web_grep — only the lines that matched, exactly as fs_grep reports them
https://docs.example.com/install:42:  zef install Foo::Bar

[stopped at 50 matching lines; raise max-results to see more]

With the default depth of 0 only the URL given is searched; with a depth above 0 the same-origin links are followed as web_crawl does, and no more pages are fetched once max-results lines have matched — which is the whole reason web_grep is cheaper than crawling and then fetching.

Configuration

Every attribute below is a JSON config key — from-config validates against exactly this set, so a typo names the valid keys instead of doing nothing quietly:

MCP::Server::Tool::Web.from-config({ providr => 'brave' });
# Unknown config key(s) for MCP::Server::Tool::Web: providr.
# Valid keys: allow-cidrs, allow-hosts, allow-loopback, allow-private,
# api-key, api-key-env, connect-timeout, crawl-delay, headers-timeout,
# max-bytes, proxy, provider, respect-robots, safesearch, search-country,
# search-lang, timeout, total-timeout, user-agent

• max-bytes has a floor of 4096: below that a page cannot hold a document, and every fetch would report more truncation notice than page.

• timeout, total-timeout, connect-timeout and headers-timeout must all be greater than zero.

• crawl-delay cannot be negative.

• user-agent must not be empty — robots.txt groups are matched against its product token, and an origin cannot report who visited it if it is blank.

• allow-hosts entries are exact host names, not URLs and not patterns: a * or ? in one is a configuration error, refused at construction, because a pattern here would match nothing and look like it matched everything — see Security.

Two more seams exist and are deliberately not config keys, because a Callable or a live object cannot come out of JSON — they are is built private attributes, so .new can pass them and from-config cannot:

• search-provider — anything doing MCP::Server::Tool::Web::SearchProvider, used instead of building the named provider. This is how a test dispatches web_search without a network, and how a host wires up an engine this pack has never heard of. See [Search provider](#Search provider).

• fetcher — a prepared MCP::Server::Tool::Web::Fetcher. Given one, the pack uses it for single-page work and clones it (with the robots policy swapped in) for traversal, so the transport — and its cache of vetted connections — is shared.

Search provider

web_search does not talk to any particular engine directly: it asks whatever object composes MCP::Server::Tool::Web::SearchProvider, a role of exactly two methods —

method name(--> Str:D) { ... }                              # 'brave', 'my-provider', ...
method search(Str:D $query, Int:D $count --> List:D) { ... } # [ {title,url,snippet,age?}, ... ]

— which MCP::Server::Tool::Web composes a concrete implementation of via its provider config key. This distribution ships one: MCP::Server::Tool::Web::Provider::Brave, talking to the Brave Search API's web/search endpoint. A provider that throws is the whole story the model gets: its first line is written to teach, and MCP::Server::Tool::Web's web_search tool surfaces it verbatim as the tool's error text.

The API key is resolved at call time, inside .search, never at construction — so a pack with no BRAVE_API_KEY in its environment still constructs and serves web_fetch, web_crawl and web_grep perfectly well, and only web_search fails, the moment it is actually called, naming the environment variable and the two config keys that would fix it:

web_search needs a Brave Search API key. Set the environment variable
BRAVE_API_KEY, or point "api-key-env" at the variable that holds it, or put
an "api-key" in the web config.

Bring your own engine by passing an object doing MCP::Server::Tool::Web::SearchProvider as search-provider — the constructor accepts it in place of provider, since a live object cannot come out of JSON config.

The search provider is not guard-subject

The search provider deliberately does not go through the SSRF guard described in Security. The endpoint web_search calls is the operator's, named in configuration — not the model's. A SearXNG on the operator's own LAN, or a self-hosted Brave-compatible proxy, is a legitimate thing to point this at, so the search provider talks to its API directly. The guard exists for URLs an attacker — or a confused model — chose via web_fetch, web_crawl or web_grep; a config key set once, by whoever deploys this pack, is not that threat model. If a provider's endpoint is ever made settable from an untrusted source, guarding it becomes that call site's responsibility — the shipped provider intentionally does not.

Security

Every URL a model supplies — to web_fetch, web_crawl or web_grep — passes through the same floor before a byte is sent anywhere.

Connect, then verify the peer

A URL is checked for shape first (scheme, credentials, host spelling, port range — MCP::Server::Tool::Web::Url), then a TCP connection is opened, and only then is the address of the socket that was actually opened checked against policy (MCP::Server::Tool::Web::Guard, enforced by MCP::Server::Tool::Web::Transport). Nothing is written to that socket until the check passes — no request line, no Host header, not even a TLS ClientHello — so a refused connection is a handshake and an immediate reset, and the peer learns nothing about what was going to be asked for.

This kills DNS rebinding rather than narrowing it: the address that is checked is the address of the established connection, so there is no window between "resolve, check" and "connect" for an attacker with a short-TTL DNS zone to win. The usual "resolve, check, then connect" shape has exactly that window; this pack does not use that shape.

What is refused

Anything that is not classified public is refused unless a configuration key says otherwise — loopback, RFC 1918 private ranges, CGNAT, link-local addresses, cloud metadata endpoints, and the handful of other reserved and special-purpose ranges IANA has carved out (documentation ranges, benchmark ranges, multicast, and so on — see MCP::Server::Tool::Web::Addr for the full table and the RFC behind each one). An address that does not parse is refused too: the guard never treats "I could not tell" as "it is fine". The file:, javascript: and similar schemes are refused on shape, before any of this runs, and no allow-list entry can rescue a URL refused on shape — allow-lists widen which machines may be reached, and file:///etc/passwd is not a machine.

The escape hatches

• allow-private — allow every non-public class at once. The blunt instrument: one switch for "this machine may reach its own network".

• allow-loopback — allow loopback only. The narrow key for test rigs and local development servers, so enabling it does not open the whole LAN.

• allow-hosts — exact, lowercase host names, matched exactly. corp permits the host corp and never wiki.corp, and emphatically never evil.com.corp: suffix matching is where allow-list CVEs come from, so there is no wildcard syntax, and a pattern in the list is refused at construction rather than silently doing nothing.

• allow-cidrs — CIDR blocks, IPv4 and IPv6, matched against the peer's actual address (a v4 address written in its v6-mapped form still matches a v4 block).

Every refusal names the rule, the address class, the RFC behind it, the host, the address it resolved to, and the configuration key that would permit it — the audience is a model deciding what to try next, and "denied" with nothing else just makes it try the same thing spelled differently.

Redirects are re-checked, every hop

Redirects are followed by MCP::Server::Tool::Web::Fetcher itself, one hop at a time, rather than by the underlying HTTP client — so a Location: http://10.0.0.1/ or a Location: file:///etc/passwd is refused on the hop that produced it, with the same rule and the same message a first request would have gotten. A redirect that drops from https to http is refused outright, downgrade or not: the bytes that would come back are readable, and rewritable, by anything on the path.

Connection reuse is a security property here, not a risk

The HTTP client is persistent, and that is deliberate: a cached pipeline is an already vetted, still established socket. Rebinding a name cannot retarget an open connection, so reusing one costs nothing in safety and saves a re-check on every subsequent request. The consequence to respect the other way: one client belongs to one guard configuration, and a Transport is never shared between packs configured with different allow-lists.

Fetch truncation is head-only

When a page runs into max-bytes or the deadline, only the head is kept — never a head-and-tail splice. The tail of a truncated HTML document is closing markup: the extractor needs a parseable prefix to produce anything at all, and web_grep's line numbers have to mean something stable. (Compare MCP::Server::Tool::Shell, whose command output keeps head and tail, because the interesting part of a build log is usually at the end — a web page is a different shape of document, and gets a different rule.)

robots.txt

Per the pack's ruling: web_crawl, and web_grep with a depth above zero, respect robots.txt by default (respect-robots, default True); web_fetch and depth-zero web_grep never consult it, regardless of that setting. A page a person asked for by name is not crawling, and no robots.txt has ever meant "this URL may not be read once, deliberately, on a human's behalf" — traversal is a different act, and that one is asked for. Setting respect-robots to False turns it off for the traversing tools too. See MCP::Server::Tool::Web::Robots for the RFC 9309 details (group matching, longest-pattern-wins, Crawl-delay, and why a robots.txt that cannot be fetched at all means "allowed", not "deny everything").

Known limitations

• Static fetch only. These tools issue a plain HTTP(S) GET and parse the bytes that come back; there is no browser anywhere in the code path, so a single-page application whose content is assembled by client-side JavaScript renders as its empty shell. Sites that server-render their content, or that offer a JSON API alongside their UI, are unaffected.

• A harmless dependency-internal deprecation notice. The installed JSON::Fast 0.19 prints a Method perl (from Mu) deprecation warning to standard error at process exit when from-json is handed input that is not valid JSON (the path web_search and web_grep exercise when a provider or a page returns something unparseable). It is harmless, entirely internal to that dependency, and a candidate to report upstream; it does not affect the result either tool returns.

Examples

• examples/web-server.raku — a research server plugging this pack with sensible traversal limits, with the invocation line in the header.

See Also

• MCP::Server — the server framework, the MCP::Server::Toolkit role this pack implements, and the raku-mcp command.

• MCP::Server::Tool::Shell — a command-runner toolkit for the same framework, with its own gate rather than a sandbox.

Author

Matt Doughty

License

Artistic-2.0