Rand Stats

Sitemap

zef:sasha

NAME

Sitemap - Sitemap generator for Raku

DESCRIPTION

Sitemap is a Raku module for generating XML sitemaps, crawling websites, and fetching sitemaps from servers.

Features

Note: Image, hreflang, lastmod, and priority data only appears in XML format output.

Dependencies

Installation

zef install Sitemap

Installing from source

cd /path/to/Sitemap
zef install .

Requires Raku 6.d or later.

SYNOPSIS

Command Line

# Crawl a website and generate XML sitemap
sitemap https://example.com

# Crawl with options
sitemap https://example.com -o sitemap.xml -f xml -v

# Generate HTML sitemap
sitemap https://example.com -f html -o sitemap.html

# Generate RSS feed
sitemap https://example.com -f rss -o feed.rss

# Build from URL list file
sitemap build urls.txt -o sitemap.xml

# Parse existing sitemap
sitemap parse existing-sitemap.xml

# Fetch sitemap from server (auto-discover from robots.txt)
sitemap fetch example.com

# Fetch with recursive support (downloads all child sitemaps in index)
sitemap fetch example.com --recursive

# Recursive fetch with descriptive directory naming
# Creates example-com-sitemap/ directory with context-aware filenames
sitemap fetch example.com --recursive -v

# Crawl local directory and generate sitemap
sitemap dir ./my-site

# Crawl directory with custom base URL
sitemap dir ./my-site --base-url https://example.com -v

# Build from JSON or YAML site tree definition (priorities inferred from depth)
sitemap tree site.json -o sitemap.xml
sitemap tree site.yaml -o sitemap.xml

# View the tree structure
sitemap tree site.json --verbose

Library Usage

Sitemap::Builder (Generate Sitemaps)

use Sitemap::Builder;

# Create sitemap from URLs
my $builder = Sitemap::Builder.new;
$builder.add-item('https://example.com/page1', :priority(0.8));
$builder.add-item('https://example.com/page2', :lastmod(DateTime.now));

# Write to file (auto-splits at 50k URLs / when add-sitemap was used)
$builder.write: 'sitemap.xml';
# When items exceed $builder.max-entries (default 50000) this auto-splits
# into sitemap-1.xml, sitemap-2.xml, ... plus a sitemap-index.xml pointing
# at the chunks.

# With compression and XSL
my $builder2 = Sitemap::Builder.new(:compress, :xsl-url('/sitemap.xsl'));
$builder2.add-item('https://example.com/page');
$builder2.write: 'sitemap.xml';

# Manually assemble a sitemap index from existing sitemaps
my $index = Sitemap::Builder.new;
$index.add-sitemap('https://example.com/posts.xml', :lastmod(DateTime.now));
$index.add-sitemap('https://example.com/news.xml',  :lastmod('2024-01-01'));
$index.add-sitemap('https://example.com/video.xml');
$index.write: 'sitemap-index.xml';        # <sitemapindex> wrapper file
my $xml = $index.render-index;            # or capture the index XML directly

# Discover sitemap from robots.txt (with /sitemap.xml fallback)
my $discovered = $builder.discover-sitemap('https://example.com');

Sitemap::Crawler (Crawl Websites)

use Sitemap::Crawler;

# Crawl a website
my $crawler = Sitemap::Crawler.new('https://example.com');

$crawler.on-add: -> $url {
    say "Found: $url";
};

my $builder = $crawler.crawl;
$builder.write: 'sitemap.xml';

# Crawl with automatic video and news extraction
my $crawler2 = Sitemap::Crawler.new(
    'https://example.com',
    :extract-videos,
    :extract-news,
);
my $builder2 = $crawler2.crawl;
$builder2.write: 'sitemap.xml';

# News builder contains extracted NewsArticle objects
# (only populated when extract-news is True)
if $crawler2.news-builder -> $nb {
    $nb.write: 'sitemap-news.xml';
}

Sitemap::DirScanner (Crawl Local Directories)

use Sitemap::DirScanner;

# scan-dir returns a Hash: builder, news-builder, stale-news-count
my %result = scan-dir('./my-site',
    :base-url<https://example.com>,
    :extract-videos,
    :extract-news,
);
my $builder = %result<builder>;

$builder.write: 'sitemap.xml';

if %result<news-builder> -> $nb {
    $nb.write: 'sitemap-news.xml';
}

Sitemap::SiteTree (Build from Site Hierarchy)

use Sitemap::SiteTree;

# Build a tree - priorities auto-inferred from depth
my $root = Sitemap::SiteTree.new;
my $blog = $root.add-child('blog', :priority(0.8));
$blog.add-child('first-post');
$blog.add-child('second-post');

my $builder = $root.to-builder('https://example.com');
$builder.write: 'sitemap.xml';
# Priorities: root=1.0, blog=0.8, first-post=0.6, second-post=0.6

# Wire parents from a flat list
my $home  = Sitemap::SiteTree.new(:stub<home>);
my $about = Sitemap::SiteTree.new(:stub<about>, :parent-stub<home>);
Sitemap::SiteTree.wire-parents([$home, $about]);

# Debug tree visualization
say $root.tree;

# Serialize to/from YAML
my $yaml = $root.to-yaml;
"site-tree.yml".IO.spurt: $yaml;

my $reconstructed = Sitemap::SiteTree.from-yaml($yaml);

YAML Serialization

Sitemap::SiteTree and Sitemap::Item support serialization to/from YAML via to-yaml/from-yaml (on SiteTree) and to-hash/from-hash (on both).

use Sitemap::SiteTree;

# Build a tree
my $root = Sitemap::SiteTree.new;
my $blog = $root.add-child('blog', :priority(0.8));
$blog.add-child('first-post');

# Roundtrip through YAML
my $yaml = $root.to-yaml;
my $tree = Sitemap::SiteTree.from-yaml($yaml);

# Rebuild the sitemap
my $builder = $tree.to-builder('https://example.com');
$builder.write: 'sitemap.xml';

Items and sub-resources (images, videos, links, news) can be serialized independently:

use Sitemap::Item;
use YAMLish;

my $item = Sitemap::Item.new(
    url => 'https://example.com/page',
    priority => 0.8,
);
$item.add-image('https://example.com/img.jpg');

my $yaml = save-yaml($item.to-hash);
my $copy = Sitemap::Item.from-hash(load-yaml($yaml));

The YAML format maps directly to the object structure - parent-child relationships are preserved via nested children arrays.

Sitemap::Fetcher (Fetch Sitemaps from Servers)

use Sitemap::Fetcher;

# Fetch a single sitemap
my $xml = fetch('https://example.com/sitemap.xml', :verbose);

# Fetch sitemap with auto-discovery (robots.txt + /sitemap.xml fallback)
my $xml = fetch('https://example.com', :ssl-verify(False));

# Recursive fetch (downloads all child sitemaps in a sitemap index)
my %result = fetch-recursive(
    'https://example.com/sitemap.xml',
    :format<xml>,
    :verbose
);
# %result<index-file> = saved index file
# %result<dir> = directory with child sitemaps
# %result<children> = list of child sitemap files

# Generate output filename with path context
my $filename = output-filename(
    'https://example.com/blog/sitemap.xml',
    :format<xml>
);
# Returns: sitemap-blog.xml

For non-crawl workflows, Video and News data can be added manually via the builder API:

use Sitemap::Builder;
my $builder = Sitemap::Builder.new;

# Add video sitemap data
$builder.add-item('https://example.com/video-page',
    videos => [
        Sitemap::Item::Video.new(
            thumbnail-loc => 'https://example.com/thumb.jpg',
            title         => 'My Video Title',
            description   => 'A description of the video',
        )
    ]
);

# Add news sitemap data
$builder.add-item('https://example.com/news-article',
    news => [
        Sitemap::Item::News.new(
            publication          => 'Example News',
            publication-language => 'en',
            title                => 'Article Headline',
            publication-date     => DateTime.new('2024-01-01'),
        )
    ]
);

Sitemap::JsonLd (Extract JSON-LD Structured Data)

use Sitemap::JsonLd;

# Extract all JSON-LD objects from HTML
my @objects = extract-jsonld($html);

# Filter by type (supports schema.org URL prefixes and @graph)
my @videos = find-by-type(@objects.List, 'VideoObject');
my @news = find-by-type(@objects.List, 'NewsArticle');

# Convenience: extract video/news objects with full field mapping
my @video-objects = extract-video-objects($html);
# Returns: url, thumbnail, title, description, duration (seconds), pub-date

my @news-objects = extract-news-objects($html);
# Returns: publication, language, title, publication-date, stale (Bool)
# Matches: NewsArticle, ReportageNewsArticle, OpinionNewsArticle,
#          ReviewNewsArticle, AnalysisNewsArticle, BackgroundNewsArticle
# Does NOT match: Article, BlogPosting, or other non-news types
# Articles older than 48h are marked stale => True

# Parse ISO 8601 durations (e.g., from VideoObject)
my $seconds = parse-iso8601-duration('PT1H2M3S');  # 3723

CLI Options

Crawl Command (sitemap <url>)

Fetch Command (sitemap fetch <url>)

Note: Recursive fetch creates a descriptive directory (e.g., example-com-sitemap/) with path-aware filenames (e.g., sitemap-blog.xml, sitemap-news-2024.xml).

Build Command (sitemap build <file>)

Note: indexes are produced automatically when you lower -m below the item count (or exceed 50k): the command writes chunk files (e.g. sitemap-1.xml, sitemap-2.xml) plus a sitemap-index.xml pointing at them. Assembling an index from already-existing sitemap files is a library operation (Sitemap::Builder.add-sitemap), not a CLI subcommand.

Parse Command (sitemap parse <file|url>)

Convert Command (sitemap convert <file>)

Tree Command (sitemap tree <file>)

Build a sitemap from a JSON (.json) or YAML (.yml/.yaml) site tree definition. Priorities are auto-inferred from tree depth.

JSON format:

{
  "base_url": "https://example.com",
  "pages": [
    {"stub": "home", "priority": 1.0},
    {"stub": "about",
     "children": [
       {"stub": "team"}
     ]},
    {"stub": "blog",
     "children": [
       {"stub": "first-post"},
       {"stub": "second-post"}
     ]}
  ]
}

YAML format:

base_url: "https://example.com"
pages:
  - stub: home
    priority: 1.0
  - stub: about
    children:
      - stub: team
  - stub: blog
    children:
      - stub: first-post
      - stub: second-post

Dir Command (sitemap dir <directory>)

DEVELOPMENT

Running tests

The suite lives in t/ as one .rakutest file per area:

raku -I lib t/04-grammars.rakutest     # a single file
raku -I lib t/*.rakutest               # the whole suite (fast, CLI skipped)

t/14-cli.rakutest spawns a fresh Raku VM for every CLI invocation (41 subprocess starts) and runs local Cro HTTP servers, so it takes ~40s on its own. For that reason it is skipped by default. Run the full suite with the CLI coverage enabled via the SITEMAP_SLOW_TESTS environment variable:

SITEMAP_SLOW_TESTS=1 raku -I lib t/*.rakutest
SITEMAP_SLOW_TESTS=1 prove6 -e 'raku -Ilib' t/*.rakutest   # CI-style
SITEMAP_SLOW_TESTS=1 zef test .   # as zef/mi6 would run them

Running the full suite can take a few minutes — give it a generous timeout instead of assuming it has hung.

AUTHOR

Sasha Abbott sashaa@disroot.org

LICENSE

This software is dedicated to the public domain under the CC0 1.0 Universal (CC0 1.0) Public Domain Dedication.

To the extent possible under law, the author has waived all copyright and related or neighboring rights to this work.