Echo Lab Tools › Sitemap URLs

Extract every URL a website publishes in its sitemaps, with lastmod and hreflang

The sitemap URL extractor is an Apify Actor that turns websites, domains or direct sitemap links into a deduplicated list of every URL the site publishes in its sitemaps, as JSON or CSV, with lastmod, changefreq, priority, hreflang alternates, image/video counts and news metadata.

By Echo Lab · Last updated 2026-11-06 · Apify Actor echolab/sitemap-url-extractor

How it works

For each start point (a domain, a homepage URL or a direct .xml, .xml.gz or .txt sitemap link) the Actor:

1. Reads robots.txt and collects its Sitemap: lines. If there are none, it tries common paths such as /sitemap.xml and /sitemap_index.xml until one works.
2. Downloads each sitemap over plain HTTP. Gzip is detected from the file's magic bytes, not its name. XML is parsed iteratively and DTD/entity declarations are rejected, so a hostile file cannot blow up memory.
3. Follows sitemap indexes up to maxDepth (default 5) with loop protection, and stops at maxSitemapsPerSite files (default 500).
4. Normalizes and filters every <loc>, then pushes one row per URL that is unique across the whole run, so overlapping start points are not billed twice.

A SUMMARY record lists, per start point, which sitemap files were found, which failed and why, and how many URLs were saved. It reads only what the site publishes for machines; it does not crawl HTML pages.

Sample output

A row from a real run on the Apify platform (run ilYbFWXlsMAqkqnTw, build 0.1.4, 2026-11-06) with two start points, https://www.sitemaps.org and https://docs.apify.com:

{
  "url": "https://www.sitemaps.org/protocol.html",
  "lastmod": "2022-12-15",
  "changefreq": null,
  "priority": null,
  "imageCount": 0,
  "videoCount": 0,
  "newsTitle": null,
  "newsPublicationDate": null,
  "alternates": [],
  "site": "https://www.sitemaps.org",
  "sitemapUrl": "https://www.sitemaps.org/sitemap.xml",
  "extractedAt": "2026-11-06T04:16:34.585233+00:00"
}

And a row from the second site, where priority and changefreq are set but lastmod is not:

{
  "url": "https://docs.apify.com/api",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "imageCount": 0,
  "videoCount": 0,
  "newsTitle": null,
  "newsPublicationDate": null,
  "alternates": [],
  "site": "https://docs.apify.com",
  "sitemapUrl": "https://docs.apify.com/sitemap_base.xml",
  "extractedAt": "2026-11-06T04:16:34.215302+00:00"
}

The run's SUMMARY record, verbatim:

[
  {
    "site": "https://www.sitemaps.org",
    "sitemapsFound": [
      "https://www.sitemaps.org/sitemap.xml"
    ],
    "sitemapsFailed": [],
    "urls": 84,
    "status": "done"
  },
  {
    "site": "https://docs.apify.com",
    "sitemapsFound": [
      "https://docs.apify.com/sitemap.xml",
      "https://docs.apify.com/sitemap_base.xml"
    ],
    "sitemapsFailed": [],
    "urls": 100,
    "status": "done"
  }
]

So the tool found the sitemap index at docs.apify.com through robots.txt and followed it into sitemap_base.xml, stopping at the 100-URL cap set for this test. sitemaps.org gave 84 URLs, all with lastmod; docs.apify.com gave 100, 0 with lastmod. 184 URLs in 9 seconds at 256 MB.

Output fields

How much does it cost to extract sitemap URLs?

Pricing is per result: $0.50 per 1,000 URLs ($0.0005 per URL) plus $0.00005 per run, platform usage included. Error logs and the SUMMARY record are not billed.

Worked examples: the sample run saved 184 URLs, about $0.092. A 10,000-URL site is $5.00. Watching a news site for new pages with lastmodAfter set to yesterday and 300 new URLs a day costs about $0.15 a day. maxUrls and maxUrlsPerSite cap any run. Apify's free plan includes $5 a month for Store tools (apify.com/pricing, checked 2026-11-06), which covers 10,000 URLs.

Limits you should know about

1. Only what is in the sitemaps. Pages the site does not list are invisible, and a JavaScript-only site with no sitemap returns nothing. This is an inventory of what the publisher declares, not a crawl.
2. lastmod is the publisher's claim. Many CMSs set it to the build time for every URL, so treat it as a hint, not proof of freshness.
3. No memory between runs. There is no stored diff; lastmodAfter filters by published date, which only works if the site maintains lastmod honestly.
4. Size ceilings. Files are capped at 50 MB uncompressed (with gzip-bomb protection). Around one million unique URLs per run is a planning ceiling at 256 MB, not a guarantee; split inputs or raise memory for bigger sites.
5. Filters are Python regular expressions, not globs, and sameDomainOnly (on by default) drops URLs on other registrable domains, for example a sitemap hosted on a CDN that lists another domain's pages.

API: call it from code

curl -X POST "https://api.apify.com/v2/acts/echolab~sitemap-url-extractor/run-sync-get-dataset-items?format=csv" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://www.sitemaps.org"], "maxUrlsPerSite": 0}'

Python, new blog posts since a date:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("echolab/sitemap-url-extractor").call(run_input={
    "startUrls": ["https://example.com"],
    "includePatterns": ["/blog/"],
    "excludePatterns": ["/tag/", "\\?"],
    "lastmodAfter": "2026-11-01",
    "maxUrlsPerSite": 0,         # 0 = unlimited
})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row["lastmod"], row["url"])

Sitemap URL extractor vs. alternatives

When not to use it: if the site has no sitemap or you need pages it does not list, use a crawler. If you need million-URL inventories every week, a self-hosted script is cheaper than paying per URL. This tool fits repeatable inventories of many sites, or new-page monitoring, that should land in a spreadsheet or pipeline without running anything yourself.

Open the sitemap URL extractor on Apify for the full input schema.

Run the sitemap URL extractor on Apify →