Data Automation & Cleaning Beginner

Generate Google Search Console XML Sitemap & robots.txt for Static Sites in Python

Automatically scan a static HTML directory (`dist/`) in Python and generate a valid `sitemap.xml` and `robots.txt` for instant Google indexing.

#pandas

Problem Statement

When deploying programmatic SEO directories with 50 to 1,000 static pages, manually writing XML sitemaps for Google Search Console is error-prone. This script scans your `dist/` folder and builds `sitemap.xml` automatically.

Formula / Core Logic

Walks all `index.html` files inside `dist/`, maps them to canonical HTTPS URLs, and writes standard sitemaps.org XML protocol.

Installation

pip install pandas

automated-sitemap-xml-generator-programmatic-seo-python.py

from pathlib import Path
from datetime import date

def generate_sitemap(dist_dir='dist', base_url='https://pyquant-library.netlify.app'):
    today = date.today().isoformat()
    urls = []
    for html_file in Path(dist_dir).rglob('index.html'):
        rel = html_file.parent.relative_to(dist_dir).as_posix()
        loc = f'{base_url}/' if rel == '.' else f'{base_url}/{rel}/'
        urls.append(f'  {loc}{today}weekly')
        
    xml = '\n\n' + '\n'.join(urls) + '\n'
    Path(dist_dir, 'sitemap.xml').write_text(xml, encoding='utf-8')
    Path(dist_dir, 'robots.txt').write_text(f'User-agent: *\nAllow: /\nSitemap: {base_url}/sitemap.xml\n', encoding='utf-8')
    print(f'Generated sitemap.xml with {len(urls)} URLs!')

Get the Complete 100+ Python Quant & Automation Bundle

Instant lifetime access to all production-ready scripts, valuation models, and trading bots ($19).

Buy Full Bundle on Gumroad ($19) →

Frequently Asked Questions

Q: Where do I submit `sitemap.xml` after deploying to Netlify?

Open Google Search Console, verify your domain or URL prefix, click 'Sitemaps' in the left sidebar, and enter `sitemap.xml`.