Website Content Clustering

Description

Discover and map all pages related to a specific topic on any website. Provide a domain and a topic, and this agent finds relevant pages using semantic search with multilingual query expansion, clusters them into themes, and scores each page for relevance. Get both a structured markdown report and a raw data file.

Core Capabilities

• Discover 50-100+ relevant pages on any public domain
• Semantic query expansion with multilingual synonyms and related concepts
• Intelligent domain redirect detection (handles complex corporate domain structures)
• AI-driven thematic clustering (5-10 clusters per analysis)
• Relevance scoring and classification (HIGH/MEDIUM/LOW) for every page
• Dual output: markdown report + structured JSON data

Good Query Examples

• "Find all pages about autonomous driving on bmw.com"
• "Map sustainability content on patagonia.com"
• "Discover all AI product pages on openai.com"
• "How much content does apple.com have about privacy?"

Bad Query Examples

• "Find all pages on bmw.com" → No topic specified. This agent needs both a domain and a topic. Fix: "Find all pages about electric vehicles on bmw.com"
• "Find the sustainability page on patagonia.com" → This agent discovers and maps dozens of pages, not a single known page. If you already know the URL, use Page Copy Assessment or Page Design Analysis to evaluate it directly.
• "Find pages about autonomous driving on bmw.com, mercedes-benz.com, and audi.com" → This agent analyzes one domain at a time. Fix: Run three separate analyses and compare the results.

Use-Case Ideas

• Content audit — map all content on a topic across a domain
• Competitive intelligence — discover a competitor's content strategy on a specific theme
• Preparation for SEO analysis — often a key preliminary step: discover and cluster pages first, then use SEO Researcher for keyword rankings, backlink analysis, and content gap evaluation
• Site migration planning — inventory pages by topic before restructuring
• Due diligence — understand a company's web presence on a subject

Limitations

• Only indexes publicly accessible pages (no login-protected or dynamic content)
• Maximum 100 pages per analysis (default 50)
• Results depend on search index coverage of the target domain
• Large domains may have partial coverage
• Some domains may block automated access

Input/Output Spec

Input:

• domain (required): Target website to search (e.g., "bmw.com")
• topic (required): Subject to find pages about (e.g., "autonomous driving")
• page_limit (optional): Number of pages to discover (default: 50, max: 100)
• clustering_preference (optional): Custom categorization approach

Output:

• Markdown report with:
• Executive summary
• Query expansion table (original terms, synonyms, related concepts)
• Top 10 pages ranked by relevance
• Thematic clusters with descriptions and key pages
• Relevance distribution (HIGH/MEDIUM/LOW breakdown)
• Full URL appendix
• Raw data file with complete structured page data

Disambiguation Signals

This agent discovers and maps pages on a domain by topic. It is a recommended prerequisite for both downstream SEO agents — the light and full versions — that benefit from its semantic content discovery:

• Website Content Clustering → Page Ranking Insights (light SEO tool): Page Ranking Insights analyzes up to 20 URLs with a narrative markdown report. When the user wants a quick focused analysis but doesn't have URLs, run Website Content Clustering first to discover them, then pass the top URLs (up to 20) to Page Ranking Insights.
• Website Content Clustering → SEO & GEO Researcher (full SEO tool): SEO & GEO Researcher handles large-scale analysis with Excel output, keyword clustering, and GEO visibility. It supports two input modes, and Website Content Clustering feeds both:
• Chain A (URL list mode, recommended for large topics): Website Content Clustering discovers pages → pass all URLs directly to SEO & GEO Researcher's URL list mode. Example: WCC finds 150 stress pages on tk.de → SEO & GEO Researcher fetches every keyword each URL ranks for. No substring guessing, complete coverage.
• Chain B (domain + patterns mode): Website Content Clustering discovers thematic clusters → extract cluster themes as topic patterns for SEO & GEO Researcher's domain + patterns mode. Broader coverage (catches domain-wide rankings) but still subject to substring matching.
• When to chain: Run Website Content Clustering first when the topic is broad or nuanced (e.g., "health content", "sustainability"), the domain is unfamiliar, or the user doesn't provide specific URLs or keyword patterns. Skip it when the user already provides exact URLs (→ Page Ranking Insights for ≤20, or SEO & GEO Researcher URL list mode for >20) or precise keyword patterns (→ SEO & GEO Researcher domain + patterns mode directly).
• Route to Page Copy Assessment when: the user wants to evaluate the writing quality of a specific page, not discover pages
• Route to Page Design Analysis when: the user wants to evaluate the visual design of a specific page
• Route to News Research Agent when: the user wants news articles from multiple sources, not pages within a single domain
• Route to Page Ranking Insights when: the user already has a small set of specific URLs (≤20) and wants a quick narrative report on what they rank for in Google SERP
• Route to SEO & GEO Researcher when: the user already has many URLs (>20, from a CSV or CMS export) or a domain + specific keyword patterns and wants large-scale analysis with Excel output, keyword clustering, or GEO visibility