Bing Indexing Architecture Rebuild: How Compare2Best Fixed a 92% SEO Coverage Gap

1. The Symptom: Bing Splits One Site Into Four Directories

The investigation began when Bing Webmaster Tools presented Compare2Best as four separate "directories" — one per subdomain — each with near-zero indexed URLs:

Total: 952 indexed pages. The platform had 12,000+ content pages across product details, comparison hubs, buying guides, specification references, glossary entries, brand profiles, and market insights. The 1,800 excluded URLs on the lighting subdomain were the first red flag — a subdomain shouldn't account for 89% of all excluded pages unless something was structurally wrong.

Exporting URL lists from each directory confirmed the pattern: Bing's CSV exports showed only 1 URL per directory, not because the directories were empty, but because Bing's internal URL grouping was treating redirect chains as separate sites.

2. Diagnosis: The Sitemap Poison Pill

2.1 robots.txt — Four Sitemaps, Three Broken

The first diagnostic check targeted robots.txt, which listed four sitemaps:

The sitemap index was correctly formed, but the three additional sitemaps — core, content, and guides — had a fatal flaw. Every URL in them used the lighting.compare2best.com subdomain.

2.2 Domain Distribution Analysis

A quick domain-count on the sitemaps revealed the structural problem:

Meanwhile, the sitemap-main.xml (302 URLs) and sitemap-insights.xml (97 URLs) correctly used www.compare2best.com — but they were buried in a sitemap index that Bing was ignoring because the three broken sitemaps were consuming its attention.

2.3 The Redirect Chain

The next step was tracing what happens when Bing crawls one of these 658 URLs:

A double 301 redirect chain for every single sitemap URL. Bing processes this as:

1. Crawl lighting.compare2best.com URL → receives 301
2. Follow redirect → receives another 301
3. Finally reaches canonical URL → but the sitemap "belongs" to the wrong subdomain directory
4. Conclusion: lighting.compare2best.com is a site of redirects, not content → exclude 1,800 URLs

2.4 Generator Root Cause

The sitemap generator script (scripts/generate-sitemap.mjs) had hardcoded the lighting subdomain URL across 8 separate locations:

This was a post-migration artifact. When the lighting content was migrated from its standalone subdomain to the main www domain (via 301 redirect), the sitemap generator was never updated to reflect the new canonical URL structure. The content was accessible — but the sitemap was pointing search engines to the wrong address.

3. The Fix: Three-Stage Deployment

Stage 1: robots.txt Cleanup (Immediate)

The first fix was surgical and immediate — remove the three broken sitemaps from robots.txt, leaving only the correctly-formed index:

Result: Bing immediately stopped wasting crawl budget on the 658 redirect-chain URLs. This single change prevented further exclusion accumulation while the deeper fix was deployed.

Stage 2: Sitemap Generator Rewrite

All 8 hardcoded lighting.compare2best.com references were replaced with www.compare2best.com/lighting — the canonical URL path under the main domain:

Stage 3: Sitemap Index Restoration

The sitemap index (sitemap.xml) was expanded from 4 child sitemaps to 6, adding the now-correctly-formatted lighting and product sitemaps that had been excluded from the index under the old architecture:

4. Submission & Verification

4.1 Immediate Actions

4.2 IndexNow Key: 9262f4c0c0e4401d9e247c54b08a04e1

Compare2Best maintains an active IndexNow key registered with Bing and Yandex. The key file is served at /9262f4c0c0e4401d9e247c54b08a04e1.txt and verified weekly. After the sitemap rebuild, 5 sitemap files and 20 priority URLs were submitted via IndexNow to trigger immediate re-crawl.

5. Architectural Lessons

5.1 Subdomain vs. Path: Choose One Canonical

Compare2Best's lighting content currently follows this pattern: subdomain URLs 301 redirect to www URLs. The correct canonical is www.compare2best.com/lighting/..., and all sitemaps now use this format. If in the future the decision is made to serve content directly from the lighting subdomain, the 301 redirect must be removed first.

5.2 Sitemap Index is the Single Source of Truth

The robots.txt should reference exactly one sitemap — the index. All child sitemaps should be listed only in the index. This prevents the situation where robots.txt has 4 sitemap entries, 3 of which are broken, and debugging requires checking multiple sources. A single entry point simplifies both crawler processing and operational troubleshooting.

5.3 Post-Migration Sitemap Audit is Mandatory

Any domain or URL structure migration must include a sitemap audit as a deployment gate. The three checks that would have caught this issue immediately:

1. Domain audit: grep -oh 'https://[^/]*' sitemap-*.xml | sort | uniq -c — count URLs by domain. Any subdomain count > 0 is a red flag.
2. Redirect trace: curl -sI -L {sample_url} | grep '^HTTP' — trace the HTTP response chain. Any URL appearing in a sitemap must return 200 on the first request, not after a redirect chain.
3. Index count baseline: Record Bing and Google indexed URL counts pre-migration. Any post-migration drop > 20% requires immediate investigation.

5.4 Crawl Budget Economics

The 658 broken sitemap URLs didn't just fail to get indexed — they consumed crawl budget that could have been spent on real content. With a double-redirect chain, each URL consumed 3 HTTP requests (initial + two redirects) for zero indexable output. At scale, a 2,000-URL sitemap full of redirects burns approximately 6,000 crawl requests — enough to fully re-crawl a 300-page site 20 times over.

6. Expected Recovery Timeline