Chad Gingrich

You’ve probably noindex-ed pages you don’t want showing up in search. It’s a solid practice for product variants or paid destination pages, and it works for Google rankings. But AI crawlers don’t care about noindex tags, and as of yet there is no corresponding version of noindex for LLMs. robots.txt is an option, but our own research shows mixed results. So what does that mean for the pages you thought were hidden from AI?

This blog was originally published on October 5, 2022 and was updated on June 9, 2026. Online shoppers are savvier than ever these days. Ecommerce giants have paved the way for customer expectations with lightning-fast site speeds, clean user experiences, and super simple checkout processes.

Finding and checking the sitemap of any website is a handy task for a digital marketer, whether you’re checking your own site, the site of a competitor, or any other domain. Explore the value of finding sitemaps, and more importantly, 8 ways to find them. Want to skip right to the tactics? Jump down to the tips on finding a sitemap. What Is A Sitemap and Why You Should Check if You Have One A sitemap is a list of the indexable pages on a website.