User-agent: * Disallow: /_mm/ Disallow: /_notes/ Disallow: /MMWIP/ # /robots.txt file for gsa-crawler User-agent: gsa-crawler Disallow: /*.csi Allow: /$ (The gsa-crawler is allowed to crawl everything else.)