User-agent: * Crawl-delay: 3 Disallow: /cgi-bin/ Disallow: /admin/ User-agent: YandexBot Disallow: / User-agent: Exabot Disallow: / User-agent: Riddler Disallow: / User-agent: SentiBot Disallow: / User-agent: SemrushBot Disallow: /