# ----------------------------------------------- # ALLOW: Legitimate Search Engine Crawlers # ----------------------------------------------- User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-News Allow: / User-agent: Bingbot Allow: / User-agent: Slurp Allow: / User-agent: DuckDuckBot Allow: / User-agent: Baiduspider Allow: / User-agent: facebot Allow: / User-agent: Twitterbot Allow: / User-agent: WhatsApp Allow: / User-agent: LinkedInBot Allow: / User-agent: Applebot Allow: / User-agent: UptimeRobot Allow: / # ----------------------------------------------- # BLOCK: Known Bad Bots & Scrapers # ----------------------------------------------- User-agent: YandexBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: DotBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: BLEXBot Disallow: / User-agent: MegaIndex Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: PetalBot Disallow: / User-agent: Seekport Disallow: / User-agent: serpstatbot Disallow: / User-agent: SEOkicks Disallow: / User-agent: Exabot Disallow: / User-agent: ia_archiver Disallow: / User-agent: archive.org_bot Disallow: / User-agent: SiteBot Disallow: / User-agent: rogerbot Disallow: / User-agent: NetcraftSurveyAgent Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: SEOdiver Disallow: / User-agent: linkdexbot Disallow: / User-agent: spbot Disallow: / User-agent: Lipperhey Disallow: / User-agent: linkfluence Disallow: / User-agent: proximic Disallow: / User-agent: TurnitinBot Disallow: / User-agent: Panscient Disallow: / User-agent: SurveyBot Disallow: / User-agent: Sogou Disallow: / User-agent: OpenindexSpider Disallow: / User-agent: AddThis Disallow: / User-agent: MojeekBot Disallow: / User-agent: YisouSpider Disallow: / User-agent: Scrapy Disallow: / User-agent: python-requests Disallow: / User-agent: curl Disallow: / User-agent: wget Disallow: / User-agent: Go-http-client Disallow: / User-agent: Java Disallow: / User-agent: libwww-perl Disallow: / User-agent: lwp-trivial Disallow: / User-agent: urllib Disallow: / User-agent: nutch Disallow: / # ----------------------------------------------- # BLOCK: Sensitive/Internal Paths from ALL bots # ----------------------------------------------- User-agent: * Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /xmlrpc.php Disallow: /comments/feed/ Disallow: /?s= Disallow: /search/ Disallow: /tag/ Disallow: /author/ Disallow: /trackback/ Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /admin/ Disallow: /login/ Disallow: /register/ Disallow: /*.php$ Disallow: /*.json$ Disallow: /cart/ Disallow: /checkout/ # ----------------------------------------------- # Sitemap # ----------------------------------------------- Sitemap: https://www.telugutimes.net/sitemap.xml Sitemap: https://www.telugutimes.net/sitemap_index.xml Sitemap: https://www.telugutimes.net/news-sitemap.xml