robots.txt Checker / Analyzer
URL for check: pagarminimalisterbaru.wordpress.com
Robots.txt: http://www.pagarminimalisterbaru.wordpress.com/robots.txt
File size: -1 bytes
Checking Results | |||
User-agent | Access | Robots.txt Rule | |
IRLbot | Allowed | Allow | |
* | Allowed | Allow |
Robots.txt Content :
- # This file was generated on Fri, 20 Nov 2015 20:48:13 +0000
- # If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead.
- # Please see http://en.wordpress.com/firehose/ for more details.
- Sitemap: https://pagarminimalisterbaru.wordpress.com/sitemap.xml
- Sitemap: https://pagarminimalisterbaru.wordpress.com/news-sitemap.xml
- User-agent: IRLbot
- Crawl-delay: 3600
- User-agent: *
- Disallow: /next/
- User-agent: *
- Disallow: /mshots/v1/
- # har har
- User-agent: *
- Disallow: /activate/
- User-agent: *
- Disallow: /public.api/
- # MT refugees
- User-agent: *
- Disallow: /cgi-bin/
- User-agent: *
- Disallow: /wp-login.php
- User-agent: *
- Disallow: /wp-admin/
If you like this tool, you can add our button for free to check access to your pages for crawlers. Choose and then insert code to those web pages where you want to see it.
Big Button 1: | ![]() | |
Big button 2: | this website | |
Small Button: | ![]() |
IMPORTANT: Please Don't modify code, otherwise we can't guarantee the correct work of our tools.