Ping IP / IP Checker Traceroute DNS Checker WHOIS Lookup What is My IP PageRank Checker Reverse DNS

robots.txt Checker / Analyzer


Update Cache

URL for check: pagarminimalisterbaru.wordpress.com
Robots.txt: http://www.pagarminimalisterbaru.wordpress.com/robots.txt
File size: -1 bytes


Checking Results
User-agentAccessRobots.txt Rule
IRLbotAllowedAllow
*AllowedAllow

Robots.txt Content :

  1. # This file was generated on Fri, 20 Nov 2015 20:48:13 +0000
  2. # If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead.
  3. # Please see http://en.wordpress.com/firehose/ for more details.
  4. Sitemap: https://pagarminimalisterbaru.wordpress.com/sitemap.xml
  5. Sitemap: https://pagarminimalisterbaru.wordpress.com/news-sitemap.xml
  6. User-agent: IRLbot
  7. Crawl-delay: 3600
  8. User-agent: *
  9. Disallow: /next/
  10. User-agent: *
  11. Disallow: /mshots/v1/
  12. # har har
  13. User-agent: *
  14. Disallow: /activate/
  15. User-agent: *
  16. Disallow: /public.api/
  17. # MT refugees
  18. User-agent: *
  19. Disallow: /cgi-bin/
  20. User-agent: *
  21. Disallow: /wp-login.php
  22. User-agent: *
  23. Disallow: /wp-admin/

If you like this tool, you can add our button for free to check access to your pages for crawlers. Choose and then insert code to those web pages where you want to see it.


 

IMPORTANT: Please Don't modify code, otherwise we can't guarantee the correct work of our tools.