{"id":39,"date":"2020-04-12T12:00:00","date_gmt":"2020-04-12T12:00:00","guid":{"rendered":"https:\/\/carrousel.studio\/?p=39"},"modified":"2020-08-21T16:09:17","modified_gmt":"2020-08-21T16:09:17","slug":"how-to-use-robots-txt-file-in-seo","status":"publish","type":"post","link":"https:\/\/carrousel.studio\/en\/how-to-use-robots-txt-file-in-seo\/","title":{"rendered":"How to use Robots.txt file in SEO"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Introduction to Robots.txt<\/h2>\n\n\n\n<p>This article gives the basic understanding of :<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>What is a robots.txt file ? <\/li><li>What is its purpose ? <\/li><li>How to create the robots.txt file in Worpdress ? <\/li><li>Introduction to robots.txt instructions. <\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">What is a robots.txt file ?<\/h2>\n\n\n\n<p>Robots.txt is a text file that tells the web robots, from search engines, which pages on your site should be crawl.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why use a robots.txt ?<\/h2>\n\n\n\n<p>Robots.txt is used for SEO. It is one of the ways how to tell search engines what urls should be indexed or not.<br>Search engines uses web crawlers to scan your website.<\/p>\n\n\n\n<p>The purpose of that file, is to minimize the crawl budget of the search engine crawler when looking at your urls.<br>The crawlers have only a limited number of urls, called crawl budget, they are allowed to scan every time they do a pass of your website.<\/p>\n\n\n\n<p>There is many web crawlers that exists<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How and where to create the robots.txt file in WordPress.<\/h2>\n\n\n\n<p>Before jumping into creating a new file, you might want to check if you already have it.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>WordPress will create a robots.txt file for you.<\/li><li>Yoast SEO plugin will also manage the robots.txt file for you.<\/li><\/ul>\n\n\n\n<p>If you don&#8217;t have it already created :<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>Create an empty file named &#8216;robots.txt&#8217;.<\/li><li>Upload robots.txt to your server under the root directory of your website (public_html in many cases).<\/li><li>Test that your new robots file is accessible by accessing in a browser the url: https:\/\/&lt;your-domain-here&gt;\/robots.txt.<\/li><\/ol>\n\n\n\n<p>If you see a blank page with no errors, that means it works, because the content of the robots.txt file is empty.<br>Next step, will be to edit the file and add instructions that the crawlers can read.<br>Note: If the file stays empty, that means that the crawlers have no limitations and will crawl all the publicly accessible url of your site. <\/p>\n\n\n\n<p>If you receive a 404 page, Permissions denied or anything else, there is most likely a misconfiguration on your server files and\/or directories. You will want to contact your hosting provider.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Introduction to robots.txt instructions.<\/h2>\n\n\n\n<p>Note that web crawlers are not forced to obey the instructions in your robots.txt.<br>Fortunately, most of them are doing things properly, just like Googlebot from Google. But keep in mind that it&#8217;s up to the crawler to decide or not if the rule will be applied.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Allowing all crawlers: <strong>User-agent: * Disallow: <\/strong><\/li><li>Blocking all crawlers from all files: <strong>User-agent: * Disallow: \/<\/strong><\/li><li>Blocking a specific file from all crawlers: <strong>User-agent: * Disallow: \/&lt;path-to-file&gt;\/<strong><strong>&lt;file-name&gt;<\/strong><\/strong><\/strong><\/li><li>Blocking a specific folder from all crawlers: <strong>User-agent: * Disallow: \/&lt;specific-folder-here&gt;\/<\/strong><\/li><li>Blocking a specific crawler: <strong>User-agent: Googlebot<\/strong><\/li><\/ul>\n\n\n\n<p>It is also a good practice to tell the robot file where is your sitemap.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Sitemap: https:\/\/&lt;your-domain>\/sitemap.xml<\/code><\/pre>\n\n\n\n<p>Here is a basic example of a robots.txt :<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Accept all Crawler, and prevent crawling the wp-admin url of WordPress.\nUser-agent: *\nDisallow: \/wp-admin\/\nSitemap: https:\/\/&lt;your-domain>\/sitemap.xml<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Fun facts<\/h2>\n\n\n\n<p>Have a look at Google&#8217;s own robots.txt: <a href=\"https:\/\/www.google.com\/robots.txt\" target=\"_blank\" rel=\"noreferrer noopener\">Google&#8217;s robots.txt<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Tell the SEO robots how to crawl your website properly.<\/p>\n","protected":false},"author":1,"featured_media":746,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[12],"tags":[],"class_list":["post-39","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-seo"],"acf":[],"_links":{"self":[{"href":"https:\/\/carrousel.studio\/en\/wp-json\/wp\/v2\/posts\/39","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/carrousel.studio\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/carrousel.studio\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/carrousel.studio\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/carrousel.studio\/en\/wp-json\/wp\/v2\/comments?post=39"}],"version-history":[{"count":37,"href":"https:\/\/carrousel.studio\/en\/wp-json\/wp\/v2\/posts\/39\/revisions"}],"predecessor-version":[{"id":1050,"href":"https:\/\/carrousel.studio\/en\/wp-json\/wp\/v2\/posts\/39\/revisions\/1050"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/carrousel.studio\/en\/wp-json\/wp\/v2\/media\/746"}],"wp:attachment":[{"href":"https:\/\/carrousel.studio\/en\/wp-json\/wp\/v2\/media?parent=39"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/carrousel.studio\/en\/wp-json\/wp\/v2\/categories?post=39"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/carrousel.studio\/en\/wp-json\/wp\/v2\/tags?post=39"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}