{"id":1134,"date":"2019-07-06T05:54:05","date_gmt":"2019-07-06T05:54:05","guid":{"rendered":"http:\/\/www.amritsardigitalacademy.in\/blog\/?p=1134"},"modified":"2022-04-06T06:39:07","modified_gmt":"2022-04-06T06:39:07","slug":"google-will-discontinue-support-for-robots-txt","status":"publish","type":"post","link":"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/","title":{"rendered":"Google Will Discontinue Support for Robots.txt from September 2019"},"content":{"rendered":"<h2>NO SUPPORT FOR ROBOTS.TXT BY GOOGLE<\/h2>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-1137\" src=\"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-content\/uploads\/2019\/07\/robot-waving.gif\" alt=\"Robots.ttxt\" width=\"299\" height=\"417\"><\/p>\n<p>Google has recently announced to step back from supporting undocumented and unsupported rules in robots.txt as per their official statement. This announcement will come into practice officially from September 1, 2019. Google\u2019s official tweet has said-<\/p>\n<p>\u201cToday we\u2019re saying goodbye to undocumented and unsupported rules in robots.txt<br \/>\nIf you were relying on these rules, learn about your options in our blog post.\u201d<\/p>\n<p>Moreover Google\u2019s announcement has clearly stated that in order to maintain a healthy ecosystem and prepare for future releases, they are retiring all unsupported and unpublished rules such as NoIndex from the said date. Google till date used to obey the NoIndex directive. According to Stone Temple in 11 out of 12 cases the results were positive but as the results are not 100%, so it does not always work.<\/p>\n<p>Robots.txt is a text file that we use to instruct the Google bots that which pages should be crawled. It also specifies the pages that should not be indexed while crawling. Many organisations use this to prevent confidential data to be scanned by searched engines or web crawlers. Moreover, it is used to prevent overloading of site with requests also. It does not mean that the pages are hidden from Google but they cannot be indexed by Google Bots.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-1136\" src=\"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-content\/uploads\/2019\/07\/f0ddfaf9e700187a1e69294c212b1e4d-root-directory-user-agent.jpg\" alt=\"Robots.txt\" width=\"800\" height=\"552\" srcset=\"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-content\/uploads\/2019\/07\/f0ddfaf9e700187a1e69294c212b1e4d-root-directory-user-agent.jpg 632w, https:\/\/www.amritsardigitalacademy.in\/blog\/wp-content\/uploads\/2019\/07\/f0ddfaf9e700187a1e69294c212b1e4d-root-directory-user-agent-300x207.jpg 300w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/><\/p>\n<h2>WAYS TO CONTROL CRAWLING<\/h2>\n<p><strong>NoIndex in Robot Meta tags<\/strong> \u2013 The NoIndex is supported in both HTTP response headers and HTML. The NoIndex tag is quite effective in masking the URLs and prevent them from getting scanned by Google bots.<\/p>\n<p><strong>HTTP Standard Response Codes<\/strong> \u2013 The response codes of 404 and 410 will remove the URLs from the Google\u2019s index if they are crawled or scanned by search engine crawler.<\/p>\n<p><strong>Protecting through password<\/strong> \u2013 If we create a login and password to enter a page, it will not be indexed by Google.<\/p>\n<p><strong>Usage of Disallow<\/strong> &#8212; As the search engines can only index the visible pages, blocking that page means it will not be indexed. However, the search engine can also index URLs according to the links from other pages and without reading the content, we aim to reduce the visibility of the page.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>NO SUPPORT FOR ROBOTS.TXT BY GOOGLE Google has recently announced to step back from supporting undocumented and unsupported rules in robots.txt as per their official statement. This announcement will come into practice officially from September 1, 2019. Google\u2019s official tweet has said- \u201cToday we\u2019re saying goodbye to undocumented and unsupported rules in robots.txt If you [&hellip;]<\/p>\n","protected":false},"author":19,"featured_media":2639,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[],"class_list":["post-1134","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-digital-marketing"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.2 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Google Will Discontinue Support for Robots.txt from September 2019<\/title>\n<meta name=\"description\" content=\"Google will discontinue supporting robots.txt files from September 2019 as they have stepped back from supporting undocumented files.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Google Will Discontinue Support for Robots.txt from September 2019\" \/>\n<meta property=\"og:description\" content=\"Google will discontinue supporting robots.txt files from September 2019 as they have stepped back from supporting undocumented files.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/AmritsarDigitalAcademy\/\" \/>\n<meta property=\"article:published_time\" content=\"2019-07-06T05:54:05+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-04-06T06:39:07+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-content\/uploads\/2019\/07\/robots-txt.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Dilpreet Kaur\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@AmritsarAcademy\" \/>\n<meta name=\"twitter:site\" content=\"@AmritsarAcademy\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Dilpreet Kaur\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/\"},\"author\":{\"name\":\"Dilpreet Kaur\",\"@id\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/#\/schema\/person\/122f5f3e918e1dc65c13bb67a15808ee\"},\"headline\":\"Google Will Discontinue Support for Robots.txt from September 2019\",\"datePublished\":\"2019-07-06T05:54:05+00:00\",\"dateModified\":\"2022-04-06T06:39:07+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/\"},\"wordCount\":368,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/#organization\"},\"articleSection\":[\"Digital Marketing\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/\",\"url\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/\",\"name\":\"Google Will Discontinue Support for Robots.txt from September 2019\",\"isPartOf\":{\"@id\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/#website\"},\"datePublished\":\"2019-07-06T05:54:05+00:00\",\"dateModified\":\"2022-04-06T06:39:07+00:00\",\"description\":\"Google will discontinue supporting robots.txt files from September 2019 as they have stepped back from supporting undocumented files.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Google Will Discontinue Support for Robots.txt from September 2019\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/#website\",\"url\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/\",\"name\":\"\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/#organization\",\"name\":\"Amritsar Digital Academy\",\"url\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-content\/uploads\/2018\/12\/ADA-Final-Logo.png\",\"contentUrl\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-content\/uploads\/2018\/12\/ADA-Final-Logo.png\",\"width\":600,\"height\":138,\"caption\":\"Amritsar Digital Academy\"},\"image\":{\"@id\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/AmritsarDigitalAcademy\/\",\"https:\/\/twitter.com\/AmritsarAcademy\",\"https:\/\/www.instagram.com\/amritsardigitalacademy\/\",\"https:\/\/www.linkedin.com\/company\/amritsardigitalacademy\/\",\"https:\/\/in.pinterest.com\/amritsardigitalacademy\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/#\/schema\/person\/122f5f3e918e1dc65c13bb67a15808ee\",\"name\":\"Dilpreet Kaur\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a4534b36f242d49bf2e997b05216349a?s=96&d=identicon&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a4534b36f242d49bf2e997b05216349a?s=96&d=identicon&r=g\",\"caption\":\"Dilpreet Kaur\"},\"sameAs\":[\"https:\/\/amritsardigitalacademy.in\/\"],\"url\":\"https:\/\/www.amritsardigitalacademy.in\/blog\/author\/dkaur\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Google Will Discontinue Support for Robots.txt from September 2019","description":"Google will discontinue supporting robots.txt files from September 2019 as they have stepped back from supporting undocumented files.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/","og_locale":"en_US","og_type":"article","og_title":"Google Will Discontinue Support for Robots.txt from September 2019","og_description":"Google will discontinue supporting robots.txt files from September 2019 as they have stepped back from supporting undocumented files.","og_url":"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/","article_publisher":"https:\/\/www.facebook.com\/AmritsarDigitalAcademy\/","article_published_time":"2019-07-06T05:54:05+00:00","article_modified_time":"2022-04-06T06:39:07+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-content\/uploads\/2019\/07\/robots-txt.png","type":"image\/png"}],"author":"Dilpreet Kaur","twitter_card":"summary_large_image","twitter_creator":"@AmritsarAcademy","twitter_site":"@AmritsarAcademy","twitter_misc":{"Written by":"Dilpreet Kaur","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/#article","isPartOf":{"@id":"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/"},"author":{"name":"Dilpreet Kaur","@id":"https:\/\/www.amritsardigitalacademy.in\/blog\/#\/schema\/person\/122f5f3e918e1dc65c13bb67a15808ee"},"headline":"Google Will Discontinue Support for Robots.txt from September 2019","datePublished":"2019-07-06T05:54:05+00:00","dateModified":"2022-04-06T06:39:07+00:00","mainEntityOfPage":{"@id":"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/"},"wordCount":368,"commentCount":0,"publisher":{"@id":"https:\/\/www.amritsardigitalacademy.in\/blog\/#organization"},"articleSection":["Digital Marketing"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/","url":"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/","name":"Google Will Discontinue Support for Robots.txt from September 2019","isPartOf":{"@id":"https:\/\/www.amritsardigitalacademy.in\/blog\/#website"},"datePublished":"2019-07-06T05:54:05+00:00","dateModified":"2022-04-06T06:39:07+00:00","description":"Google will discontinue supporting robots.txt files from September 2019 as they have stepped back from supporting undocumented files.","breadcrumb":{"@id":"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.amritsardigitalacademy.in\/blog\/google-will-discontinue-support-for-robots-txt\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.amritsardigitalacademy.in\/blog\/"},{"@type":"ListItem","position":2,"name":"Google Will Discontinue Support for Robots.txt from September 2019"}]},{"@type":"WebSite","@id":"https:\/\/www.amritsardigitalacademy.in\/blog\/#website","url":"https:\/\/www.amritsardigitalacademy.in\/blog\/","name":"","description":"","publisher":{"@id":"https:\/\/www.amritsardigitalacademy.in\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.amritsardigitalacademy.in\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.amritsardigitalacademy.in\/blog\/#organization","name":"Amritsar Digital Academy","url":"https:\/\/www.amritsardigitalacademy.in\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.amritsardigitalacademy.in\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-content\/uploads\/2018\/12\/ADA-Final-Logo.png","contentUrl":"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-content\/uploads\/2018\/12\/ADA-Final-Logo.png","width":600,"height":138,"caption":"Amritsar Digital Academy"},"image":{"@id":"https:\/\/www.amritsardigitalacademy.in\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/AmritsarDigitalAcademy\/","https:\/\/twitter.com\/AmritsarAcademy","https:\/\/www.instagram.com\/amritsardigitalacademy\/","https:\/\/www.linkedin.com\/company\/amritsardigitalacademy\/","https:\/\/in.pinterest.com\/amritsardigitalacademy\/"]},{"@type":"Person","@id":"https:\/\/www.amritsardigitalacademy.in\/blog\/#\/schema\/person\/122f5f3e918e1dc65c13bb67a15808ee","name":"Dilpreet Kaur","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.amritsardigitalacademy.in\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/a4534b36f242d49bf2e997b05216349a?s=96&d=identicon&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a4534b36f242d49bf2e997b05216349a?s=96&d=identicon&r=g","caption":"Dilpreet Kaur"},"sameAs":["https:\/\/amritsardigitalacademy.in\/"],"url":"https:\/\/www.amritsardigitalacademy.in\/blog\/author\/dkaur\/"}]}},"_links":{"self":[{"href":"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-json\/wp\/v2\/posts\/1134"}],"collection":[{"href":"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-json\/wp\/v2\/users\/19"}],"replies":[{"embeddable":true,"href":"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-json\/wp\/v2\/comments?post=1134"}],"version-history":[{"count":0,"href":"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-json\/wp\/v2\/posts\/1134\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-json\/wp\/v2\/media\/2639"}],"wp:attachment":[{"href":"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-json\/wp\/v2\/media?parent=1134"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-json\/wp\/v2\/categories?post=1134"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.amritsardigitalacademy.in\/blog\/wp-json\/wp\/v2\/tags?post=1134"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}