{"id":36212,"date":"2023-08-24T11:45:00","date_gmt":"2023-08-24T11:45:00","guid":{"rendered":"https:\/\/handlewife.com\/?p=36212"},"modified":"2023-08-23T08:45:56","modified_gmt":"2023-08-23T08:45:56","slug":"webcrawler-gptbot","status":"publish","type":"post","link":"https:\/\/handlewife.com\/webcrawler-gptbot\/","title":{"rendered":"OpenAI Unveils GPTBot Web Crawler: How to Block it?"},"content":{"rendered":"\n<p>Open AI has released a new web crawler that goes by the name of GPTBot. It is a way Open AI will teach more new things to the GPT-3, 4, and 5 large language models (LLMs). To date, ChatGPT is one of the most capable AI systems ever built. Web crawlers come to your Bing and Google sites to collect data and to train future AI models with the same data.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"How to Block ChatGPT Crawlers\" width=\"1200\" height=\"675\" src=\"https:\/\/www.youtube.com\/embed\/qBah5SaaMhc?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>If you allow access to the webcrawler GPTBot you are contributing to making it more efficient in a smaller period. Open AI makes a note of all that the GPTBot crawls. Using a web crawler is a means to an end huge amount of data can be channeled through LLMs that can easily learn it in no time.<\/p>\n\n\n\n<p>The company has recently claimed that it will siphon out details of pages that have paywall access. They will collect information that personally identifies these sites and their owners as well as their site details. They will also look into text that is violating the Open AI policies currently.<\/p>\n\n\n\n<p><strong>Also Read: <a href=\"https:\/\/handlewife.com\/screensharing-for-video-calls\/\">WhatsApp Introduces Video Call Screensharing Feature<\/a><\/strong><\/p>\n\n\n\n<p>But at the same time, they have also released data on how to block the webcrawler GPTBot. You can disallow GPTBot to crawl the Robot.txt files and can even block its IP address. When you block GPTBot from crawling your data, you are taking the first step towards inhibiting ChatGPT from using your data to train the LLMs.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Here\u2019s what you can do to prevent using your data for training LLMs;<\/strong><\/h3>\n\n\n\n<ul>\n<li><strong>Add GPTBot to your site\u2019s robot.txt.<\/strong><\/li>\n<\/ul>\n\n\n\n<p>User-agent: GPTBot&nbsp;<\/p>\n\n\n\n<p>Disallow: \/&nbsp;<\/p>\n\n\n\n<ul>\n<li><strong>If you want to block some parts of your website from being read by GPTBot&nbsp;<\/strong><\/li>\n<\/ul>\n\n\n\n<p>User-agent: GPTBot<\/p>\n\n\n\n<p>Allow: \/directory-1\/<\/p>\n\n\n\n<p>Disallow: \/directory-2\/<\/p>\n\n\n\n<p>After releasing it, Open AI is already surrounded by a pile of lawsuits.\u00a0<\/p>\n\n\n\n<p><strong>Also Read: <a href=\"https:\/\/handlewife.com\/steal-your-passwords\/\">AI: Listening to Keystrokes to Steal Passwords<\/a><\/strong><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What are the charges that webcrawler GPTBot have been accused of?<\/h3>\n\n\n\n<ol type=\"1\" start=\"1\">\n<li>AI toll stealing data from users without permission is one allegation that stands against them in the lawsuit.<\/li>\n\n\n\n<li>Even a copyright infringement case has been put forth against them.<\/li>\n\n\n\n<li>Some companies like Stack Overflow, Twitter, and Reddit plan to charge these AI companies for granting access to their data.<\/li>\n\n\n\n<li>Companies like Adobe plan to mark data as not for training by using an anti-impersonation law.<\/li>\n\n\n\n<li>Open AI and other AI companies like them have already signed a deal with the White House. According to the deal, the companies would develop a watermarking system to intimate people if anything was AI generation.<\/li>\n\n\n\n<li>However, these companies made no promise whatsoever about using internet data or anything around stopping to use them any longer.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>Open AI has released a new web crawler that goes by the name of GPTBot. It is a way Open &#8230; <\/p>\n<p class=\"read-more-container\"><a title=\"OpenAI Unveils GPTBot Web Crawler: How to Block it?\" class=\"read-more button\" href=\"https:\/\/handlewife.com\/webcrawler-gptbot\/#more-36212\" aria-label=\"More on OpenAI Unveils GPTBot Web Crawler: How to Block it?\">Read more<\/a><\/p>\n","protected":false},"author":3,"featured_media":36213,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3552],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>OpenAI Unveils GPTBot Web Crawler: How to Block it?<\/title>\n<meta name=\"description\" content=\"The webcrawler GPTBot is introduced by OpenAI and it also enlists the way to block it Learn more about the process here.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/handlewife.com\/webcrawler-gptbot\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"OpenAI Unveils GPTBot Web Crawler: How to Block it?\" \/>\n<meta property=\"og:description\" content=\"The webcrawler GPTBot is introduced by OpenAI and it also enlists the way to block it Learn more about the process here.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/handlewife.com\/webcrawler-gptbot\/\" \/>\n<meta property=\"og:site_name\" content=\"Handlewife.com\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Handle-wife-931932293630274\/\" \/>\n<meta property=\"article:published_time\" content=\"2023-08-24T11:45:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-08-23T08:45:56+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/handlewife.com\/wp-content\/uploads\/2023\/08\/OpenAI-releases-webcrawler-GPTBot-how-to-block-it.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"640\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"author\" content=\"Ajay Kumar\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@handlewife123\" \/>\n<meta name=\"twitter:site\" content=\"@handlewife123\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"NewsArticle\",\"@id\":\"https:\/\/handlewife.com\/webcrawler-gptbot\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/handlewife.com\/webcrawler-gptbot\/\"},\"author\":{\"name\":\"Ajay Kumar\",\"@id\":\"https:\/\/handlewife.com\/#\/schema\/person\/7ec337322eee0b7086db53d716101a25\"},\"headline\":\"OpenAI Unveils GPTBot Web Crawler: How to Block it?\",\"datePublished\":\"2023-08-24T11:45:00+00:00\",\"dateModified\":\"2023-08-23T08:45:56+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/handlewife.com\/webcrawler-gptbot\/\"},\"wordCount\":460,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/handlewife.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/handlewife.com\/webcrawler-gptbot\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/handlewife.com\/wp-content\/uploads\/2023\/08\/OpenAI-releases-webcrawler-GPTBot-how-to-block-it.webp\",\"articleSection\":[\"News\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/handlewife.com\/webcrawler-gptbot\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/handlewife.com\/webcrawler-gptbot\/\",\"url\":\"https:\/\/handlewife.com\/webcrawler-gptbot\/\",\"name\":\"OpenAI Unveils GPTBot Web Crawler: How to Block it?\",\"isPartOf\":{\"@id\":\"https:\/\/handlewife.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/handlewife.com\/webcrawler-gptbot\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/handlewife.com\/webcrawler-gptbot\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/handlewife.com\/wp-content\/uploads\/2023\/08\/OpenAI-releases-webcrawler-GPTBot-how-to-block-it.webp\",\"datePublished\":\"2023-08-24T11:45:00+00:00\",\"dateModified\":\"2023-08-23T08:45:56+00:00\",\"description\":\"The webcrawler GPTBot is introduced by OpenAI and it also enlists the way to block it Learn more about the process here.\",\"breadcrumb\":{\"@id\":\"https:\/\/handlewife.com\/webcrawler-gptbot\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/handlewife.com\/webcrawler-gptbot\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/handlewife.com\/webcrawler-gptbot\/#primaryimage\",\"url\":\"https:\/\/handlewife.com\/wp-content\/uploads\/2023\/08\/OpenAI-releases-webcrawler-GPTBot-how-to-block-it.webp\",\"contentUrl\":\"https:\/\/handlewife.com\/wp-content\/uploads\/2023\/08\/OpenAI-releases-webcrawler-GPTBot-how-to-block-it.webp\",\"width\":1024,\"height\":640,\"caption\":\"webcrawler GPTBot\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/handlewife.com\/webcrawler-gptbot\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/handlewife.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"OpenAI Unveils GPTBot Web Crawler: How to Block it?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/handlewife.com\/#website\",\"url\":\"https:\/\/handlewife.com\/\",\"name\":\"Handlewife.com\",\"description\":\"Tech, Apps, Games &amp; Entertainment\",\"publisher\":{\"@id\":\"https:\/\/handlewife.com\/#organization\"},\"alternateName\":\"Handlewife\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/handlewife.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/handlewife.com\/#organization\",\"name\":\"Handlewife.com\",\"url\":\"https:\/\/handlewife.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/handlewife.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/handlewife.com\/wp-content\/uploads\/2023\/07\/handlewife-logo.png\",\"contentUrl\":\"https:\/\/handlewife.com\/wp-content\/uploads\/2023\/07\/handlewife-logo.png\",\"width\":350,\"height\":70,\"caption\":\"Handlewife.com\"},\"image\":{\"@id\":\"https:\/\/handlewife.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/Handle-wife-931932293630274\/\",\"https:\/\/twitter.com\/handlewife123\",\"https:\/\/www.instagram.com\/handlewife123\/\",\"https:\/\/www.linkedin.com\/in\/handlewife-dotcom-210a28158\/\",\"https:\/\/myspace.ge\/profile-381204\",\"https:\/\/in.pinterest.com\/handlewifecom\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/handlewife.com\/#\/schema\/person\/7ec337322eee0b7086db53d716101a25\",\"name\":\"Ajay Kumar\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/handlewife.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/af44c35c7334edd5d2cfa3a404dc3834?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/af44c35c7334edd5d2cfa3a404dc3834?s=96&d=mm&r=g\",\"caption\":\"Ajay Kumar\"},\"description\":\"Ajay Kumar is an accomplished writer known for crafting immersive and compelling stories that capture the imagination.\",\"sameAs\":[\"https:\/\/handlewife.com\/\"],\"url\":\"https:\/\/handlewife.com\/author\/ajay766\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"OpenAI Unveils GPTBot Web Crawler: How to Block it?","description":"The webcrawler GPTBot is introduced by OpenAI and it also enlists the way to block it Learn more about the process here.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/handlewife.com\/webcrawler-gptbot\/","og_locale":"en_US","og_type":"article","og_title":"OpenAI Unveils GPTBot Web Crawler: How to Block it?","og_description":"The webcrawler GPTBot is introduced by OpenAI and it also enlists the way to block it Learn more about the process here.","og_url":"https:\/\/handlewife.com\/webcrawler-gptbot\/","og_site_name":"Handlewife.com","article_publisher":"https:\/\/www.facebook.com\/Handle-wife-931932293630274\/","article_published_time":"2023-08-24T11:45:00+00:00","article_modified_time":"2023-08-23T08:45:56+00:00","og_image":[{"width":1024,"height":640,"url":"https:\/\/handlewife.com\/wp-content\/uploads\/2023\/08\/OpenAI-releases-webcrawler-GPTBot-how-to-block-it.webp","type":"image\/webp"}],"author":"Ajay Kumar","twitter_card":"summary_large_image","twitter_creator":"@handlewife123","twitter_site":"@handlewife123","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"NewsArticle","@id":"https:\/\/handlewife.com\/webcrawler-gptbot\/#article","isPartOf":{"@id":"https:\/\/handlewife.com\/webcrawler-gptbot\/"},"author":{"name":"Ajay Kumar","@id":"https:\/\/handlewife.com\/#\/schema\/person\/7ec337322eee0b7086db53d716101a25"},"headline":"OpenAI Unveils GPTBot Web Crawler: How to Block it?","datePublished":"2023-08-24T11:45:00+00:00","dateModified":"2023-08-23T08:45:56+00:00","mainEntityOfPage":{"@id":"https:\/\/handlewife.com\/webcrawler-gptbot\/"},"wordCount":460,"commentCount":0,"publisher":{"@id":"https:\/\/handlewife.com\/#organization"},"image":{"@id":"https:\/\/handlewife.com\/webcrawler-gptbot\/#primaryimage"},"thumbnailUrl":"https:\/\/handlewife.com\/wp-content\/uploads\/2023\/08\/OpenAI-releases-webcrawler-GPTBot-how-to-block-it.webp","articleSection":["News"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/handlewife.com\/webcrawler-gptbot\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/handlewife.com\/webcrawler-gptbot\/","url":"https:\/\/handlewife.com\/webcrawler-gptbot\/","name":"OpenAI Unveils GPTBot Web Crawler: How to Block it?","isPartOf":{"@id":"https:\/\/handlewife.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/handlewife.com\/webcrawler-gptbot\/#primaryimage"},"image":{"@id":"https:\/\/handlewife.com\/webcrawler-gptbot\/#primaryimage"},"thumbnailUrl":"https:\/\/handlewife.com\/wp-content\/uploads\/2023\/08\/OpenAI-releases-webcrawler-GPTBot-how-to-block-it.webp","datePublished":"2023-08-24T11:45:00+00:00","dateModified":"2023-08-23T08:45:56+00:00","description":"The webcrawler GPTBot is introduced by OpenAI and it also enlists the way to block it Learn more about the process here.","breadcrumb":{"@id":"https:\/\/handlewife.com\/webcrawler-gptbot\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/handlewife.com\/webcrawler-gptbot\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/handlewife.com\/webcrawler-gptbot\/#primaryimage","url":"https:\/\/handlewife.com\/wp-content\/uploads\/2023\/08\/OpenAI-releases-webcrawler-GPTBot-how-to-block-it.webp","contentUrl":"https:\/\/handlewife.com\/wp-content\/uploads\/2023\/08\/OpenAI-releases-webcrawler-GPTBot-how-to-block-it.webp","width":1024,"height":640,"caption":"webcrawler GPTBot"},{"@type":"BreadcrumbList","@id":"https:\/\/handlewife.com\/webcrawler-gptbot\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/handlewife.com\/"},{"@type":"ListItem","position":2,"name":"OpenAI Unveils GPTBot Web Crawler: How to Block it?"}]},{"@type":"WebSite","@id":"https:\/\/handlewife.com\/#website","url":"https:\/\/handlewife.com\/","name":"Handlewife.com","description":"Tech, Apps, Games &amp; Entertainment","publisher":{"@id":"https:\/\/handlewife.com\/#organization"},"alternateName":"Handlewife","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/handlewife.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/handlewife.com\/#organization","name":"Handlewife.com","url":"https:\/\/handlewife.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/handlewife.com\/#\/schema\/logo\/image\/","url":"https:\/\/handlewife.com\/wp-content\/uploads\/2023\/07\/handlewife-logo.png","contentUrl":"https:\/\/handlewife.com\/wp-content\/uploads\/2023\/07\/handlewife-logo.png","width":350,"height":70,"caption":"Handlewife.com"},"image":{"@id":"https:\/\/handlewife.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Handle-wife-931932293630274\/","https:\/\/twitter.com\/handlewife123","https:\/\/www.instagram.com\/handlewife123\/","https:\/\/www.linkedin.com\/in\/handlewife-dotcom-210a28158\/","https:\/\/myspace.ge\/profile-381204","https:\/\/in.pinterest.com\/handlewifecom\/"]},{"@type":"Person","@id":"https:\/\/handlewife.com\/#\/schema\/person\/7ec337322eee0b7086db53d716101a25","name":"Ajay Kumar","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/handlewife.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/af44c35c7334edd5d2cfa3a404dc3834?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/af44c35c7334edd5d2cfa3a404dc3834?s=96&d=mm&r=g","caption":"Ajay Kumar"},"description":"Ajay Kumar is an accomplished writer known for crafting immersive and compelling stories that capture the imagination.","sameAs":["https:\/\/handlewife.com\/"],"url":"https:\/\/handlewife.com\/author\/ajay766\/"}]}},"_links":{"self":[{"href":"https:\/\/handlewife.com\/wp-json\/wp\/v2\/posts\/36212"}],"collection":[{"href":"https:\/\/handlewife.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/handlewife.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/handlewife.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/handlewife.com\/wp-json\/wp\/v2\/comments?post=36212"}],"version-history":[{"count":5,"href":"https:\/\/handlewife.com\/wp-json\/wp\/v2\/posts\/36212\/revisions"}],"predecessor-version":[{"id":37340,"href":"https:\/\/handlewife.com\/wp-json\/wp\/v2\/posts\/36212\/revisions\/37340"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/handlewife.com\/wp-json\/wp\/v2\/media\/36213"}],"wp:attachment":[{"href":"https:\/\/handlewife.com\/wp-json\/wp\/v2\/media?parent=36212"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/handlewife.com\/wp-json\/wp\/v2\/categories?post=36212"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/handlewife.com\/wp-json\/wp\/v2\/tags?post=36212"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}