Connect with us

Hi, what are you looking for?

Tech News

Now you can block OpenAI’s web crawler

An image of OpenAI’s logo, which looks like a stylized and symmetrical braid.
Image: OpenAI

OpenAI now lets you block its web crawler from scraping your site to help train GPT models.

In a blog post, OpenAI said website operators can specifically disallow its GPTBot crawler on their site’s Robots.txt file or block its IP address. “Web pages crawled with the GPTBot user agent may potentially be used to improve future models and are filtered to remove sources that require paywall access, are known to gather personally identifiable information (PII), or have text that violates our policies,” OpenAI said in the blog post. For sources that don’t fit the excluded criteria, “allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety.”

Blocking the GPTBot may be the…

Continue reading…

You May Also Like


Collaboratively administrate turnkey channels whereas virtual e-tailers. Objectively seize scalable metrics whereas proactive e-services.


Quickly coordinate e-business applications through revolutionary catalysts for change. Seamlessly underwhelm optimal testing procedures processes.

Editor's Pick

David Boaz I’ve written before about whether athletes take state taxes into account when they weigh competing offers. Here’s another example: Grant Williams left...

Editor's Pick

Gene Healy Last week, the New York Times ran a front-page story admiring President Biden’s political acumen on culture-war issues (“Biden Sidesteps Any Notion...