↓Skip to main content
  1. Dispatches/

TIL - Block OpenAI and Meta's LLM web crawlers

·53 words·1 min
TIL llm development web

Thanks to this post from Adam Johnson, I’ve now updated my configuration to block OpenAI1 and Meta2 from crawling this website to feed their LLMs.

If you would like to do the same you only need to add these entries to your robots.txt:

User-agent: GPTBot
Disallow: /

User-agent: FacebookBot
Disallow: /

  1. https://platform.openai.com/docs/gptbot ↩︎

  2. https://developers.facebook.com/docs/sharing/bot ↩︎