from Hacker News

The Facebook crawler is hammering the internet

by jedisct1 on 9/1/25, 1:35 PM with 9 comments

• by jedisct1 on 9/1/25, 2:45 PM

  Interestingly, they rotate user agents.

  A few days ago they were identifying as “FaceBot” (not "FacebookBot").

  When that began to be blocked, they switched to reusing the “Facebookexternalhit” user agent they also use for redirects; one people are less likely to block.

• by BoredPositron on 9/1/25, 7:29 PM

  Meta as a company always acts like the most unprofessional idiots if they feel under pressure. They have all the time and ressources to do it right and they never do.

• by emot on 9/1/25, 9:35 PM

  isn't this crawler to generate previews on Facebook? they have others for training and AI stuff. oh well, one never knows with Meta...

  —— The facebookexternalhit/1.1 user agent you're seeing in the logs is a Facebook crawler, specifically used by Facebook’s servers to fetch content (like Open Graph metadata) when:

  Someone shares a link on Facebook or Messenger

  Facebook needs to generate a preview (title, image, description) for that URL

• by N19PEDL2 on 9/1/25, 5:37 PM

  Genuine question: how do they know the bot is from Facebook, apart from what's written in the user agent?

• by bediger4000 on 9/1/25, 1:59 PM

  Would a 404 or a 403 be more appropriate? What if you just want Meta crawlers to go away forever?