Detect Social Media Bots from User-Agent
Identify social media crawlers (Facebook, Twitter, LinkedIn) from User-Agent strings. Understand how social platforms fetch Open Graph and Twitter Card data.
Detailed Explanation
Detecting Social Media Crawlers
Social media platforms use dedicated bots to fetch page metadata, Open Graph tags, and Twitter Cards for generating link previews.
Facebook Crawler
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
Or the rendering variant:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0
Key tokens: facebookexternalhit, Facebot
Twitter/X Crawler
Twitterbot/1.0
Twitter's bot fetches pages to generate Twitter Card previews. It reads <meta name="twitter:card"> and Open Graph tags.
LinkedIn Crawler
LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)
LinkedIn's crawler fetches Open Graph data for link previews in posts and messages.
Other Social Crawlers
- Pinterest:
Pinterest/0.2 (+http://www.pinterest.com/) - Slack:
Slackbot-LinkExpanding 1.0orSlackbot 1.0 - Discord:
Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com) - Telegram:
TelegramBot (like TwitterBot) - WhatsApp:
WhatsApp/2.x
Why Social Bot Detection Matters
Social crawlers affect your site in specific ways:
- They trigger server requests when users share links — this can spike server load
- They execute minimal JavaScript — if your OG tags are rendered client-side, social crawlers may not see them
- They cache aggressively — updates to OG tags may not appear immediately
- They may not follow redirects the same way browsers do
Server-Side Rendering for Social Bots
When you detect a social media bot, consider serving a server-rendered page with proper OG tags, even if your site normally uses client-side rendering. This ensures link previews display correctly.
Use Case
Web developers detect social media bots to ensure Open Graph and Twitter Card meta tags are served correctly for link previews. Marketing teams use social bot detection to track how frequently their pages are being shared. DevOps teams monitor social bot traffic patterns during viral content events.