Detect Command-Line HTTP Clients from User-Agent

Detecting Command-Line HTTP Clients

Command-line HTTP clients use simple, distinctive User-Agent strings that are easy to identify. They often indicate automated access, scripting, or API testing.

Common CLI Client User-Agents

curl:

curl/8.4.0

The most widely used command-line HTTP client. The UA is just curl/VERSION unless the user overrides it with -A or --user-agent.

wget:

Wget/1.21.4

GNU Wget, commonly used for downloading files and mirroring websites.

Python Requests:

python-requests/2.31.0

The most popular Python HTTP library. This is the default UA; developers often override it.

Python urllib:

Python-urllib/3.11

Python's built-in HTTP library.

Go HTTP Client:

Go-http-client/2.0

Go's standard library HTTP client.

Node.js:

node-fetch/2.6.7

Or axios/1.6.2, got/14.0.0, or undici.

Java:

Java/17.0.2

Java's HttpURLConnection default UA.

Ruby:

Faraday v2.7.0

Or rest-client/2.1.0, HTTParty.

Detection Patterns

CLI clients are identified by the absence of typical browser tokens and the presence of library-specific identifiers:

1. No Mozilla/5.0 prefix → likely CLI client or bot
2. Simple NAME/VERSION format → curl, wget, python-requests
3. Language name in UA → Java, Python, Go, Ruby

Security Implications

CLI client UAs often indicate:

• API testing and development
• Automated scripts and cron jobs
• Potential scraping or vulnerability scanning
• Health check monitoring systems
• Webhook delivery systems

Rate Limiting Considerations

Many APIs apply different rate limits to CLI clients vs. browser traffic. Detecting curl, python-requests, or similar UAs can trigger API-appropriate rate limits and authentication requirements.