Parsing Logs with Multiline Stack Traces

Handling Multiline Log Entries

Many application errors produce multiline output — Java stack traces, Python tracebacks, and Node.js error stacks span multiple lines. Understanding how to work with these in the parser helps you extract maximum value from error logs.

The Multiline Challenge

Consider a Java application log:

2024-01-15 10:30:00 ERROR com.example.api.UserService - Failed to create user
java.sql.SQLException: Connection pool exhausted
    at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
    at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
    at com.example.api.UserService.createUser(UserService.java:45)
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)

The first line is a valid log entry, but the subsequent lines are continuation of the same error.

Strategies for Multiline Logs

Strategy 1: Pre-process before parsing Combine multiline entries into single-line JSON before pasting:

{"timestamp":"2024-01-15 10:30:00","level":"ERROR","logger":"com.example.api.UserService","message":"Failed to create user\njava.sql.SQLException: Connection pool exhausted\n    at com.zaxxer.hikari..."}

Strategy 2: Use JSON structured logging Configure your application to emit JSON logs with the stack trace as a field:

{"timestamp":"2024-01-15T10:30:00Z","level":"error","logger":"UserService","message":"Failed to create user","stackTrace":"java.sql.SQLException: Connection pool exhausted\n    at ...","errorCode":"CONNECTION_POOL_EXHAUSTED"}

Strategy 3: Parse first lines only In many cases, the first line of each log entry contains the essential information (timestamp, severity, source, error message). The stack trace provides detail but the first line is sufficient for triage.

Best Practice: Structured Error Logging

The most effective approach is to configure your application to emit structured (JSON) logs where stack traces are embedded as a single field. This makes logs machine-parseable regardless of the error type:

# Python example with structlog
import structlog
logger = structlog.get_logger()

try:
    process_data()
except Exception as e:
    logger.error("Processing failed", error=str(e), stack=traceback.format_exc())