Semantic Logger Structured logs. Better decisions.

Upgrading

Contents

Upgrading to Semantic Logger v5.2

v5.2 sends the configured facility to remote syslog, deprecates the :splunk appender, stops replacing Ruby’s Thread#name, and fixes several security issues. None of the security fixes change the public API, but a few change the output, which matters if you parse the logs or alert on them, and the file appender’s permissions: option can now make add_appender raise for a log file that worked in v5.1.

Set the Sentry user and fingerprint with named tags

The :sentry_ruby appender takes the Sentry user (user_id, username, user_email, ip_address, and user) and fingerprint from named tags or the payload. Taking them from the payload is deprecated and will be removed in v6, since the payload often holds untrusted data, such as a request body, which could then set the user an error is reported for. It emits a deprecation warning the first time it happens. Until then, the named tags win over the payload, which no longer replaces a user field they set, and only a String, Symbol, or Integer is taken from the payload for a user field. Set them with named tags instead, which can now also hold the fingerprint:

# Before:
logger.error("Payment failed", user_id: user.id, fingerprint: ["payment-failed"])

# After:
SemanticLogger.tagged(user_id: user.id, fingerprint: ["payment-failed"]) do
  logger.error("Payment failed")
end

Replace the :splunk appender with :splunk_http

The :splunk appender depends on the splunk-sdk-ruby gem, which was last released in 2015 and does not verify the server’s TLS certificate. It now emits a deprecation warning and will be removed in v6. Send logs to the Splunk HTTP Event Collector instead:

# Before:
SemanticLogger.add_appender(appender: :splunk, index: "main")

# After:
SemanticLogger.add_appender(
  appender: :splunk_http,
  url:      "https://splunk.example.com:8088/services/collector/event",
  token:    "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  index:    "main"
)

Remote syslog sends the configured facility

When logging to a remote syslog server over TCP or UDP, earlier versions sent the wrong facility. The facility: option is documented as one of Ruby’s ::Syslog::LOG_* facility constants, but the value was sent as if it were already the syslog protocol’s facility number. Those constants are the facility number shifted left by 3 bits, so the default ::Syslog::LOG_USER (8) went out as facility 8, which is uucp, instead of facility 1, user. For example a :info message, at the default LOG_NOTICE severity, was sent as <69> rather than <13>. Logging to a local syslog daemon was not affected.

v5.2 sends the facility the constant names. If your syslog server filters or routes on facility, for example an rsyslog rule for uucp.*, change it to the facility you configured: user unless you set one.

The facility: option now only accepts the ::Syslog::LOG_* facility constants, and raises ArgumentError for other values when the appender is created. Previously a constant other than ::Syslog::LOG_KERN, ::Syslog::LOG_USER, or ::Syslog::LOG_MAIL raised ArgumentError while logging. If you passed a bare facility number to get around that, switch to the constant. Check this even if nothing raises: a bare 0, 8, or 16 is now read as ::Syslog::LOG_KERN, ::Syslog::LOG_USER, or ::Syslog::LOG_MAIL, so 16, which used to send local0, now sends mail:

# Before (16 was sent as facility local0, and is now read as ::Syslog::LOG_MAIL):
SemanticLogger.add_appender(appender: :syslog, url: "udp://myloghost:514", facility: 16)

# After:
SemanticLogger.add_appender(appender: :syslog, url: "udp://myloghost:514", facility: ::Syslog::LOG_LOCAL0)

This applies to the :syslog and :syslog_cee formatters. An application that logs only to a remote server, and so does not need the syslog gem, can use the same constants from SemanticLogger::Formatters::Syslog, for example SemanticLogger::Formatters::Syslog::LOG_LOCAL0.

The file appender’s permissions: refuses a log file it cannot secure

With permissions:, the file appender only writes to a regular file owned by the process, with a single hard link, that the file name names directly rather than through a symbolic link. SemanticLogger.add_appender now raises IOError, giving the reason, for any other log file, which v5.1 wrote to, changing the permissions of whatever a link pointed at. So a configuration that worked in v5.1 can now fail at boot, for example permissions: on a log file that is a symbolic link to /dev/stdout, or on a log file that another user created. With permissions:, a log file that cannot be opened also raises IOError, instead of the operating system’s error, such as Errno::ENOENT, which is its cause. A log file that fails the check, or to open, when it is reopened later, on a rotation or after a fork, is reported to the internal logger and not written to until it is fixed, instead of raising from the logging thread.

If you set permissions: on such a log file, either fix the file or remove permissions:. To log to standard output, use io: $stdout:

# Before, with log/production.log a symbolic link to /dev/stdout:
SemanticLogger.add_appender(file_name: "log/production.log", permissions: 0o640)

# After:
SemanticLogger.add_appender(io: $stdout)

Ruby’s own Thread#name is no longer replaced

Earlier versions replaced Thread#name and Thread#name= with Semantic Logger’s own versions, which were written before Ruby had them and kept the name away from Ruby itself. v5.2 leaves Ruby’s own in place, so the name now also reaches the operating system, profilers, and Thread#inspect. On MRI, log entries do not change: a thread that has no name is still logged with its object id.

Code anywhere in the application that calls Thread#name or Thread#name= now gets Ruby’s behaviour:

# Before:
Thread.current.name = worker_number
Thread.current.name.start_with?("worker")

# After:
Thread.current.name = "worker-#{worker_number}"
Thread.current.name&.start_with?("worker")

Tests that expect thread_name: Thread.current.name from the test helpers no longer work on a thread that has no name, such as the main thread on MRI: Thread.current.name is now nil, while the log entry still holds the thread’s object id. The RSpec matchers be_a_semantic_logger_event and log_semantic_logger_event then fail. Worse, assert_semantic_logger_event in Minitest takes nil to mean “do not check”, so the test still passes but no longer checks the thread name. Expect the object id when the thread has no name:

# Before:
assert_semantic_logger_event(event, thread_name: Thread.current.name)

# After:
assert_semantic_logger_event(event, thread_name: Thread.current.name || Thread.current.object_id.to_s)

Semantic Logger’s own threads are now named after their appender, such as SL File, with SL Appenders for the main one, instead of SemanticLogger::Appender::File and SemanticLogger::Appenders. Linux keeps only the first 15 bytes of a thread name, so the longer names all looked the same in tools that list threads, such as top -H. The new names also appear in the thread column of the messages Semantic Logger logs about its own problems, such as an appender that cannot write.

On JRuby:

Output changes

Other behaviour changes

Stricter logger_measure_method arguments

logger_measure_method now raises ArgumentError for a method name that is not a valid Ruby method name, a min_duration that is not a finite number, and an invalid log_exception or level. Numeric Strings, such as min_duration: "100", are still accepted.

Upgrading to Semantic Logger v5.1

v5.1 deprecates several legacy appenders and one appender option, and changes when the file appender opens its log file.

The deprecated appenders and option all still work in v5.1, but each now emits a Ruby deprecation warning and will be removed in v6. The warnings use category: :deprecated, so they are silent by default and appear when deprecation warnings are enabled (for example ruby -W:deprecated, or Rails development mode). Switch away now to be ready for v6.

Replace the :sentry appender with :sentry_ruby

The :sentry appender depends on the end-of-life sentry-raven gem. Use the :sentry_ruby appender, backed by the maintained sentry-ruby gem.

In your Gemfile:

# Remove:
gem "sentry-raven"
# Add:
gem "sentry-ruby"

When adding the appender:

# Before:
SemanticLogger.add_appender(appender: :sentry)

# After:
SemanticLogger.add_appender(appender: :sentry_ruby)

Replace the :new_relic appender with :new_relic_logs

The :new_relic appender reports :error and :fatal entries to New Relic as error events via newrelic_rpm. It is superseded by :new_relic_logs, which forwards log entries to New Relic Logs:

# Before:
SemanticLogger.add_appender(appender: :new_relic)

# After:
SemanticLogger.add_appender(appender: :new_relic_logs)

Note that the behaviour differs. :new_relic created New Relic error events (and defaulted to the :error level), whereas :new_relic_logs forwards log messages to New Relic’s log ingestion. Set level: to match what you want forwarded, and update any New Relic dashboards or alerts that relied on the old error events. Both appenders require the newrelic_rpm gem.

Remove the type: option from the Elasticsearch and OpenSearch appenders

Document _type has been unused since Elasticsearch 7, so the type: option (on :elasticsearch, :elasticsearch_http, and :opensearch) is now ignored on all server versions: the bulk appenders no longer send a _type, and the :elasticsearch_http appender posts to the typeless _doc endpoint. Passing type: logs a deprecation warning. Remove it:

# Before:
SemanticLogger.add_appender(appender: :elasticsearch, url: "http://localhost:9200", type: "log")

# After:
SemanticLogger.add_appender(appender: :elasticsearch, url: "http://localhost:9200")

The file appender opens its log file when it is created

The file appender now opens its log file at creation rather than on the first write, matching the other appenders. A bad path or insufficient permissions therefore raises immediately from SemanticLogger.add_appender, instead of surfacing later on the appender thread while log messages are lost. This restores the pre-4.9 behavior.

As a result the log file is created as soon as the appender is added, even if nothing has been logged yet, the same as the standard Ruby and Rails loggers. If you add appenders inside a rescue or a conditional that assumed creation could never fail, review it: the failure now happens earlier and in the calling thread.

Rails: log file failures are caught at boot

If you use rails_semantic_logger, the companion gem’s v5.1 release pairs with the eager open above: a log file that cannot be opened is now caught during boot and the app falls back to standard error at the :warn level, rather than booting against a silently broken appender. v5.1 also drops support for Sidekiq v4, v5, and v6. See Upgrading to v5.1 for details.

Upgrading to Semantic Logger v5.0

Minimum Ruby version is now 3.2

Semantic Logger v5 requires Ruby 3.2 or later. Earlier Ruby versions are end-of-life and are no longer supported or tested.

If you are on an older Ruby, upgrade Ruby before upgrading Semantic Logger, or stay on the v4.x series.

SemanticLogger::Appender::AsyncBatch has been removed

The internal asynchronous proxy classes have been consolidated. Batch processing now runs through the same SemanticLogger::Appender::Async proxy (backed by an internal QueueProcessor), so the separate SemanticLogger::Appender::AsyncBatch class no longer exists.

This only affects code that referenced the class directly, which is uncommon since appenders are added through SemanticLogger.add_appender. The batch:, batch_size:, and batch_seconds: options are unchanged:

SemanticLogger.add_appender(appender: :http, url: "https://example.com/log", batch: true)

What changed:

If you have a custom appender or test asserting on the proxy class, change instance_of?/is_a?(SemanticLogger::Appender::AsyncBatch) checks to SemanticLogger::Appender::Async and, if needed, check appender.batch?.

Appenders are reopened automatically after fork

Previously, applications that fork (Puma, Unicorn, Spring, Resque, Process.daemon, etc.) had to call SemanticLogger.reopen themselves in an after-fork hook, otherwise the child shared the parent’s file handles and background thread.

In v5 this happens automatically: a Process._fork / Process.daemon hook calls SemanticLogger.reopen in the child process. The call is guarded to run once per process.

For most applications you can now remove the manual reopen hook, for example:

# No longer needed in v5 — handled automatically:
before_fork { SemanticLogger.flush }
after_fork  { SemanticLogger.reopen }

If you have a reason to manage this yourself (for example you reopen at a very specific point in your boot sequence), disable the automatic behaviour:

SemanticLogger.reopen_on_fork = false

By default SemanticLogger[SomeClass] returns a new Logger instance on every call. This means that if one part of the code obtains a logger and later changes its level (or filter), other holders of “the same” logger do not see the change, because they hold different instances.

v5 adds opt-in caching so that a Class or Module maps to a single shared Logger instance:

SemanticLogger.cache_loggers = true

With caching enabled:

Enabling caching is recommended for most applications. It is opt-in (default false) to preserve existing behaviour for code that relied on getting an independent logger per call. If you need to discard cached loggers (for example in tests, or after redefining a class), call SemanticLogger.clear_logger_cache.

Control characters are escaped in Syslog output

To prevent log-injection via embedded control characters (newlines, escape sequences, etc.), the Syslog formatter now escapes control characters in records by default. If you relied on raw control characters reaching syslog, this output now differs. The text formatters (default, color) also gain an opt-in escape_control_chars option (default false) for the same protection. See the Security page for details.

Formatters::Base#cleanse renamed

The internal SemanticLogger::Formatters::Base#cleanse method was renamed to #escape_control_characters. This only affects custom formatters or subclasses that called the old method name directly; update them to call #escape_control_characters.

Rails: appenders are now configured in one block

If you use rails_semantic_logger, the companion gem’s v5 release moves appender configuration into a single config.rails_semantic_logger.appenders do |appenders| ... end block. The v4 options (format, add_file_appender, ap_options, filter, console_logger) still work but are deprecated and will be removed in v6. See Migrating from v4 to v5 for the full before/after mapping.

Upgrading to Semantic Logger v4.18

Async queue is now bounded by default

The default max_queue_size for async appenders changed from -1 (unbounded) to 10_000. Under sustained high log volume, messages that exceed the queue limit will be dropped rather than causing memory to grow without bound.

If your application logs at very high throughput and you were relying on the unbounded default, set the limit explicitly when adding each appender:

SemanticLogger.add_appender(io: $stdout, async: true, max_queue_size: -1)

Alternatively, tune the value to match your application’s tolerance for log loss vs. memory usage.

Invalid exception values are now wrapped

Previously, passing a non-Exception object to the exception: keyword argument would silently misbehave. It is now wrapped in an ArgumentError with a description of the invalid value and a backtrace pointing to the call site:

# This previously caused undefined behaviour — now raises a descriptive ArgumentError
logger.error("Something went wrong", exception: "a plain string")

New Relic formatter — trace and span metadata

Previously, New Relic trace metadata (trace.id, span.id, entity.name, etc.) was captured at format time, so it worked with any appender type. In v4.18, this metadata is captured via an on_log callback that is only registered when SemanticLogger::Appender::NewRelicLogs is instantiated.

If you use a plain IO or File appender with SemanticLogger::Formatters::NewRelicLogs as the formatter (rather than the dedicated NewRelicLogs appender), you must register the callback manually in your initializer, otherwise trace.id, span.id, and entity.name will be absent from all logs:

require "semantic_logger/appender/new_relic_logs"
SemanticLogger.on_log(SemanticLogger::Appender::NewRelicLogs::CAPTURE_CONTEXT)

This line should be added before any log messages are emitted.


Upgrading to Semantic Logger v4.17

New Relic formatter (breaking changes for New Relic users)

The New Relic log formatter (SemanticLogger::Formatters::NewRelicLogs) has been significantly updated to produce a structure that New Relic can natively parse and index as individual attributes. Most users will benefit from richer log correlation in New Relic, but there are several breaking changes to be aware of.

Log output structure

Previously, all semantic fields (payload, tags, named_tags, metric, etc.) were serialized as a JSON string inside the message key. They are now emitted as top-level keys:

Before (v4.16):

{
  "message": "{\"message\":\"Order processed\",\"payload\":{\"order_id\":123},\"tags\":[],\"named_tags\":{}}",
  "timestamp": 1234567890000,
  "log.level": "INFO"
}

After (v4.17):

{
  "message": "Order processed",
  "payload": { "order_id": 123 },
  "timestamp": 1234567890000,
  "logger": { "name": "OrderService" },
  "thread": { "name": "main" }
}

Impact: Any New Relic alert queries, dashboards, or NRQL that references fields inside the old message JSON string (e.g. message.payload.order_id) must be updated to reference the new top-level keys (e.g. payload.order_id).

Note that New Relic’s log ingestion flattens nested objects. A payload value that is itself a nested Hash may not be fully indexed as individually searchable attributes. If you need payload fields to be queryable in New Relic, promote them to named_tags instead:

# Payload fields may not be individually queryable in New Relic:
logger.info "Order processed", order_id: 123, amount: 49.99

# Use named_tags for fields you want to query directly in New Relic:
SemanticLogger.tagged(order_id: 123, amount: 49.99) do
  logger.info "Order processed"
end
named_tags are now merged to the top level

Previously named_tags were nested inside the message JSON string. They are now merged directly into the top-level log entry, making them first-class attributes in New Relic.

If a named tag key conflicts with an existing top-level key (e.g. message, timestamp), it is dropped and the conflicting key names are recorded under named_tag_conflicts.

Error, logger, and thread fields restructured

Before (v4.16):

{
  "error.message": "Something went wrong",
  "error.class":   "RuntimeError",
  "error.stack":   "...",
  "log.level":     "ERROR",
  "logger.name":   "OrderService",
  "thread.name":   "main"
}

After (v4.17):

{
  "error":  { "message": "Something went wrong", "class": "RuntimeError", "stack": "..." },
  "logger": { "name": "OrderService" },
  "thread": { "name": "main" }
}

Update any New Relic alert conditions or NRQL queries that reference the old dot-notation keys.

Suppressing timestamps in formatters

v4.17 added :notime as an explicit time_format value to suppress timestamp output. If you were previously passing time_format: nil expecting it to suppress output, use :notime instead:

SemanticLogger.add_appender(io: $stdout, formatter: MyFormatter.new(time_format: :notime))

Upgrading to Semantic Logger v4.9

These changes should not be noticeable by the majority of users of Semantic Logger, since they are to the internal API. It is possible that advanced users may be using these internal API’s directly.

This does not affect any calls to the public api SemanticLogger.add_appender.

File and IO are now separate appenders. When creating the File appender explicitly, its arguments have changed. For example, when requesting an IO stream, it needs to be changed from:

SemanticLogger::Appender::File.new(io: $stderr)

to:

SemanticLogger::Appender::IO.new($stderr)

Additionally, this needs to be changed from:

SemanticLogger::Appender::File.new(file_name: "file.log")

to:

SemanticLogger::Appender::File.new("file.log")

Rails Semantic Logger, if used, needs to be upgraded to v4.9 when upgrading to Semantic Logger v4.9.


Upgrading to Semantic Logger v4.4

With some forking frameworks it is necessary to call reopen after the fork. With v4.4 the workaround for Ruby 2.5 crashes is no longer needed. I.e. Please remove the following line if being called anywhere:

SemanticLogger::Processor.instance.instance_variable_set(:@queue, Queue.new)

Upgrading to Semantic Logger v4.0

The following changes need to be made when upgrading to V4:

As a result any calls like the following:

logger.debug foo: 'foo', bar: 'bar'

Must be replaced with the following in v4:

logger.debug payload: {foo: 'foo', bar: 'bar'}

Similarly, for measure blocks:

logger.measure_info('How long is the sleep', foo: 'foo', bar: 'bar') { sleep 1 }

Must be replaced with the following in v4:

logger.measure_info('How long is the sleep', payload: {foo: 'foo', bar: 'bar'}) { sleep 1 }

The common log call has not changed, and the payload is still logged directly:

logger.debug('log this', foo: 'foo', bar: 'bar')