Splunk logging driver

Thesplunklogging driver sends container logs to HTTP Event Collector in Splunk Enterprise and Splunk Cloud.

Usage

You can configure Docker logging to use thesplunkdriver by default or on a per-container basis.

To use thesplunkdriver as the default logging driver, set the keys log-driverandlog-optsto appropriate values in thedaemon.json configuration file and restart Docker. For example:

{
"log-driver":"splunk",
"log-opts":{
"splunk-token":"",
"splunk-url":"",
...
}
}

The daemon.json file is located in/etc/docker/on Linux hosts or C:\ProgramData\docker\config\daemon.jsonon Windows Server. For more about configuring Docker usingdaemon.json,see daemon.json.

Note

log-optsconfiguration options in thedaemon.jsonconfiguration file must be provided as strings. Boolean and numeric values (such as the value for splunk-gziporsplunk-gzip-level) must therefore be enclosed in quotes (").

To use thesplunkdriver for a specific container, use the commandline flags --log-driverandlog-optwithdocker run:

$docker run --log-driver=splunk --log-opt splunk-token=VALUE --log-opt splunk-url=VALUE...

Splunk options

The following properties let you configure the Splunk logging driver.

  • To configure thesplunkdriver across the Docker environment, edit daemon.jsonwith the key,"log-opts": { "NAME": "VALUE",...}.
  • To configure thesplunkdriver for an individual container, usedocker run with the flag,--log-opt NAME=VALUE....
OptionRequiredDescription
splunk-tokenrequiredSplunk HTTP Event Collector token.
splunk-urlrequiredPath to your Splunk Enterprise, self-service Splunk Cloud instance, or Splunk Cloud managed cluster (including port and scheme used by HTTP Event Collector) in one of the following formats:https://your_splunk_instance:8088,https://input-prd-p-XXXXXXX.cloud.splunk:8088,orhttps://http-inputs-XXXXXXXX.splunkcloud.
splunk-sourceoptionalEvent source.
splunk-sourcetypeoptionalEvent source type.
splunk-indexoptionalEvent index.
splunk-capathoptionalPath to root certificate.
splunk-canameoptionalName to use for validating server certificate; by default the hostname of thesplunk-urlis used.
splunk-insecureskipverifyoptionalIgnore server certificate validation.
splunk-formatoptionalMessage format. Can beinline,jsonorraw.Defaults toinline.
splunk-verify-connectionoptionalVerify on start, that Docker can connect to Splunk server. Defaults to true.
splunk-gzipoptionalEnable/disable gzip compression to send events to Splunk Enterprise or Splunk Cloud instance. Defaults to false.
splunk-gzip-leveloptionalSet compression level for gzip. Valid values are -1 (default), 0 (no compression), 1 (best speed)... 9 (best compression). Defaults to DefaultCompression.
tagoptionalSpecify tag for message, which interpret some markup. Default value is{{.ID}}(12 characters of the container ID). Refer to the log tag option documentationfor customizing the log tag format.
labelsoptionalComma-separated list of keys of labels, which should be included in message, if these labels are specified for container.
labels-regexoptionalSimilar to and compatible withlabels.A regular expression to match logging-related labels. Used for advanced log tag options.
envoptionalComma-separated list of keys of environment variables, which should be included in message, if these variables are specified for container.
env-regexoptionalSimilar to and compatible withenv.A regular expression to match logging-related environment variables. Used for advanced log tag options.

If there is collision between thelabelandenvkeys, the value of theenv takes precedence. Both options add additional fields to the attributes of a logging message.

Below is an example of the logging options specified for the Splunk Enterprise instance. The instance is installed locally on the same machine on which the Docker daemon is running.

The path to the root certificate and Common Name is specified using an HTTPS scheme. This is used for verification. TheSplunkServerDefaultCertis automatically generated by Splunk certificates.

$docker run\
--log-driver=splunk \
--log-opt splunk-token=176FCEBF-4CF5-4EDF-91BC-703796522D20 \
--log-opt splunk-url=https://splunkhost:8088 \
--log-opt splunk-capath=/path/to/cert/cacert.pem \
--log-opt splunk-caname=SplunkServerDefaultCert \
--log-opt tag= "{{.Name}}/{{.FullID}}" \
--log-opt labels=location \
--log-opt env=TEST \
--env "TEST=false" \
--label location=west \
your/application

Thesplunk-urlfor Splunk instances hosted on Splunk Cloud is in a format likehttps://http-inputs-XXXXXXXX.splunkcloudand does not include a port specifier.

Message formats

There are three logging driver messaging formats:inline(default),json, andraw.


The default format isinlinewhere each log message is embedded as a string. For example:

{
"attrs":{
"env1":"val1",
"label1":"label1"
},
"tag":"MyImage/MyContainer",
"source":"stdout",
"line":"my message"
}
{
"attrs":{
"env1":"val1",
"label1":"label1"
},
"tag":"MyImage/MyContainer",
"source":"stdout",
"line":"{\" foo\ ": \" bar\ "}"
}

To format messages asjsonobjects, set--log-opt splunk-format=json.The driver attempts to parse every line as a JSON object and send it as an embedded object. If it can't parse the message, it's sentinline.For example:

{
"attrs":{
"env1":"val1",
"label1":"label1"
},
"tag":"MyImage/MyContainer",
"source":"stdout",
"line":"my message"
}
{
"attrs":{
"env1":"val1",
"label1":"label1"
},
"tag":"MyImage/MyContainer",
"source":"stdout",
"line":{
"foo":"bar"
}
}

To format messages asraw,set--log-opt splunk-format=raw.Attributes (environment variables and labels) and tags are prefixed to the message. For example:

MyImage/MyContainer env1=val1 label1=label1 my message
MyImage/MyContainer env1=val1 label1=label1 { "foo": "bar" }

Advanced options

The Splunk logging driver lets you configure a few advanced options by setting environment variables for the Docker daemon.

Environment variable nameDefault valueDescription
SPLUNK_LOGGING_DRIVER_POST_MESSAGES_FREQUENCY5sThe time to wait for more messages to batch.
SPLUNK_LOGGING_DRIVER_POST_MESSAGES_BATCH_SIZE1000The number of messages that should accumulate before sending them in one batch.
SPLUNK_LOGGING_DRIVER_BUFFER_MAX10 * 1000The maximum number of messages held in buffer for retries.
SPLUNK_LOGGING_DRIVER_CHANNEL_SIZE4 * 1000The maximum number of pending messages that can be in the channel used to send messages to background logger worker, which batches them.