30 May 2020

Managing Kubernetes Ingress JSON Log Format with Sanity

The nginx ingress supports a custom log format. The option can be set via the log_format field in the ingress configmap. As you can see, the log format must be a single line. I wanted to implement the Elasticsearch Common Schema for our nginx ingress access log. Writing the ECS JSON for our access log results in:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


{
  "timestamp": "$time_iso8601",
  "network": {"forwarded_ip": "$http_x_forwarded_for"},
  "user":{"name":"$remote_user"},
  "user_agent":{"original":"$http_user_agent"},
  "http":{
    "version": "$server_protocol",
    "request":{
      "body":{"bytes":$body_bytes_sent},
      "bytes": $request_length,
      "method":"$request_method",
      "referrer":"$http_referer"
    },
    "response":{
      "body":{"bytes":$body_bytes_sent},
      "bytes": $bytes_sent,
      "status_code":"$status",
      "time":$request_time
    },
    "upstream": {
      "bytes": $upstream_response_length,
      "status_code":"$upstream_status",
      "time":$upstream_response_time,
      "address": "$upstream_addr",
      "name": "$proxy_upstream_name"
    }
  },
  "url":{
    "domain":"$host",
    "path":"$uri",
    "query":"$args",
    "original":"$request_uri"
  }
}

This 34 lines of JSON must be condensed to one line. Doing this manually is error prone. Moreover, maintaining this JSON struct in one line is nearly impossible. Since we manage the deployment of our ingress in a helm chart, I used the nospace sprig function. This function removes every whitespace from the input string. Be aware, this approach works only if your JSON struct contains no whitespace. The whitespaces from your JSON key and values will be stripped as well.

I saved the JSON struct to the file config/logpattern.json in the helm chart. My templates/configmap.yml looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress
data:
  log-format-escape-json: "true"
  log-format-upstream: '{{ .Files.Get "config/logpattern.json" | nospace }}'

The .Files.Get loads the JSON log pattern from the log pattern file and pipes that into the nospace function. This results in the following rendered configmap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress
data:
  log-format-escape-json: "true"
  log-format-upstream: '{"network":{"forwarded_ip":"$http_x_forwarded_for"},"user":{"name":"$remote_user"},"user_agent":{"original":"$http_user_agent"},"http":{"version":"$server_protocol","request":{"body":{"bytes":$body_bytes_sent},"bytes":$request_length,"method":"$request_method","referrer":"$http_referer"},"response":{"body":{"bytes":$body_bytes_sent},"bytes":$bytes_sent,"status_code":"$status","time":$request_time}},"url":{"domain":"$host","path":"$uri","query":"$args","original":"$request_uri","foo":"bar"}}'

Chart directory layout:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


.
├── Chart.yaml
├── config
│   └── logpattern.json
├── templates
│   ├── configmap.yml
│   ├── ....
....
│   └── ....
└── values.yaml