How to customize Nginx JSON log format configuration

Preface

In modern Web service architecture, logging is an important part of monitoring, debugging and security auditing. As one of the most popular web servers, Nginx's flexible log configuration capabilities allow us to customize the log format according to our needs. This article will introduce in detail how to configure Nginx to record access logs in JSON format. This structured log format is particularly suitable for log analysis systems such as ELK.

Why choose JSON format log?

Compared with the traditional NCSA general log format, JSON format logs have the following advantages:

• Structured data: Easy to parse and process log analysis systems (such as ELK Stack)
• Field Extensibility: Fields can be added or deleted easily without affecting existing log parsing
• Definite data type: Data types such as numerical, string, etc. can be clearly distinguished
• Good compatibility: Compatible with most modern log processing tools and database systems

Detailed explanation of configuration steps

1. Install Nginx service

Install Nginx on a Debian/Ubuntu-based system:

apt -y install nginx

For RHEL/CentOS systems, please use:

yum -y install nginx

2. Customize JSON log format

Edit Nginx main configuration file (usually located in/etc/nginx/）：

http {
    ...
    log_format oldboyedu_nginx_json '{"@timestamp":"$time_iso8601",'
                                  '"host":"$server_addr",'
                                  '"clientip":"$remote_addr",'
                                  '"SendBytes":$body_bytes_sent,'
                                  '"responsetime":$request_time,'
                                  '"upstreamtime":"$upstream_response_time",'
                                  '"upstreamhost":"$upstream_addr",'
                                  '"http_host":"$host",'
                                  '"uri":"$uri",'
                                  '"domain":"$host",'
                                  '"xff":"$http_x_forwarded_for",'
                                  '"referer":"$http_referer",'
                                  '"tcp_xff":"$proxy_protocol_addr",'
                                  '"http_user_agent":"$http_user_agent",'
                                  '"status":"$status"}';
    access_log  /var/log/nginx/  oldboyedu_nginx_json;
    # Important: Comment or remove the default log format    # access_log /var/log/nginx/;
    ...
}

Description of each field

3. Configuration file syntax check

Execute the following command to check whether the Nginx configuration is correct:

nginx -t

If the configuration is correct, you should see an output like:

nginx: the configuration file /etc/nginx/ syntax is ok
nginx: configuration file /etc/nginx/ test is successful

4. Restart Nginx service

Apply new configuration:

systemctl restart nginx

5. Test verification

Logging can be tested using simple loop commands:

while true; do curl 10.0.0.91; sleep 0.5; done

Check the content of the log file:

tail -f /var/log/nginx/

You should see a JSON format log entry similar to the following:

{"@timestamp":"2023-05-15T14:30:45+08:00","host":"10.0.0.91","clientip":"10.0.0.1","SendBytes":612,"responsetime":0.002,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.91","uri":"/","domain":"10.0.0.91","xff":"-","referer":"-","tcp_xff":"","http_user_agent":"curl/7.68.0","status":"200"}

Production environment suggestions

Log rotation: Configure logrotate to prevent excessive log files

vim /etc//nginx

• Sensitive information filtering: Avoid recording sensitive information such as passwords, credit card numbers, etc.

• Performance considerations: In high traffic environments, JSON logs may increase disk I/O load

• Field optimization: Adjust the fields according to actual needs and record only necessary information

• Log segmentation: Consider splitting log files by virtual host or date

Extended configuration

Add more fields

You can extend the log format as needed, such as adding a request method:

log_format oldboyedu_nginx_json '{"@timestamp":"$time_iso8601",'
                              ...
                              '"method":"$request_method",'
                              ...
                              '"status":"$status"}';

Conditional logging

For unimportant requests such as static resources, logging can be skipped:

map $uri $is_static {
    default 0;
    ~*\.(jpg|jpeg|png|gif|ico|css|js|woff|woff2|ttf|svg) 1;
}
server {
    ...
    access_log /var/log/nginx/ oldboyedu_nginx_json if=$is_static;
    ...
}

Frequently Asked Questions

• No log file write：

  • Check the permissions of Nginx process users to log directory
  • Confirm that no other configurations overwrite the access_log directive

• JSON format error：

  • Make sure all string values ​​are surrounded by double quotes
  • Make sure there are no extra commas

• Performance degradation：

  • Consider reducing the number of log fields
  • Evaluation using buffered writes (access_log ... buffer=32k flush=5m）

in conclusion

By configuring Nginx to record access logs in JSON format, you can get more structured and easy to analyze log data. This format is especially suitable for integration with log analysis systems such as ELK Stack, Splunk, etc., providing better observability for your web services. Depending on your specific needs, you can flexibly adjust the log fields to balance information richness and system performance.

This is the article about the custom Nginx JSON log format configuration guide. For more information about Nginx JSON log format, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!