Detailed explanation of the indicators of Prometheus crawling nginx access logs

To crawl log information in NGINX through Prometheus' NGINX Exporter, such as logs with status codes 4xx or 5xx, you need to combine the following tools and methods:

1. Basic functions of NGINX Exporter

NGINX Exporter is a Prometheus Exporter used to use it from NGINX/statusThe endpoint collects metric data, but it does not process the log files directly by itself. Therefore, additional steps are required to monitor log information with status codes 4xx or 5xx in NGINX.

2. Use logstash or fluentd to collect NGINX logs

In order to extract specific information from NGINX's log files and convert it into metrics that Prometheus can crawl, we can uselogstashorfluentdetc. log collection tool.

usefluentdExampleInstall Fluentd and Prometheus plug-insFirst, you need to install Fluentd and related plug-ins to ensure that Fluentd can extract data from the log and expose it in Prometheus format:

gem install fluentd
gem install fluent-plugin-prometheus

Configure FluentdConfigure Fluentd, parse NGINX logs and generate metrics in Prometheus format. Create a Fluentd configuration file, the content is as follows:

<source>
  @type tail
  path /var/log/nginx/
  pos_file /var/log/td-agent/
  tag 
  <parse>
    @type nginx
  </parse>
</source>
<filter >
  @type grep
  <regexp>
    key status
    pattern ^4|5[0-9][0-9]$
  </regexp>
</filter>
<match >
  @type prometheus
  <metric>
    name nginx_http_status_total
    type counter
    desc Nginx HTTP status code counter
    keys status
    <labels>
      status ${status}
    </labels>
  </metric>
  <metric>
    name nginx_http_request_bytes_total
    type counter
    desc Total bytes received by the Nginx server
    key bytes
  </metric>
</match>
<source>
  @type prometheus_monitor
  <labels>
    tag 
  </labels>
</source>
<source>
  @type prometheus_output_monitor
  <labels>
    tag 
  </labels>
</source>
<match **>
  @type stdout
</match>

- **Source**：Read `/var/log/nginx/` document，And parse the log。
- **Filter**：Filter the status code through regular expressions 4xx or 5xx Logs。
- **Match**：将解析后Logs信息转化为 Prometheus Format indicators。
- **Prometheus Output**：Start a HTTP Endpoint，for Prometheus Crawl these metrics。

Start FluentdRun Fluentd, apply the configuration file:

fluentd -c /path/to/

Fluentd will start a listening port, the default ishttp://localhost:24231/metrics, expose parsed log data in Prometheus format.

3. Configure Prometheus to crawl Fluentd exposed metrics

Prometheus configuration fileIn, add Fluentd as the new crawling target:

scrape_configs:
  - job_name: 'nginx_logs'
    static_configs:
      - targets: ['localhost:24231']

4. Use Grafana to visualize

Now that Prometheus can crawl NGINX's 4xx and 5xx status code log information, you can create a dashboard in Grafana to show the time trends of this information and analyze the health of NGINX services.

Summarize

Although NGINX Exporter itself cannot directly crawl log information, combined with log collection tools such as Fluentd, you can easily convert specific information in NGINX logs (such as 4xx and 5xx status codes) into metrics that Prometheus can crawl and eventually visualize in Grafana.

This is the article about Prometheus crawling nginx access log indicators. For more related Prometheus nginx indicator content, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!