Create a fluentd server for testing

In the process of creating a system that uses fluentd, I was wondering how to test that data was sent to fluentd in the first place, so I investigated various things.

Thing you want to do

In the process of creating a system that uses fluentd, I want to write a quick test to confirm that data is being sent to fluentd.

policy

I want to shorten the test time as much as possible when performing CI, so I want to link the data thrown by FluentLogger to output in real time as much as possible.

I used to write the code to open the socket and wait by myself, but since the version went up and SSL became involved and it became troublesome, I would like to do something with plugin as much as possible.

Since I often use docker to set up surrounding middleware when turning CI, I tried putting Elasticsearch in the middleware kit once before, but I purposely built Elasticsearch that I did not call directly and elasticsearch-ruby for testing It's also lazy to put it in and connect it. .. .. Especially if it is an official image, it does not happen on a single node, and it is troublesome to follow and update with all microservices when the version changes, so I would like to test without additional middleware as much as possible.

Method study

Real-time

Fluentd buffering is always done while tuning the value of flush_interval, but for testing it seems that you can go with flush_mode = immediate

Usability in CI

This time I thought about whether I could make good use of Redis, which is also used in ActiveJob etc., but I found a good plugin in the Official plugin list, so I tried it. Try.

There was also MySQL, but the data structure sent from FluentLogger is rather different, so the redis plugin that puts it in as it is was more suitable.

I actually tried it

We will proceed with the necessary Gem installation and fluentd settings while considering the method.

0. Preparation

As for gem, it is enough if there is a plugin of fluent-plugin-redis-store in addition to the main body, so I will put it all together

command


gem install fluentd fluent-plugin-redis-store --no-document

I will omit the installation of Redis, but I will build redis using brew or docker.

Since fluentd is received from forward and sent to redis_store, fluent.conf looks like this.

fluent.conf


<source>
  @type forward
  @id input1
  @label @mainstream
  port 24224
</source>

<label @mainstream>
  <filter **>
    @type record_transformer
    <record>
      tag ${tag}
    </record>
  </filter>
  <match **>
    @type copy
    <store>
      @type stdout
    </store>
    <store>
      @type redis_store
      key_path tag
      <buffer>
        flush_mode immediate
      </buffer>
    </store>
  </match>
</label>

What I'm doing is like this.

It's a bonus to put it on stdout, but if you put it out, it's easy to debug later.

Write this to ./fluent.conf and start fluentd

fluentd -c fluent.conf

Now that I'm ready, I'd like to throw in the data, but it's easier to understand the movement if you monitor Redis, so open another window and monitor it.

redis-cli monitor

1. Throw data to fluentd

Where to send data to fluentd, the actual application sends data using fluent-logger-ruby, but this time fluent-cat Try using as a substitute

echo '{ "key" : "sample" }' | bundle exe fluent-cat debug.test

This is a command that sends the data with the tag debug.test and the time stamp of the current time to the data{"key": "sample"}. If not set, it will be addressed to localhost, so the data will be sent to the fluentd server built above.

Then, the following output will be output to the standard output on the fluentd side.

fluentd run window


2020-11-03 09:37:28.016591000 +0900 debug.test: {"key":"sample","tag":"debug.test"}

This is output by @type stdout set in fluent.conf, and it can be confirmed that the filter is tagged and has this form after actually receiving the data.

Furthermore, looking at the window that monitors Redis, it is displayed as follows.

redis-cli run window


1604363848.025602 [0 172.28.0.1:40998] "zadd" "debug.test" "1604363848.016591" "{\"key\":\"sample\",\"tag\":\"debug.test\"}"

It is zadded as described in the readme of the plugin. The setting of key_path tag is effective for key, and debug.test, which is the value of tag included in the data, is set.

2. Confirmation of thrown data

Try to get the data from Redis.

redis-cli zrange debug.test 0 -1 withscores
1) "{\"key\":\"sample\",\"tag\":\"debug.test\"}"
2) "1604363848.016591"

Since the time stamp is a score, it is sorted so that you can put in a lot.

By the way, it looks like this with ruby.

require 'redis'

Redis.new.zrange 'debug.test', 0, -1, withscores: true

output


 => [["{\"key\":\"sample\",\"tag\":\"debug.test\"}", 1604363848.016591]]

For the time being, the value can be taken simply, so it seems to be relatively easy to use.

bonus

Consider usability in parallel testing

When executing tests in parallel using parallel_tests or test-queue, it becomes necessary to determine which test process is the log. At that time, you can include the process ID in the data to be sent and use it to set the key.

I want to put the process ID, time, and request ID in the Rails log file, so I don't think it will cause any harm.

In that case, the data to be sent will be as follows

{ "key" : "sample", "pid" : 123 }

Along with this, change fluent.conf as follows.

  <label @mainstream>
    <filter **>
      @type record_transformer
      <record>
        tag ${tag}
+       tag_with_pid '${tag}.${record["pid"]}'
      </record>
    </filter>
    <match **>
      @type copy
      <store>
        @type stdout
      </store>
      <store>
        @type redis_store
-       key_path tag
+       key_path tag_with_pid
        <buffer>
          flush_mode immediate
        </buffer>
      </store>
    </match>
  </label>

Now that the process ID is entered in the key when it is registered in Redis, use that process ID to identify the log of your own process and evaluate it to determine whether data was actually sent. You will be able to do it.

Change Redis destination

It seems that you can change the setting in the following form as described in the README

  <label @mainstream>
    <filter **>
      @type record_transformer
      <record>
        tag ${tag}
      </record>
    </filter>
    <match **>
      @type copy
      <store>
        @type stdout
      </store>
      <store>
        @type redis_store
        key_path tag
+       host 10.0.0.1
+       db   11
        <buffer>
          flush_mode immediate
        </buffer>
      </store>
    </match>
  </label>

Helper when using with RSpec and Cucumber

When actually using it in a test, if you make a helper like this, it will be easy to use

# frozen_string_literal: true

require 'redis'

module FluentdLogHelper

  def fetch_fluentd_log_by(tag:, pid: nil)
    redis_key = pid ? "#{tag}.#{pid}" : tag
    redis.zrange redis_key, 0, -1
  end

  def redis(options = {})
    options[:db] ||= 0
    @redis ||= Redis.new(options)
  end

end

Simple name


fetch_fluentd_log_by tag: 'debug.test'

Calling with process ID


fetch_fluentd_log_by tag: 'debug.test', pid: Process.pid

Recommended Posts

Create a fluentd server for testing
Create a MySQL container for application testing in Ansible AWX
[Android] Create validation for date input!
Create your own encode for String.getBytes ()
Create a fluentd server for testing
How to create a Maven repository for 2020
How to create a database for H2 Database anywhere
How to create pagination for a "kaminari" array
[Java] Let's create a mod for Minecraft 1.16.1 [Introduction]
[Java] Let's create a mod for Minecraft 1.14.4 [99. Mod output]
Create a web api server with spring boot
Create a docker environment for Oracle 11g XE
Create a Docker container for your development web server in Ansible on MacOS
[Java] Let's create a mod for Minecraft 1.14.4 [0. Basic file]
[Java] Let's create a mod for Minecraft 1.14.4 [4. Add tools]
Build a docker container for a python simple web server
Create an HTTPS file server for development with ring-jetty-adapter
[Java] Create a filter
[Java] Let's create a mod for Minecraft 1.14.4 [5. Add armor]
[Java] Let's create a mod for Minecraft 1.14.4 [Extra edition]
[Java] Let's create a mod for Minecraft 1.14.4 [7. Add progress]
[Java] Let's create a mod for Minecraft 1.14.4 [6. Add recipe]
Create a widget template for iOS14 with Intent Configuration.
[Java] Let's create a mod for Minecraft 1.16.1 [Basic file]
[Java] Let's create a mod for Minecraft 1.14.4 [1. Add items]
Try Easy Ramdom, a PropertyBase Testing tool for java
[Java] Let's create a mod for Minecraft 1.14.4 [2. Add block]
I want to create a generic annotation for a type
[Java] Let's create a mod for Minecraft 1.16.1 [Add block]
Tutorial to create a blog with Rails for beginners Part 1
How to create a lightweight container image for Java apps
[Java twig] Create a parser combinator for recursive descent parsing 2
Finally, create a method for whether there is any character
Tutorial to create a blog with Rails for beginners Part 2
Create a Kibana container image for ARM64 (Raspberry Pi/Mac M1)
How to create and launch a Dockerfile for Payara Micro
Create a java method [Memo] [java11]
[Java] Create a temporary file
Create a VS Code Plugin.
Create a playground with Xcode 12
Create a fortune using Ruby
How to create a method
Create a name input function
Create a simple web server with the Java standard library com.sun.net.httpserver
I want to create a chat screen for the Swift chat app!
[Java] Let's create a mod for Minecraft 1.16.1 [Add and generate trees]
[Java] Let's create a mod for Minecraft 1.14.4 [9. Add and generate trees]
How to create a web server on an EC2 instance on AWS
Create a development environment for Ruby 3.0.0 and Rails 6.1.0 on Ubuntu 20.04.1 LTS
Create a Docker Image for redoc-cli and register it on Docker Hub
Creating a dual boot environment for Ubuntu Server 20.04.1 LTS and Windows 10
[Java] Let's create a mod for Minecraft 1.14.4 [8. Add and generate ore]