[LINUX] Aggregated logs transferred between containers and from other servers with Docker's rsyslog


After setting up various containers with docker, I wanted to aggregate the logs, so I made it with rsyslog.

Target device and environment

Verification environment

How to store logs

Logs that fly remotely are sorted into individual files using the syslog format % hostname%. It is saved as a log file for each host name under / var / log / remotelog /.

|-- dhcp.log
|-- esxi01.prosper2.net.log
|-- FWdc01.prosper2.net.log
|-- proxy.log
|-- radius.log
`-- ....

Depending on the syslog format setting on the sending side, the FQDN or just the host name may be slightly different. I should do it properly, but I haven't figured out how to do it yet. .. ..

The logs that flew from other Docker containers that belonged to ʻinfraserv-networkcreated in another entry seem to match the% hostname%` field properly, and they are separated by host name.

Work content

Set masquerade in firewall

firewall-cmd --add-masquerade --zone=public --permanent
firewall-cmd --reload

Docker image preparation

Create an image so that it can be reused. Create an appropriate directory to put the Dockerfile and the configuration file.

mkdir -p /opt/docker/syslog
cd /opt/docker/syslog

Create a Dockerfile


FROM centos:centos8
ENV TZ='Asia/Tokyo'
RUN ln -sf /usr/share/zoneinfo/Asia/Tokyo /etc/localtime ; \
    dnf -y update ; dnf install -y rsyslog cronie logrotate ; \
    mkdir /var/log/remotelog
COPY rsyslog.conf /etc/rsyslog.conf
COPY remotelog /etc/logrotate.d
CMD [ "/usr/sbin/init" ]

Accept remote logs

Since it is via the bridge, the container is only accepted from the host (


$template logfile_hostname,      "/var/log/remotelog/%hostname%.log"
$template logformat_verbose,     "%timegenerated%,%timereported%,%hostname%,%syslogtag%,%syslogseverity%,%syslogfacility%,%msg%\n"

$ModLoad imudp
$UDPServerRun 514
:fromhost-ip, ereregex, "" -?logfile_hostname;logformat_verbose
& stop

logrotate settings

Logs are compressed and stored for 40 generations.


/var/log/remotelog/*.log {
        rotate 40
          /bin/systemctl restart rsyslog

Build an image from a Dockerfile to create a container

Once you have created the Dockerfile, build it. --privileges seems to have a bad reputation (?), So create a container using the recommended method.

docker build --force-rm -t infraserv:syslog . -f ./syslog.df && \
docker run --cap-add sys_admin --security-opt seccomp:unconfined  -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  --network infraserv-network -it -d --name syslog --hostname syslog  -p 514:514/udp infraserv:syslog
docker exec -it syslog /bin/bash

Connection test

After starting the container, wait for a while and then check the log directory.

$ sudo docker exec -it syslog /bin/bash
[syslog]# ls -1sh /var/log/remotelog/
total 18M
 12K dhcp.log
704K esxi01.prosper2.net.log
 16M FWdc01.prosper2.net.log
4.0K _gateway.log
576K proxy.log
4.0K radius.log
256K VMinfraserv05.log

The contents of the file

# tail -n 1 *
==> dhcp.log <==
Feb 11 21:37:20,Feb 11 21:37:20,dhcp,pseudolog_kea_dhcp4_log,5,16, 2020-02-11 21:37:15.253 INFO  [kea-dhcp4.dhcpsrv/26] DHCPSRV_MEMFILE_LFC_EXECUTE executing Lease File Cleanup using: /usr/sbin/kea-lfc -4 -x /var/lib/kea/dhcp4.leases.2 -i /var/lib/kea/dhcp4.leases.1 -o /var/lib/kea/dhcp4.leases.output -f /var/lib/kea/dhcp4.leases.completed -p /var/lib/kea/dhcp4.leases.pid -c ignored-path

==> esxi01.prosper2.net.log <==
Feb 11 21:56:05,Feb 11 12:55:58,esxi01.prosper2.net,snmpd:,6,3, send_env_notifications: sent 0 of 0 SEL entries as notifications, 0 already sent

==> FWdc01.prosper2.net.log <==
Feb 11 21:56:27,Feb 11 21:56:27,FWdc01.prosper2.net,1,2020/02/11,6,1, 21:56:27,001606007877,TRAFFIC,end,2049,2020/02/11 21:56:27,,,,,any_to_DNS,,,dns,vsys1,zTrust,zTrust,ethernet1/2.11,ethernet1/2.252,LFP_default,2020/02/11 21:56:27,3327,1,54264,53,0,0,0x19,udp,allow,290,78,212,2,2020/02/11 21:55:55,0,any,0,6764778,0x0,,,0,1,1,aged-out,0,0,0,0,,FWdc01,from-policy,,,0,,0,,N/A,0,0,0,0

==> _gateway.log <==
Feb 11 21:06:05,Feb 11 21:06:05,_gateway,VMwlc01:,3,16, *Dot1x_NW_MsgTask_1: Feb 11 21:06:05.056: %DOT1X-3-INVALID_REPLAY_CTR: 1x_eapkey.c:449 Invalid replay counter from client ac:63:be:92:9d:f9 - got 00 00 00 00 00 00 00 1d, expected 00 00 00 00 00 00 00 1e

==> proxy.log <==
Feb 11 21:55:57,Feb 11 21:55:57,proxy,pseudolog_squid_access_log,5,16, 1581425757.416 284915 TCP_TUNNEL/200 2563 CONNECT www.google-analytics.com:443 - HIER_DIRECT/ -

==> radius.log <==
Feb 11 20:52:12,Feb 11 20:52:12,radius,pseudolog_radius_log,5,16, Tue Feb 11 20:52:12 2020 : Auth: (3) Login OK: [admin](from client radius_clients port 34)

==> VMinfraserv05.log <==
Feb 11 21:55:11,Feb 11 21:55:11,VMinfraserv05,pseudolog_cacti_log,5,16, 2020/02/11 21:55:08 - SNMPAGENT WARNING: No notification receivers configured for event: cactiNotifyDeviceFailedPoll (CACTI-MIB), severity: medium

You can get it.

at the end

Devices that do not add the hostname field during syslog transfer will have the host name _gateway. Cisco equipment. Even if you do logging origin-id hostname in Cisco IOS, it seems that the host name is only described in the message itself, so it is a bit difficult to use. ↓ It looks like this.

Feb 11 20:22:16,Feb 11 20:22:16,_gateway,844:,5,23, Rhq01: Feb 11 20:22:16: %IKEV2-5-SA_UP: SA UP


https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/7/html/system_administrators_guide/s1-basic_configuration_of_rsyslog https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/7/html/system_administrators_guide/s1-using_rsyslog_modules

Recommended Posts

Aggregated logs transferred between containers and from other servers with Docker's rsyslog
Control other programs from Python (communication between Python and exe)