This article is the 15th day article of NTT Communications Advent Calendar 2019. Yesterday was @ Mahito's article, Story of proposing Chaos Engineering to a nursery school.

Introduction

As mentioned in the recently published NTT Communications Developer Blog Article, the NTT Communications Group We hold a security contest "ComCTF" for group employees.

I asked the question "Pentest" that was asked in the final. Pentest is an abbreviation for security test called penetration test, which verifies whether an attacker with a clear intention can achieve its purpose. [^ 1]

The problem is that it breaks into a virtual corporate network, exploits vulnerabilities on multiple servers, and finally asks if information can be obtained from the server where important data is stored. It was a problem of having a penetration test to verify whether the purpose of obtaining important data was achievable.

This time, to lay the foundation for this problem, we used an OSS called ** Naumachia ** that allows us to build a training environment for penetration testing using Docker.

https://github.com/nategraf/Naumachia

In this article, we will introduce an overview of Naumachia, how to build it, and how to build a training environment for penetration testing using this foundation.

What is Naumachia

Naumachia is an OSS that allows you to build closed networks and vulnerable servers using Docker.

I got to know this OSS through a CTF called TAMUctf 19 hosted by Texas A & M University. This Naumachia is used as the basis for problems in the genre NetworkPentest.

This CTF issue is published on GitHub, so if you are interested, please take a look.

https://github.com/tamuctf/TAMUctf-2019

The following functions are implemented in Naumachia.

Create and manage training Docker containers and networks for each user
Separate training environment from other users' environments
Training environment using OpenVPN Providing L2 level connection to the network

This allows you to build a dedicated training environment that can only be accessed by users with VPN connection information from the Internet, such as:

For example, if you try to create a problem that attempts to break into a system using Drupal's Arbitrary Code Execution Vulnerability (CVE-2018-7600), if you try to create a problem server that can be accessed from the Internet, it is a vulnerability on the Internet. In the worst case, the server can be used as a stepping stone by being caught in a sex scan. With Naumachia, only users with VPN connection information can challenge the problem from the Internet, so you can ask questions without such risk.

It also provides access at the L2 level, so you can create problems such as ARP spoofing that try out attack methods that take place within the same LAN.

The detailed function and mechanism are described in Naumachia README, so you should read this.

Building Naumachia

From here, I will introduce the procedure for building Naumachia.

Operating environment

In the README

Obtain a Linux server (tested on Ubuntu 16.04 and 18.04)

Since it says, ** Ubuntu 18.04 ** is the best OS to use.

However, since I used CentOS 7 for various reasons in this contest, I will write the construction procedure verified with CentOS 7. The OS information that verified the construction procedure is as follows.

# uname -a
Linux localhost.localdomain 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)

Preparing to install Naumachia

To build Naumachia, you need docker, docker-compose, Python3, pip3, so you need to install them first. After that, clone the source code from the Naumachia repository on GitHub and install the Python3 library described in requirements.txt.

install docker

# yum install -y yum-utils device-mapper-persistent-data lvm2
# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# yum install -y docker-ce docker-ce-cli containerd.io
# systemctl start docker
# systemctl enable docker

Install docker-compose

# curl -L https://github.com/docker/compose/releases/download/1.22.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
# chmod +x /usr/local/bin/docker-compose

Installation of Python3.6, pip3

# yum install -y https://centos7.iuscommunity.org/ius-release.rpm
# yum install python36u python36u-libs python36u-devel python36u-pip

Clone the source code from GitHub

# git clone https://github.com/nategraf/Naumachia.git

Install Python3 library

# cat requirements.txt
jinja2==2.10.1
PyYAML==4.2b4
requests==2.21.0
nose2==0.8.0
pytest==4.5.0
hypothesis==4.23.5
# pip3 install -r requirements.txt

Naumachia setup

Preparing Docker container and network for training

The training Docker container and network that Naumachia provides to users is defined in docker-compose.yml. After starting Naumachia, when a user connects with OpenVPN, docker-compose is automatically executed based on this docker-compose.yml and a training environment is built.

In this explanation, I will move the challenge ʻexample` from Naumachia problem collection (nategraf / Naumachia-challenges).

Clone the GitHub repository of the exam questions and copy the files needed for the ʻexample challenge to the challenges` directory inside the Naumachia directory.

# git clone https://github.com/nategraf/Naumachia-challenges
# mkdir Naumachia/challenges
# cp -r Naumachia-challenges/example Naumachia/challenges

By the way, the docker-compose.yml of ʻexampleis as follows. You can see that two containers,bob and ʻalice, and one network, default, are created.

`docker-compose.yml`


version: '2.4'

# The file defines the configuration for simple Nauachia challenge where a
# sucessful man-in-the-middle (MTIM) attack (such as ARP poisoning) provides a
# solution

# If you are unfamiliar with docker-compose this might be helpful:
# * https://docs.docker.com/compose/
# * https://docs.docker.com/compose/compose-file/
#
# But the gist is that the services block below specifies two containers, which
# act as parties in a vulnerable communication

services:
    bob:
        build: ./bob
        image: naumachia/example.bob
        environment:
            - CTF_FLAG=fOOBaR
        restart: unless-stopped
        networks:
            default:
                ipv4_address: 172.30.0.2

    alice:
        build: ./alice
        image: naumachia/example.alice
        depends_on:
            - bob
        environment:
            - CTF_FLAG=fOOBaR
        restart: unless-stopped
        networks:
            default:
                ipv4_address: 172.30.0.3

networks:
    default:
        driver: l2bridge
        ipam:
            driver: static
            config:
                - subnet: 172.30.0.0/28

Install and launch a customized Docker libnetwork Driver

If you look at the driver specified in networks in docker-compose.yml in the example challenge, you'll see that the unusual drivers l2bridge and static are specified.

The Naumachia challenge above uses a customized Docker libnetowrk driver to provide the same environment for all users and build a secure training environment.

https://github.com/nategraf/l2bridge-driver https://github.com/nategraf/static-ipam-driver

With it, you can do the following things that the default Docker libnetowrk driver can't do:

Allow duplicate IP subnets
Prohibition of access to the Internet from the container network

If it is Ubuntu or Debian, how to install Driver as a service is introduced here, but this time it was CentOS, so I created the following script and forcibly moved the Driver program. (I couldn't afford to rewrite sysv.sh for RedHat operating systems ...

`driver_start.sh`


# Download the static-ipam driver to usr/local/bin
if [ ! -e /usr/local/bin/l2bridge ]; then
  echo "[!] l2bridge driver is not installed"
  echo "[+] Download the l2bridge driver to usr/local/bin" 
  curl -L https://github.com/nategraf/l2bridge-driver/releases/latest/download/l2bridge-driver.linux.amd64 -o /usr/local/bin/l2bridge
  chmod +x /usr/local/bin/l2bridge
else
  echo "[*] l2bridge driver is installed"
fi

# Download the static-ipam driver to usr/local/bin
if [ ! -e /usr/local/bin/static-ipam ]; then
  echo "[!] static-ipam driver is not installed"
  echo "[+] Download the static-ipam driver to usr/local/bin" 
  curl -L https://github.com/nategraf/static-ipam-driver/releases/latest/download/static-ipam-driver.linux.amd64 -o /usr/local/bin/static-ipam
  chmod +x /usr/local/bin/static-ipam
else
  echo "[*] static-ipam driver is installed"
fi

# Activate the service
echo "[+] Startup the servicies" 

if [ ! -e /run/docker/plugins/l2bridge.sock ]; then
  nohup /usr/local/bin/l2bridge > /dev/null 2>&1 &
  echo "[*] Done: l2bridge" 
else
  echo "[!] Started l2bridge driver"
fi

if [ ! -e /run/docker/plugins/static.sock ]; then
  nohup /usr/local/bin/static-ipam > /dev/null 2>&1 &
  echo "[*] Done: static-ipam" 
else
  echo "[!] Started static-ipam driver"
fi

sleep 0.5

# Verify that it is running
echo "[+] Verify that it is running"

echo ""
echo "[*] stat /run/docker/plugins/l2bridge.sock"
stat /run/docker/plugins/l2bridge.sock
#  File: /run/docker/plugins/l2bridge.sock
#  Size: 0               Blocks: 0          IO Block: 4096   socket
#  ...

echo ""
echo "[*] stat /run/docker/plugins/static.sock"
stat /run/docker/plugins/static.sock
#  File: /run/docker/plugins/static.sock
#  Size: 0               Blocks: 0          IO Block: 4096   socket
#  ...

echo ""
echo "[*] Complete!!"

If you shut down, the Driver program will stop and will not start up when you restart, so you need to execute this when you restart.

Disable filtering of packets passing through the bridge

It seems that packets passing through the bridge may not work well if they are filtered, so run disable-bridge-nf-iptables.sh.

`disable-bridge-nf-iptables.sh`


echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 0 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

Modify config.yml

Copy config.example.yml to config.yml and rewrite part of it. The part to be rewritten is the challenges. The points to be changed are as follows.

In files:, write the location of the docker-compose.yml file created in "Preparing the Docker container and network for training".
Write the server address (domain, IP address) in commonname:

# [required] Configurations for each challenge
challenges:
    # [required] An indiviual challenge config. The key is the challenge name
    # This should be a valid unix filename and preferably short
    example:
        # [default: 1194] The exposed external port for this challenges OpenVPN server
        port: 2000
        # [default: [{challenge name}/docker-compose.yml] ] The compose files to which define this challenge
        # Paths should be relative to the challenges directory
        files:
            - example/docker-compose.yml
        # [default: {challenge name}.{domain}] The commonname used for the OpenVPN's certificates
        # This should be the domain name or ip that directs to this challenge
        commonname: 192.168.91.130
        # [default: None] If set, the OpenVPN management interface will be opened on localhost and the given port
        openvpn_management_port: null
        # [default: None] If set, the OpenVPN server will inform the client what IPv4 address and mask to apply to their tap0 interface
        ifconfig_push: 172.30.0.14/28

Build Naumachia

Running configure.py will build Naumachia based on what is written in config.yml. This will automatically generate Naumachia's docker-compose.yml and OpenVPN key, certificate and configuration files.

# ./configure.py
[INFO] Using config from /root/Naumachia/config.yml
[INFO] Using easyrsa installation at /root/Naumachia/tools/EasyRSA-v3.0.6/easyrsa
[INFO] Rendered /root/Naumachia/docker-compose.yml from /root/Naumachia/templates/docker-compose.yml.j2 
[INFO] Configuring 'example'
[INFO] Created new openvpn config directory /root/Naumachia/openvpn/config/example
[INFO] Initializing public key infrastructure (PKI)
[INFO] Building certificiate authority (CA)
[INFO] Generating Diffie-Hellman (DH) parameters
[INFO] Building server certificiate
[INFO] Generating certificate revocation list (CRL)
[INFO] Rendered /root/Naumachia/openvpn/config/example/ovpn_env.sh from /root/Naumachia/templates/ovpn_env.sh.j2 
[INFO] Rendered /root/Naumachia/openvpn/config/example/openvpn.conf from /root/Naumachia/templates/openvpn.conf.j2

Also, build a container for competition.

# docker-compose -f ./challenges/example/docker-compose.yml build

Execution of the competition environment

After doing the work so far, docker-compose.yml should be generated automatically, so build and up.

# docker-compose build
# docker-compose up -d

If you look at the container launched with docker ps -a in this state, the following container should be launched.

# docker ps -a
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS                      PORTS                    NAMES
dd9e858277bd        naumachia/manager     "python -m app"          27 seconds ago      Up 25 seconds                                        build_manager_1
f80057d9dc2e        naumachia/openvpn     "/scripts/naumachia-…"   27 seconds ago      Up 25 seconds               0.0.0.0:2000->1194/udp   build_openvpn-example_1
86fc3709d4e3        redis:alpine          "docker-entrypoint.s…"   27 seconds ago      Up 26 seconds                                        build_redis_1
a0f45e1f292a        naumachia/registrar   "gunicorn -c python:…"   27 seconds ago      Up 26 seconds               0.0.0.0:3960->3960/tcp   build_registrar_1
9d1ef7902351        alpine                "/bin/true"              27 seconds ago      Exited (0) 27 seconds ago                            build_bootstrapper_1

Generation of OpenVPN configuration file to be distributed to users

A configuration file is required for users to connect to the OpenVPN server and access the training environment. This is also automatically generated by Naumachia.

There are two ways to generate it.

Use registrar CLI Python scripts
Use the registrar server REST API --The container waiting at 3960 / tcp is so --Be careful when publishing to the outside because there is no authentication

This time, we will use the registrar CLI Python script to create and get the configuration file. If you execute registrar-cli as follows, you can create an OpenVPN configuration file that includes the OpenVPN key, server certificate, and certificate authority certificate, and distribute it to the users.

# ./registrar-cli example add user1
# ./registrar-cli example get user1 > user1.ovpn
# cat user1.ovpn

client
nobind
dev tap
remote-cert-tls server
float
explicit-exit-notify

remote 192.168.91.130 2000 udp



<key>
-----BEGIN PRIVATE KEY-----
(abridgement)
-----END PRIVATE KEY-----
</key>
<cert>
-----BEGIN CERTIFICATE-----
(abridgement)
-----END CERTIFICATE-----
</cert>
<ca>
-----BEGIN CERTIFICATE-----
(abridgement)
-----END CERTIFICATE-----
</ca>
key-direction 1

cipher AES-256-CBC
auth SHA256
comp-lzo

Play with the built training environment

Now, let's access the training environment we built and play with it.

Environment used for verification

This time, the user side uses Kali Linux, which has the OpenVPN client and the tools for penetration testing installed by default.

# grep VERSION /etc/os-release 
VERSION="2018.1"
VERSION_ID="2018.1"

Connect to training environment with OpenVPN

Use the generated OpenVPN configuration file to access the training environment on Naumachia. ʻInitialization Sequence Completed` is a success!

# openvpn user1.ovpn
Sun Dec 15 06:33:45 2019 OpenVPN 2.4.5 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Mar  4 2018
Sun Dec 15 06:33:45 2019 library versions: OpenSSL 1.1.0h  27 Mar 2018, LZO 2.08
Sun Dec 15 06:33:45 2019 TCP/UDP: Preserving recently used remote address: [AF_INET]192.168.91.130:2000
Sun Dec 15 06:33:45 2019 UDP link local: (not bound)
Sun Dec 15 06:33:45 2019 UDP link remote: [AF_INET]192.168.91.130:2000
Sun Dec 15 06:33:45 2019 [192.168.91.130] Peer Connection Initiated with [AF_INET]192.168.91.130:2000
Sun Dec 15 06:33:46 2019 Options error: Unrecognized option or missing or extra parameter(s) in [PUSH-OPTIONS]:1: dhcp-renew (2.4.5)
Sun Dec 15 06:33:46 2019 TUN/TAP device tap0 opened
Sun Dec 15 06:33:46 2019 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Sun Dec 15 06:33:46 2019 /sbin/ip link set dev tap0 up mtu 1500
Sun Dec 15 06:33:46 2019 /sbin/ip addr add dev tap0 172.30.0.14/28 broadcast 172.30.0.15
Sun Dec 15 06:33:46 2019 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Sun Dec 15 06:33:46 2019 Initialization Sequence Completed

Looking at the state of the interface with ifconfig, I think that the interface tap0 is created and the IP address 172.30.0.14 is assigned.

# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.91.129  netmask 255.255.255.0  broadcast 192.168.91.255
        inet6 fe80::20c:29ff:fe18:a0c8  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:18:a0:c8  txqueuelen 1000  (Ethernet)
        RX packets 14781  bytes 9483880 (9.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6484  bytes 645921 (630.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 31612  bytes 10003030 (9.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 31612  bytes 10003030 (9.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.30.0.14  netmask 255.255.255.240  broadcast 172.30.0.15
        inet6 fe80::c0d8:eeff:fe38:d79b  prefixlen 64  scopeid 0x20<link>
        ether c2:d8:ee:38:d7:9b  txqueuelen 100  (Ethernet)
        RX packets 16  bytes 1272 (1.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 21  bytes 1622 (1.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

At this time, if you look at the status of the Docker container and network on the Naumachia server, you should see that a new container and network with the prefix ʻuser1_example_` has been created.

This is a user-specific training container and network. As the number of users increases, so does the number of containers and networks.

# docker ps -a
CONTAINER ID        IMAGE                     COMMAND                  CREATED              STATUS                      PORTS                    NAMES
17c4ef2ccbb9        naumachia/example.alice   "python /app/alice.py"   About a minute ago   Up About a minute                                    user1_example_alice_1
ff271a01eba9        naumachia/example.bob     "python /app/bob.py"     About a minute ago   Up About a minute                                    user1_example_bob_1
dd9e858277bd        naumachia/manager         "python -m app"          32 minutes ago       Up 32 minutes                                        build_manager_1
f80057d9dc2e        naumachia/openvpn         "/scripts/naumachia-…"   32 minutes ago       Up 32 minutes               0.0.0.0:2000->1194/udp   build_openvpn-example_1
86fc3709d4e3        redis:alpine              "docker-entrypoint.s…"   32 minutes ago       Up 32 minutes                                        build_redis_1
a0f45e1f292a        naumachia/registrar       "gunicorn -c python:…"   32 minutes ago       Up 32 minutes               0.0.0.0:3960->3960/tcp   build_registrar_1
9d1ef7902351        alpine                    "/bin/true"              32 minutes ago       Exited (0) 32 minutes ago                            build_bootstrapper_1

# docker network ls
NETWORK ID          NAME                     DRIVER              SCOPE
743f747a01b3        bridge                   bridge              local
7017ddd37ba8        build_default            bridge              local
dce5de7a2fa2        build_internal           bridge              local
de7c1746cc32        host                     host                local
6dc0c89a9ccf        none                     null                local
b1649b2f2e93        user1_example_default    l2bridge            local

Try ARP spoofing

This issue is a MITM (man-in-the-middle attack) issue, such as ARP spoofing, as described in docker-compose.yml in the example.

The file defines the configuration for simple Nauachia challenge where a sucessful man-in-the-middle (MTIM) attack (such as ARP poisoning) provides a solution

This time, we have two terminals with IP addresses of 172.30.0.2 and 172.30.0.3, so we will try to ARP spoof the communication that these two terminals are doing and eavesdrop.

The mechanism and specific method of ARP spoofing will not be explained in detail here, but if successful, packet capture will be performed as follows to communicate between the two hosts, 172.30.0.2 and 172.30.0.. Can be seen.

# tcpdump -i tap0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap0, link-type EN10MB (Ethernet), capture size 262144 bytes
06:40:47.791591 ARP, Reply 172.30.0.2 is-at 3e:d6:f2:ca:92:81 (oui Unknown), length 28
06:40:48.042999 ARP, Reply 172.30.0.3 is-at 3e:d6:f2:ca:92:81 (oui Unknown), length 28
06:40:48.696193 IP 172.30.0.3.55672 > 172.30.0.2.5005: UDP, length 30
06:40:49.792320 ARP, Reply 172.30.0.2 is-at 3e:d6:f2:ca:92:81 (oui Unknown), length 28
06:40:50.044301 ARP, Reply 172.30.0.3 is-at 3e:d6:f2:ca:92:81 (oui Unknown), length 28
06:40:51.700769 IP 172.30.0.3.55672 > 172.30.0.2.5005: UDP, length 30
06:40:51.793616 ARP, Reply 172.30.0.2 is-at 3e:d6:f2:ca:92:81 (oui Unknown), length 28
06:40:52.044971 ARP, Reply 172.30.0.3 is-at 3e:d6:f2:ca:92:81 (oui Unknown), length 28
06:40:53.794367 ARP, Reply 172.30.0.2 is-at 3e:d6:f2:ca:92:81 (oui Unknown), length 28
06:40:54.045958 ARP, Reply 172.30.0.3 is-at 3e:d6:f2:ca:92:81 (oui Unknown), length 28
06:40:54.705584 IP 172.30.0.3.55672 > 172.30.0.2.5005: UDP, length 30
06:40:55.795642 ARP, Reply 172.30.0.2 is-at 3e:d6:f2:ca:92:81 (oui Unknown), length 28
06:40:56.047136 ARP, Reply 172.30.0.3 is-at 3e:d6:f2:ca:92:81 (oui Unknown), length 28

You should be aware of these risks if you are on the same network connected by L2.

in conclusion

In this article, I introduced how to build a training environment for penetration testing using OSS called Naumachia.

Since the penetration test is actually an attack, does training it want to train an attacker? You might think, but it's not.

Cyber attacks have become more sophisticated in recent years, and it is becoming more and more difficult to protect them from the perspective of the defender alone. In order to protect against such attacks, it is important to know the actual attack method and devise an effective defense method that matches it. By moving my hands in training and trying out actual attack methods, I think that understanding of attack methods will improve and we will be able to develop human resources who can defend more efficiently.

Tomorrow I will be in charge of @nyakuo.

Have a nice year!

[^ 1]: About Penetration Test by Vulnerability Diagnostician Skill Map Project (https://github.com/ueno1000/about_PenetrationTest)

[LINUX] Building a training environment for penetration testing using Naumachia