[LINUX] Construction of Cyber Threat Information Aggregation System (EXIST) and Malware Information Sharing Platform (MISP) Summary ①

Introduction

――A former colleague told me about OSS that seems to be interesting, so I looked it up and built it. ――Here, it feels like summarizing the parts and procedures that were traversed when building. --When building with the ALL-in-One method, I made a mistake when adding DiskFull-> LVM-> VM crash, probably because of the acquired data size, so I decided to build it individually. --I plan to write an article in three parts (common part / EXIST / MISP).

Premise

--We are building using Windows10pro Hyper-V under the following conditions. (Common) --Please use VMware or vBOX as you like --The OS uses the latest version of CentOS 7 and is built with different VMs. (Common) --Since the behavior of CentOS 8 itself was unstable under the verification environment, we thought that it was necessary to investigate the cause separately and selected CentOS 7. --All accounts at the time of construction are implemented as root account. (Common) --VM resources are vCPU: 2core, mem: 8GB, DISK: 50GB, Network: IP address is fixed distribution by registering MAC address by DHCP. (Common) --Average memory usage is 6GB when running --DISK is 50GB because it exceeds 100% at 20GB. --Install Python ver3.6.x using pyenv. (EXIST) --Because an error occurs in the version of the specified Package if Verison 3.7 or later --We will build without using venv-exist --git: Ver is 2.29.x (latest at the moment). (Common) ―― 2.x can be anything --Install wget / curl / tmux / htop as appropriate. ――Unnecessary people can go through --Please disable Firewall and SElinux at your own risk. (Common) --Firewall later included opening 8000 ports in the procedure. --SElinux is dogeza and disabled

OSS product documentation

Cyber Threat Information Aggregation System (EXIST) Malware Information Sharing Platform(MISP)

Construction of common parts

--The OS status is installed by selecting "Minimum".

Kernel-ml related installation and read order change

# cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)
# 
# uname -a
Linux [hostname] 3.10.0 ~ omitted ~
#
#  rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
# 
# yum install https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm
Loaded plugin:fastestmirror
   |
   |
abridgement
   |
   |
warning:RPMDB has changed except for yum.
During installation: elrepo-release-7.0-5.el7.elrepo.noarch                                                      1/1 
During verification: elrepo-release-7.0-5.el7.elrepo.noarch                                                      1/1 

Installation:
  elrepo-release.noarch 0:7.0-5.el7.elrepo                                                                                  

Has completed!

# 
# yum --enablerepo=elrepo-kernel install kernel-ml
Loaded plugins: fastestmirror, langpacks
   |
   |
abridgement
   |
   |
Dependencies Resolved

======================================================================================================================== Package                  Arch                  Version                              Repository                    Size
========================================================================================================================Installing:
 kernel-ml                x86_64                5.9.8-1.el7.elrepo                   elrepo-kernel                 51 M

Transaction Summary
========================================================================================================================Install  1 Package

Total download size: 51 M
Installed size: 233 M
Is this ok [y/d/N]: y
Downloading packages:
kernel-ml-5.9.8-1.el7.elrepo.x86_64.rpm                                                          |  51 MB  00:00:09
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : kernel-ml-5.9.8-1.el7.elrepo.x86_64                                                                  1/1
  Verifying  : kernel-ml-5.9.8-1.el7.elrepo.x86_64                                                                  1/1

Installed:
  kernel-ml.x86_64 0:5.9.8-1.el7.elrepo

Complete!

Old and new kernels are mixed, so replace with new one

# 
# rpm -qa |grep kernel
kernel-ml-5.9.8-1.el7.elrepo.x86_64
kernel-tools-3.10.0-1062.el7.x86_64
kernel-tools-libs-3.10.0-1062.el7.x86_64
kernel-3.10.0-1062.el7.x86_64
# 
# yum swap --enablerepo=elrepo-kernel kernel-* -- kernel-ml-* <--Specified as an option because the repository is not enabled
Loaded plugins: fastestmirror, langpacks
Skipping the running kernel: kernel-3.10.0-1127.19.1.el7.x86_64
Loading mirror speeds from cached hostfile
   |
   |
abridgement
   |
   |
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/7): kernel-ml-devel-5.9.8-1.el7.elrepo.x86_64.rpm                                          |  13 MB  00:00:04
(2/7): kernel-ml-doc-5.9.8-1.el7.elrepo.noarch.rpm                                            | 8.9 MB  00:00:01
(3/7): kernel-ml-headers-5.9.8-1.el7.elrepo.x86_64.rpm                                        | 1.4 MB  00:00:00
(4/7): kernel-ml-tools-5.9.8-1.el7.elrepo.x86_64.rpm                                          | 229 kB  00:00:00
(5/7): kernel-ml-tools-libs-5.9.8-1.el7.elrepo.x86_64.rpm                                     | 117 kB  00:00:00
(6/7): kernel-ml-tools-libs-devel-5.9.8-1.el7.elrepo.x86_64.rpm                               |  95 kB  00:00:00
(7/7): kernel-ml-5.9.8-1.el7.elrepo.x86_64.rpm                                                |  51 MB  00:00:09
---------------------------------------------------------------------------------------------------------------------
Total                                                                                7.6 MB/s |  76 MB  00:00:09
   |
   |
abridgement
   |
   |
Complete!

# 
# rpm -qa |grep kernel
kernel-ml-tools-libs-5.9.8-1.el7.elrepo.x86_64
kernel-ml-doc-5.9.8-1.el7.elrepo.noarch
kernel-ml-tools-libs-devel-5.9.8-1.el7.elrepo.x86_64
kernel-ml-5.9.8-1.el7.elrepo.x86_64
kernel-ml-tools-5.9.8-1.el7.elrepo.x86_64
kernel-ml-devel-5.9.8-1.el7.elrepo.x86_64
kernel-ml-headers-5.9.8-1.el7.elrepo.x86_64
kernel-3.10.0-1062.el7.x86_64

Remove old kernel
#
# rpm -e kernel-3.10.0-1062.el7.x86_64

Change the Kernel loading order at OS startup <-Basic through is OK if the old kernel is deleted

#
# awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
0 : CentOS Linux (3.10.0-1127.19.1.el7.x86_64) 7 (Core)
1 : CentOS Linux (5.9.8-1.el7.elrepo.x86_64) 7 (Core) <--Select this guy
2 : CentOS Linux 7 Rescue 58dbc0c56878451db0f22c3af5129299 (3.10.0-693.17.1.el7.x86_64)
3 : CentOS Linux (0-rescue-55a201f65e044fb291c90c84936d9385) 7 (Core)
4 : CentOS Linux (0-rescue-e139e4d1f5ec417fb1ae634270a99f8b) 7 (Core

#
# grub2-editenv list
saved_entry=0
# 
# grub2-set-default 1
# 
# grub2-editenv list
saved_entry=1

Reboot to confirm reflection
#
# reboot

Update each package & install required packages individually

Exclude settings to prevent reinstallation of old kernel.
#
# echo "exclude=kernel*" >> /etc/yum.conf
# 
# yum update --enablerepo=elrepo-kernel
   |
   |
abridgement
   |
   |
Complete!

#
# yum install epel-release
   |
   |
abridgement
   |
   |
Complete!

#
# yum install tmux htop wget curl gcc autoconf curl-devel expat-devel gettext-devel openssl-devel perl-devel zlib-devel
   | 
   |
abridgement
   |
   |
Complete!

Git update

--Please install Git from the reference site below. --The packages required for installation are supported by the above yum.

that's all

This is the end of the common part. Next is the construction of EXIST.

Reference site / Many thanks

Kernel related elrepo.org kernel-ml Enable elrepo repository on CentOS7 and put kernel-ml Change / specify the default kernel on CentOS 7 YUM Cheat Sheet

Git related git/INSTALL Install the latest version of git on CentOS from source

Recommended Posts

Summary of construction of cyber threat information aggregation system (EXIST) and Malware Information Sharing Platform (MISP) ②
Construction of Cyber Threat Information Aggregation System (EXIST) and Malware Information Sharing Platform (MISP) Summary ①
Installation of Python3 and Flask [Environment construction summary]