[LINUX] Summary of construction of cyber threat information aggregation system (EXIST) and Malware Information Sharing Platform (MISP) ②
Introduction
――A former colleague told me about OSS that seems to be interesting, so I checked it and built it. ――Here, it feels like summarizing the parts and procedures that were traversed when building. --When building with the ALL-in-One method, I made a mistake in adding DiskFull-> LVM-> VM crash, probably because of the acquired data size, so I decided to build it individually. --I plan to write an article in three parts (common part / EXIST / MISP).
1. Premise
--We are building using Windows10pro Hyper-V under the following conditions. (Common) --Please use VMware or vBOX as you like --The OS uses the latest version of CentOS 7 and is built with different VMs. (Common) --Since the behavior of CentOS 8 itself was unstable under the verification environment, we thought that it was necessary to investigate the cause separately and selected CentOS 7. --All accounts at the time of construction are implemented as root account. (Common) --VM resources are vCPU: 2core, mem: 8GB, DISK: 50GB, Network: IP address is fixed distribution by registering MAC address by DHCP. (Common) --Average memory usage is 6GB when running --DISK is 50GB because it exceeds 100% at 20GB. --Install Python ver3.6.x using pyenv. (EXIST) --Because an error occurs in the version of the specified Package if Verison 3.7 or later --We will build it without using venv, but only install it. --git: Ver is 2.29.x (latest at the moment). (Common) ―― 2.x can be anything --Install wget / curl / tmux / htop as appropriate. ――Unnecessary people can go through --Please disable Firewall and SElinux at your own risk. (Common) --Firewall later included opening 8000 ports in the procedure. --SElinux is dogeza and disabled
2. OSS product documentation
Cyber Threat Information Aggregation System (EXIST) Malware Information Sharing Platform(MISP)
3. Construction contents
- pyenv --Cyber Threat Information Aggregation System (EXIST)
3-1. Installation of pyenv
Considering the purpose of this time, I don't think it is necessary to do it with pyenv, but I will do it here in consideration of future updates.
# git clone https://github.com/pyenv/pyenv.git
Cloning into 'pyenv'...
remote: Enumerating objects: 18376, done.
remote: Total 18376 (delta 0), reused 0 (delta 0), pack-reused 18376
Receiving objects: 100% (18376/18376), 3.67 MiB | 2.80 MiB/s, done.
Resolving deltas: 100% (12514/12514), done.
#May not be needed this time
# git clone https://github.com/pyenv/pyenv-virtualenv.git pyenv/plugins/pyenv-virtualenv
Cloning into 'pyenv/plugins/pyenv-virtualenv'...
remote: Enumerating objects: 2064, done.
remote: Total 2064 (delta 0), reused 0 (delta 0), pack-reused 2064
Receiving objects: 100% (2064/2064), 580.34 KiB | 753.00 KiB/s, done.
Resolving deltas: 100% (1413/1413), done.
#pyenv PATH setting
# vim ~/.bash_profile
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:/bin:$PATH"
eval "$(pyenv init -)"
#Reflect the above settings in the logged-in account
# source ~/.bash_profile
#Install the specified version
# pyenv install 3.6.12
Downloading Python-3.6.12.tar.xz...
-> https://www.python.org/ftp/python/3.6.12/Python-3.6.12.tar.xz
Installing Python-3.6.12...
WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?
WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib?
Installed Python-3.6.12 to /root/.pyenv/versions/3.6.12
#Reflect any version on the system
# pyenv global 3.6.12
#Confirmation of reflection
# python --version
#pip command update(Otherwise you will get an error later)
# pip install --upgrade pip
pip 18.1 from /root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/pip (python 3.6)
[root@exist opt]# pip install --upgrade pip
Collecting pip
Downloading https://files.pythonhosted.org/packages/cb/28/91f26bd088ce8e22169032100d4260614fc3da435025ff389ef1d396a433/pip-20.2.4-py2.py3-none-any.whl (1.5MB)
100% |################################| 1.5MB 7.5MB/s
Installing collected packages: pip
Found existing installation: pip 18.1
Uninstalling pip-18.1:
Successfully uninstalled pip-18.1
Successfully installed pip-20.2.4
Digression: If you want to do it other than pyenv, use the following method (currently ver3.8.6 seems to be installed)
# yum install python3 python3-libs python3-devel python3-pip
=====================================================================================================================
Package Arch Version Repository Size
=====================================================================================================================
Installing:
python3 x86_64 3.6.8-17.el7 base 70 k
python3-devel x86_64 3.6.8-17.el7 base 217 k
python3-libs x86_64 3.6.8-17.el7 base 6.9 M
python3-pip noarch 9.0.3-8.el7 base 1.6 M
Installing for dependencies:
dwz x86_64 0.11-3.el7 base 99 k
libtirpc x86_64 0.2.4-0.16.el7 base 89 k
perl-srpm-macros noarch 1-8.el7 base 4.6 k
python-rpm-macros noarch 3-34.el7 base 9.1 k
python-srpm-macros noarch 3-34.el7 base 8.8 k
python3-rpm-generators noarch 6-2.el7 base 20 k
python3-rpm-macros noarch 3-34.el7 base 8.1 k
python3-setuptools noarch 39.2.0-10.el7 base 629 k
redhat-rpm-config noarch 9.1.0-88.el7.centos base 81 k
zip x86_64 3.0-11.el7 base 260 k
Transaction Summary
=====================================================================================================================
Install 4 Packages (+10 Dependent packages)
3-2. Installation of EXIST
--The installation destination is assumed to be "/ opt /" and will be installed with a script.
# cd ~/
# git clone https://github.com/r4sd/exist_auto_install.git
Cloning into 'exist_auto_install'...
remote: Enumerating objects: 24, done.
remote: Counting objects: 100% (24/24), done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 24 (delta 6), reused 20 (delta 5), pack-reused 0
Receiving objects: 100% (24/24), 6.83 KiB | 3.42 MiB/s, done.
Resolving deltas: 100% (6/6), done.
#
# cd exist_auto_install/
# . exist_install.sh
[info] Repository file successfully written to /etc/yum.repos.d/mariadb.repo
[info] Adding trusted package signing keys...
[info] Successfully added trusted package signing keys
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Including mirror: ftp.tsukuba.wide.ad.jp
Including mirror: ftp.jaist.ac.jp
Including mirror: ftp.iij.ad.jp
Including mirror: ftp.yz.yamagata-u.ac.jp
Including mirror: ftp.nara.wide.ad.jp
* base: ftp.tsukuba.wide.ad.jp
Including mirror: ftp.ne.jp
Including mirror: ftp.yz.yamagata-u.ac.jp
* elrepo: ftp.ne.jp
Including mirror: ftp.yz.yamagata-u.ac.jp
* epel: ftp.yz.yamagata-u.ac.jp
Including mirror: ftp.tsukuba.wide.ad.jp
Including mirror: ftp.jaist.ac.jp
Including mirror: ftp.iij.ad.jp
|
|
abridgement
|
|
Nothing to do
------------------------------------------------
Please execute [ systemctl start exist.service ]
Admin (root) DB Password: [random number]
User (exist) DB Password: [random number]
#
# systemctl start exist.service
4. Access the site
--Access http: // [server IP address: 8000] from your browser.
5. In the end
――Since manual installation is complicated (although I'm used to it), I prepared a script, but I spent more than a day creating troubles in detail. ――Thanks to you, the simplification of the code and the understanding of the cooperation part have deepened, and the result is OK (* ´ω ｀ *) ――Next is the construction of MISP, but before that, add API registration etc. at once or UP with 2.5.
6. Reference site / Many thanks
nict-csl/exist Building a cyber threat information aggregation system EXIST I installed NICT EXIST I tried to build SOC in-house ③ #EXIST construction MISP vodkappa/misp-install-centos-7 EXPECT MariaDB 10.4.1 ~ user authentication is a chaotic story Mins/mysql_secure.sh
Recommended Posts