Introduction

Build the server with ACT / SBY

environment

	version	Remarks
CentOS	7.4
DRBD	9.0
cryptSetup	2.0.3
paceMaker	1.1.20-5.el7_7.2
corosync	2.4.3

Add disk

This time we are using KVM. It is assumed that you have instances of SERVER-1 and SERVER-2. Add disks.

Instance confirmation

[root@kvm ~]# virsh list
Id name status
----------------------------------------------------
 126   SERVER-1 running
 127   SERVER-2 running

Creating xml of definition of adding disk

`/tmp/server-1-adddisk.xml`


[root@kvm ~]# vi /tmp/server-1-adddisk.xml
<disk type='file' device='disk'>
  <driver name='qemu' type='qcow2' cache='none' />
  <source file='/var/lib/libvirt/images/SERVER-1-vdb.qcow2' />
  <target dev='vdb' bus='virtio' />
</disk>

`/tmp/server-2-adddisk.xml`


[root@kvm ~]# vi /tmp/server-2-adddisk.xml
<disk type='file' device='disk'>
  <driver name='qemu' type='qcow2' cache='none' />
  <source file='/var/lib/libvirt/images/SERVER-2-vdb.qcow2' />
  <target dev='vdb' bus='virtio' />
</disk>

Created in 16G on vdb for both

[root@kvm ~]# qemu-img create -f qcow2 /var/lib/libvirt/images/SERVER-1-vdb.qcow2 16G
[root@kvm ~]# qemu-img create -f qcow2 /var/lib/libvirt/images/SERVER-2-vdb.qcow2 16G

Guess the settings

[root@kvm ~]# virsh attach-device SERVER-1 --file /tmp/server-1-adddisk.xml
[root@kvm ~]# virsh attach-device SERVER-2 --file /tmp/server-2-adddisk.xml

Permanently reflect settings

[root@kvm ~]# virsh attach-device SERVER-1 --file /tmp/server-1-adddisk.xml --config
[root@kvm ~]# virsh attach-device SERVER-2 --file /tmp/server-2-adddisk.xml --config

SERVER setting

Decide on an IP, host name, and domain.

Unit	hostname	IP
Unit 1	SERVER-1	192.168.200.10
Unit 2	SERVER-2	192.168.200.11

Let's write each other's IP in hosts.

[root@server-1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.200.10 SERVER-1
192.168.200.11 SERVER-2

[root@server-2 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.200.10 SERVER-1
192.168.200.11 SERVER-2

Material installation

First, install the necessary items for both systems.

[root@server-1 ~]# rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
[root@server-1 ~]# yum --enablerepo=extras -y install kmod-drbd90
[root@server-1 ~]# yum -y install cryptsetup cryptsetup-libs
[root@server-1 ~]# yum -y install pacemaker
[root@server-1 ~]# yum -y install pcs fence-agents-all

[root@server-2 ~]# rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
[root@server-2 ~]# yum --enablerepo=extras -y install kmod-drbd90
[root@server-2 ~]# yum -y install cryptsetup cryptsetup-libs
[root@server-2 ~]# yum -y install pacemaker
[root@server-2 ~]# yum -y install pcs fence-agents-all

Extension disk initialization

Initialize with both

[root@server-1 ~]# parted /dev/vdb -s mklabel msdos -s mkpart primary 0% 100%

[root@server-2 ~]# parted /dev/vdb -s mklabel msdos -s mkpart primary 0% 100%

DRBD settings

Editing /etc/drbd.d/drbd0.res

`:/etc/drbd.d/drbd0.res`


[root@server-1 ~]# vi /etc/drbd.d/drbd0.res
resource drbd0 {
  protocol  C;
  volume 0 {
    device /dev/drbd0;
    disk /dev/vdb;
    meta-disk internal;
  }
  on SERVER-1 {	
    node-id 0;
    address 192.168.200.10:7789;
  }
  on SERVER-2 {	
    node-id   1;
    address   192.168.200.11:7789;
  }
}

Distribution to SERVER-2

[root@server-1 ~]# scp -p /etc/drbd.d/drbd0.res root@server-2:/etc/drbd.d/drbd0.res

Create and launch DRBD

I will create-md.

[root@server-1 ~]# drbdadm create-md drbd0
md_offset 17179865088
al_offset 17179832320
bm_offset 17179308032

Found some data

 ==> This might destroy existing data! <==

Do you want to proceed?
[need to type 'yes' to confirm] yes //Input location

initializing activity log
initializing bitmap (512 KB) to all zero
Writing meta data...
New drbd meta data block successfully created.

[root@server-2 ~]# drbdadm create-md drbd0
md_offset 17179865088
al_offset 17179832320
bm_offset 17179308032

Found some data

 ==> This might destroy existing data! <==

Do you want to proceed?
[need to type 'yes' to confirm] yes //Input location

initializing activity log
initializing bitmap (512 KB) to all zero
Writing meta data...
New drbd meta data block successfully created.

Check the state before startup with lsblk Unit 1

[root@server-1 ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0              11:0    1 1024M  0 rom
vda             252:0    0   16G  0 disk
├─vda1          252:1    0    1G  0 part /boot
└─vda2          252:2    0   15G  0 part
  ├─centos-root 253:0    0 13.4G  0 lvm  /
  └─centos-swap 253:1    0  1.6G  0 lvm  [SWAP]
vdb             252:16   0   16G  0 disk

Unit 2

[root@server-2 ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0              11:0    1 1024M  0 rom
vda             252:0    0   16G  0 disk
├─vda1          252:1    0    1G  0 part /boot
└─vda2          252:2    0   15G  0 part
  ├─centos-root 253:0    0 13.4G  0 lvm  /
  └─centos-swap 253:1    0  1.6G  0 lvm  [SWAP]
vdb             252:16   0   16G  0 disk

Start DRBD If you just start one, it will not end. It seems that we are meeting, so at the same timing.

[root@server-1 ~]# systemctl start drbd

[root@server-2 ~]# systemctl start drbd

Check after startup with lsblk Unit 1 drbd0 is increasing

[root@server-1 ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0              11:0    1 1024M  0 rom
vda             252:0    0   16G  0 disk
├─vda1          252:1    0    1G  0 part /boot
└─vda2          252:2    0   15G  0 part
  ├─centos-root 253:0    0 13.4G  0 lvm  /
  └─centos-swap 253:1    0  1.6G  0 lvm  [SWAP]
vdb             252:16   0   16G  0 disk
└─drbd0         147:0    0   16G  0 disk

Confirmed Unit 2

[root@server-2 ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0              11:0    1 1024M  0 rom
vda             252:0    0   16G  0 disk
├─vda1          252:1    0    1G  0 part /boot
└─vda2          252:2    0   15G  0 part
  ├─centos-root 253:0    0 13.4G  0 lvm  /
  └─centos-swap 253:1    0  1.6G  0 lvm  [SWAP]
vdb             252:16   0   16G  0 disk
└─drbd0         147:0    0   16G  0 disk

Make SERVER-1 the main system

[root@server-1 ~]# drbdadm primary --force drbd0

Make the file system ext4

[root@server-1 ~]# mkfs.ext4 /dev/drbd0

I will mount it here for confirmation. Create a directory to mount on both systems.

[root@server-1 ~]# mkdir /opt/mntpoint
[root@server-1 ~]# mount /dev/drbd0 /opt/mntpoint

[root@server-1 ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0              11:0    1 1024M  0 rom
vda             252:0    0   16G  0 disk
├─vda1          252:1    0    1G  0 part /boot
└─vda2          252:2    0   15G  0 part
  ├─centos-root 253:0    0 13.4G  0 lvm  /
  └─centos-swap 253:1    0  1.6G  0 lvm  [SWAP]
vdb             252:16   0   16G  0 disk
└─drbd0         147:0    0   16G  0 disk /opt/mntpoint

Try to create a file appropriately by checking if it is synchronized further

[root@server-1 ~]# touch /opt/mntpoint/hoge.txt

Unmount from Unit 1

[root@server-1 ~]# umount /opt/mntpoint

Mounted on Unit 2 and lsblk

[root@server-2 ~]# mkdir /opt/mntpoint
[root@server-2 ~]# mount /dev/drbd0 /opt/mntpoint
[root@server-2 ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0              11:0    1 1024M  0 rom
vda             252:0    0   16G  0 disk
├─vda1          252:1    0    1G  0 part /boot
└─vda2          252:2    0   15G  0 part
  ├─centos-root 253:0    0 13.4G  0 lvm  /
  └─centos-swap 253:1    0  1.6G  0 lvm  [SWAP]
vdb             252:16   0   16G  0 disk
└─drbd0         147:0    0   16G  0 disk /opt/mntpoint

It's synchronized properly. Let's delete hoge.txt.

[root@server-2 ~]# ll /opt/mntpoint/
-rw-r--r--1 root root 0 September 11 13:07 hoge.txt

Disk encryption

This time it may not make much sense because it is KVM, but there are cases where someone can feel relieved because it is encrypted.

First, if drbd is mounted, unmount it.

[root@server-2 ~]# umount /opt/mntpoint

format

[root@server-1 ~]# cryptsetup luksFormat -c aes-cbc-essiv:sha256 -s 256 /dev/drbd0
WARNING!
========
This will overwrite data on /dev/drbd0 irrevocably.

Are you sure? (Type uppercase yes): YES　//Only the uppercase YES was recognized
Enter passphrase for /dev/drbd0:
Verify passphrase: //【password input
encrypt success //【Re-enter password
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
1048576 inodes, 4193655 blocks
209682 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2151677952
128 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Create an encrypted directory with the name cryptdir

[root@server-1 ~]#echo [password]| cryptsetup luksOpen /dev/drbd0 cryptdir

Then

[root@server-1 ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sr0              11:0    1 1024M  0 rom
vda             252:0    0   16G  0 disk
├─vda1          252:1    0    1G  0 part  /boot
└─vda2          252:2    0   15G  0 part
  ├─centos-root 253:0    0 13.4G  0 lvm   /
  └─centos-swap 253:1    0  1.6G  0 lvm   [SWAP]
vdb             252:16   0   16G  0 disk
└─drbd0         147:0    0   16G  0 disk
  └─cryptdir    253:2    0   16G  0 crypt 　　　　　　　　//I can do something

Since the cryptdir mentioned earlier is created under / dev / mapper /, create a file system with ext4

[root@server-1 ~]# mkfs.ext4 /dev/mapper/cryptdir

Check if you want to synchronize with DRBD. confirm. /dev/mapper/cryptdir Use this as the mount point.

mount /dev/mapper/cryptdir /opt/mntpoint

It was mounted.

[root@server-1 ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sr0              11:0    1 1024M  0 rom
vda             252:0    0   16G  0 disk
├─vda1          252:1    0    1G  0 part  /boot
└─vda2          252:2    0   15G  0 part
  ├─centos-root 253:0    0 13.4G  0 lvm   /
  └─centos-swap 253:1    0  1.6G  0 lvm   [SWAP]
vdb             252:16   0   16G  0 disk
└─drbd0         147:0    0   16G  0 disk
  └─cryptdir    253:2    0   16G  0 crypt /opt/mntpoint

As before, try to create a file appropriately

[root@server-1 ~]# touch /opt/mntpoint/hoge.txt

Unmount from Unit 1

[root@server-1 ~]# umount /opt/mntpoint

Decrypt

[root@server-1 ~]# cryptsetup luksClose cryptdir

Mount after encrypting with Unit 2

[root@server-2 ~]#echo [password]| cryptsetup luksOpen /dev/drbd0 cryptdir
[root@server-2 ~]# mount /dev/mapper/cryptdir /opt/mntpoint

There is.

[root@server-2 ~]# ll /opt/mntpoint/
-rw-r--r--1 root root 0 September 11 13:51 hoge.txt

Unmount and decrypt from Unit 2

[root@server-1 ~]# umount /opt/mntpoint
[root@server-1 ~]# cryptsetup luksClose cryptdir

In order to let pacemaker manage resources later Create /usr/lib/systemd/system/encrypt.service as a systemd service.

`/usr/lib/systemd/system/encrypt.service`


[root@server-1 ~]# vi /usr/lib/systemd/system/encrypt.service
[Unit]
Description=encrypt
[Service]
ExecStart=/opt/bin/encryptstart.sh
ExecStop=/opt/bin/encryptstop.sh
Type=oneshot
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target

systemctl start script executed when encrypted

`/opt/bin/encryptstart.sh`


[root@server-1 ~]# mkdir /opt/bin
[root@server-1 ~]# vi /opt/bin/encryptstart.sh
#!/usr/bin/sh
/usr/bin/echo [password]| /usr/sbin/cryptsetup luksOpen /dev/drbd0 cryptdir

systemctl stop script executed when encrypted

`/opt/bin/encryptstop.sh`


[root@server-1 ~]# vi /opt/bin/encryptstop.sh
#!/usr/bin/sh
/usr/sbin/cryptsetup luksClose cryptdir

Let's send it to Unit 2 as well.

[root@server-1 ~]# scp -p /usr/lib/systemd/system/encrypt.service root@server-2:/usr/lib/systemd/system/encrypt.service
[root@server-1 ~]# scp -rp /opt/bin root@server-2:/opt/bin

systemd setting reflection Unit 1

[root@server-1 ~]# systemctl daemon-reload

Unit 2

[root@server-2 ~]# systemctl daemon-reload

pacemaker and corosync settings

First of all, stop DRBD.

[root@server-1 ~]# systemctl stop drbd

[root@server-2 ~]# systemctl stop drbd

Edit /etc/corosync/corosync.conf

`/etc/corosync/corosync.conf`


[root@server-1 ~]# vi /etc/corosync/corosync.conf
totem {
	version: 2
	cluster_name: XXX_cluster
	secauth: off
	transport: udpu
}

nodelist {
	node {
		ring0_addr: SERVER-1
		nodeid: 1
	}

	node {
		ring0_addr: SERVER-2
		nodeid: 2
	}
}

quorum {
	provider: corosync_votequorum
	two_node: 1
}

logging {
	to_logfile: yes
	logfile: /var/log/cluster/corosync.log
	to_syslog: yes
}

Transfer to Unit 2

[root@server-1 ~]# scp -p /etc/corosync/corosync.conf　192.168.200.11:/etc/corosync/corosync.conf

Editing / etc / sysconfig / pacemaker

`/etc/sysconfig/pacemaker`


[root@server-1 ~]# vi /etc/sysconfig/pacemaker
# PCMK_fail_fast=no
↓
PCMK_fail_fast=yes

Transfer to Unit 2

[root@server-1 ~]# scp -p /etc/sysconfig/pacemaker　192.168.200.11:/etc/sysconfig/pacemaker

I haven't changed the contents, but for the time being, copy the service to / etc / systemd / system /

[root@server-1 ~]# cp -p /usr/lib/systemd/system/corosync.service /etc/systemd/system/
[root@server-1 ~]# cp -p /usr/lib/systemd/system/pacemaker.service /etc/systemd/system/
[root@server-1 ~]# scp -p /usr/lib/systemd/system/corosync.service [email protected]:/usr/lib/systemd/system/corosync.service
[root@server-1 ~]# scp -p /usr/lib/systemd/system/pacemaker.service [email protected]:/usr/lib/systemd/system/pacemaker.service

Reflect systemd settings again Unit 1

[root@server-1 ~]# systemctl daemon-reload

Unit 2

[root@server-2 ~]# systemctl daemon-reload

Start pacemaker

[root@server-1 ~]# systemctl start pacemaker
[root@server-1 ~]# systemctl start pcsd
[root@server-1 ~]# systemctl enable pcsd

Also, give a password to a user named hacluster Unit 1

[root@server-1 ~]#echo [ha password]| passwd -f hacluster --stdin

Unit 2

[root@server-2 ~]#echo [ha password]| passwd -f hacluster --stdin

Authenticate using hacluster user

[root@server-1 ~]# pcs cluster auth SERVER-1 SERVER-2 -u hacluster -p [ha password]

cluster settings

[root@server-1 ~]# pcs cluster setup --name HA_cluster SERVER-1 SERVER-2 --force

Start the cluster

[root@server-1 ~]# pcs cluster start --all
[root@server-1 ~]# pcs cluster enable --all

Resource registration

DRBD registration

[root@server-1 ~]# pcs resource create DRBD ocf:linbit:drbd drbd_resource=drbd0 op monitor interval=10s role=Master monitor interval=30s role=Slave
[root@server-1 ~]# pcs resource master MS_DRBD DRBD master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true

Disk encryption

[root@server-1 ~]# pcs resource create ENCRYPT systemd:encrypt --group HA

File system mount

[root@server-1 ~]# pcs resource create FS ocf:heartbeat:Filesystem device=/dev/mapper/criptdir directory=/opt/mntpoint fstype=ext4 --group HA

Something various

[root@server-1 ~]# pcs constraint colocation add HA MS_DRBD INFINITY with-rsc-role=Master
[root@server-1 ~]# pcs constraint order promote MS_DRBD then start HA

Clean resources

[root@server-1 ~]# pcs resource cleanup

I made this kind of setting, but I forgot what it was ...

[root@server-1 ~]# pcs property set stonith-enabled=false
[root@server-1 ~]# pcs property set no-quorum-policy=ignore

Also VIP is added to the resource

[root@server-1 ~]# pcs resource create VIP ocf:heartbeat:IPaddr2 ip=192.168.200.12 cidr_netmask=24 --group HA

Verification

[root@server-1 ~]# pcs status
Cluster name: HA_cluster
Stack: corosync
Current DC: SERVER-2 (version 1.1.20-5.el7_7.2-3c4c782f70) - partition with quorum
Last updated: Sun Sep 13 06:57:08 2020
Last change: Thu Sep 10 14:31:58 2020 by root via cibadmin on SERVER-1

2 nodes configured
7 resources configured

Online: [ SERVER-1 SERVER-2 ]

Full list of resources:

 Master/Slave Set: MS_DRBD [DRBD]
     Masters: [ SERVER-1 ]
     Slaves: [ SERVER-2 ]
 Resource Group: HA
     ENCRYPT (systemd:encrypt):              Started SERVER-1
     FS      (ocf::heartbeat:Filesystem):    Started SERVER-1
     VIP     (ocf::heartbeat:IPaddr2):       Started SERVER-1

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

[LINUX] DRBD + CryptSetup + PaceMaker on CentOS7 ACT / SBY

Introduction

environment

Add disk

/tmp/server-1-adddisk.xml

/tmp/server-2-adddisk.xml

SERVER setting

Material installation

Extension disk initialization

DRBD settings

:/etc/drbd.d/drbd0.res

Create and launch DRBD

Disk encryption

/usr/lib/systemd/system/encrypt.service

/opt/bin/encryptstart.sh

/opt/bin/encryptstop.sh

pacemaker and corosync settings

/etc/corosync/corosync.conf

/etc/sysconfig/pacemaker

Resource registration

`/tmp/server-1-adddisk.xml`

`/tmp/server-2-adddisk.xml`

`:/etc/drbd.d/drbd0.res`

`/usr/lib/systemd/system/encrypt.service`

`/opt/bin/encryptstart.sh`

`/opt/bin/encryptstop.sh`

`/etc/corosync/corosync.conf`

`/etc/sysconfig/pacemaker`