Linux RabbitMQ-server cluster

https://www.rabbitmq.com/clustering.html
https://kamaok.org.ua/?p=2689
https://blog.flux7.com/blogs/tutorials/how-to-creating-highly-available-message-queues-using-rabbitmq

1) add hostname (do real hostname) between master and slave to /etc/hosts

2) copy /var/lib/rabbitmq/.erlang.cookie from master to slave

systemctl restart rabbitmq-server (on slave)


3) join to cluster

rabbitmqctl stop_app

Stopping rabbit application on node rabbit@<hostname_slave> ...

rabbitmqctl reset

Resetting node rabbit@<hostname_slave> ...

rabbitmqctl join_cluster rabbit@<hostname_slave>

Clustering node rabbit@<hostname_slave> with rabbit@<hostname_master>

rabbitmqctl start_app

Starting node rabbit@<hostname_slave> ...
completed with 0 plugins.


4) check cluster

rabbitmqctl cluster_status

Cluster status of node rabbit@<hostname_slave> ...

[{nodes,[{disc,[rabbit@<hostname_master>,rabbit@<hostname_slave>]}]},

{running_nodes,[rabbit@<hostname_master>,rabbit@<hostname_slave>]},

{cluster_name,<<"rabbit@localhost">>},
{partitions,[]},

{alarms,[{rabbit@<hostname_master>,[]},{rabbit@<hostname_slave>,[]}]}]

5) Configure syncing queues police between master and slave nodes

Run on master

rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

for all vhost

https://stackoverflow.com/questions/38611583/rabbitmq-is-there-a-way-to-apply-policy-on-all-virtual-host-in-cluster-environ

from
https://www.rabbitmq.com/ha.html

ha-sync-mode: manual
- this is the default mode. A new queue mirror will not receive existing messages, it will only receive new messages. The new queue mirror will become an exact replica of the master over time, once consumers have drained messages that only exist on the master. If the master queue fails before all unsychronised messages are drained, those messages will be lost. You can fully synchronise a queue manually, refer to unsynchronised mirrors section for details.

ha-sync-mode: automatic
- a queue will automatically synchronise when a new mirror joins. It is worth reiterating that queue synchronisation is a blocking operation. If queues are small, or you have a fast network between RabbitMQ nodes and the ha-sync-batch-size was optimised, this is a good choice.

for / vhost
rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

Setting policy "ha-all" for pattern "" to "{"ha-mode":"all","ha-sync-mode":"automatic"}" with priority "0" for vhost "/" ...

for vhost1 vhost
rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}' -p vhost1

Setting policy "ha-all" for pattern "" to "{"ha-mode":"all","ha-sync-mode":"automatic"}" with priority "0" for vhost "vhost1" ...

Linux ucarp promote slave to master

Проверяем, что nginx конфиги идентичны на обоих серверах.

https://kamaok.org.ua/?p=1994

# номер группы (число от 1 до 255) одинаковый для всех серверов;
ucarp-vid 1

# отказоустойчивый IP-адрес(VIP)
ucarp-vip 192.168.1.125

# пароль/ключ для шифрования сетевого протокола, одинаковый для серверов в пределах группы;
ucarp-password fahDo7zaze

# управление приоритетом выбора мастера из нескольких кандидатов группы (Чем больше число, тем меньше вероятность, что хост станет Мастером)
ucarp-advskew 1

# интервал оповещения хостов в группе о своем статусе;
ucarp-advbase 1

# Роль в группе при запуске. используется для повышения приоритета при выборе мастера.
ucarp-master yes

# Виртуальный интерфейс
iface eth0:ucarp inet static
address 192.168.1.125
netmask 255.255.255.0

Отсюда следует, 

1. меняем ucarp-advskew 50 для slave и 10 для master
2. ucarp-master yes и ucarp-master no меняем соответсвенно

перегружаем настройки на лету

https://serverfault.com/questions/298776/ucarp-promote-slave-to-master

на мастере запускаем

killall -USR2 ucarp

проверяем ip a s 

Linux nginx configs backup

Буду использовать отдельно созданный для этого ssh key из специально созданной папки

mkdir -p /root/sshkey-for-nginx-backup
chmod 0600 /root/sshkey-for-nginx-backup

ssh-keygen -t rsa -b 4096 -f /root/sshkey-for-nginx-backup/id_rsa -C sshkey-for-nginx-backup@hostname

copy ssh pub key to remote server

apt install zstd

#!/bin/bash

function logging {
    echo $(date +%F_%H-%M) "______" "$1"
}

current_date=`date +%F`

tar -cf - /etc/nginx /etc/letsencrypt | zstd -19 | ssh -p 2202 -i /root/sshkey-for-nginx-backup/id_rsa -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no hostname-backup@1.2.3.4 "cat - > /var/www/backup/hostname/nginx.hostname.$current_date.tar.zst" && logging "Backup was sent successfully ..." || logging "Backup wasn't created ..."

Скрипт будет запускаться по крону каждый день

#Backup nginx configs to backup server

20 5 * * * /usr/local/bin/backup-nginx-config.sh >> /var/log/backup-nginx-config.log 2>&1

LXC Operation not permitted - Failed to create veth pair "vc1" and "veth

Error:

lxc ct100 20190722062355.822 ERROR network - network.c:instantiate_veth:106 - Operation not permitted - Failed to create veth pair "vc1" and "vethWGA60H"
lxc ct100 20190722062355.822 ERROR network - network.c:lxc_create_network_priv:2457 - Failed to create network device
lxc ct100 20190722062355.822 ERROR start - start.c:lxc_spawn:1626 - Failed to create the network
lxc ct100 20190722062355.822 ERROR start - start.c:__lxc_start:1939 - Failed to spawn container "ct100"
lxc abc100 20190722062355.822 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:842 - Received container state "STOPPING" instead of "RUNNING"
lxc 20190722062355.832 WARN commands - commands.c:lxc_cmd_rsp_recv:132 - Connection reset by peer - Failed to receive response for command "get_state"
Решение

modprobe veth

Linux Bash parse IPs from log by word

Парсить IP с отозванными сертификатами из лога

Mon Jul 22 05:38:54 2019 1.2.3.4:6252 VERIFY ERROR: depth=0, error=certificate revoked: ...

grep revoked openvpn.log | awk '{ print $6 }' | awk -F":" '{ print $1 }' | sort | uniq -u

awk -F":" разделитель, это парсим 1.2.3.4:6252 для получения только IP
Чтобы использовать uniq, сначала нужно отсортировать sort

Linux grep remove # and white_space

Для удобства чтения конфигов

grep -v '#' redis.conf | grep -v "^[[:space:]]*$"

Python запуск команд на удаленном сервере over SSH

Установка

pip3 install paramiko

Пример проверки есть ли уже пользователь на удаленном компьютере:

import paramiko

host = '10.81.156.13'
user = 'root'
#файл может находится в любом удобном месте
keyfilename = '/root/ssh_key_for cheking/id_rsa'
create_user = "newuser"

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
k = paramiko.RSAKey.from_private_key_file(keyfilename)
ssh.connect(hostname=host, port=22, username=user, pkey=k)

cmd = 'grep -c -w ' + create_user + ' /etc/passwd'

ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(cmd)

result_ssh = ssh_stdout.readlines()
result_ssh = "".join(result_ssh)
#print(result_ssh[0])

if result_ssh[0] == '0' and result_ssh[0].isdigit():
    print("user not exists and it will be created")