Linux Rsync with particular extension

 rsync -zarv --prune-empty-dirs --include "*/"  --include="*.gif" --exclude="*" ./ ../gifs/

Linux Bash rsync remote to remote with forwarding keys

We need it for loading files directly from one remote server to another one where we have SSH access.

SSH access from a Workstation -> Server1 -> rsync loading files -> Server2

ssh -v username@server1

find an identity SSH key for sure

ssh-add <the_key>

ssh -A username@server1

run with A option for allowing forwarding keys 

rsync -azP --exclude '.4y8-folder' --exclude 'file.tar.gz' pub/media/ username@server2:pub/media/

Linux Bash rsync only specific a files list

Check rsync --help whether it supports --files-from= option 

lst content

./folder-source/1/1

./folder-source/2/2

...

cd to a folder where folder-source as a sub-folder (it's an explanation of period "." in rsync command)

rsync --files-from=/tmp/lst . /app/var/folder-sync

Linux Vim replace two spaces to four spaces

 Open a file in vim

then execute a command

:%s/^\s*/&&

from here

Linux logrotate jira logs

 Jira has its own log rotate

However, it doesn't delete old files.

It can be done with crontab though.

#### remove old jira log files longer 180 days

30 2 * * * /usr/bin/find /opt/atlassian/servicedesk/logs/ -maxdepth 1 -type f -mtime +180 -exec rm -rf {} \;

45 2 * * * /usr/bin/find /opt/atlassian/servicedesk/logs/ -maxdepth 1 -type f -mtime +6 -exec gzip -q {} \;

Linux logrotate twice per day

Before:

/etc/logrotate.d/logs

/mnt/shared/project/var/log/*.log 

{

  su www-project www-data

  rotate 30

  daily

  missingok

  compress

  notifempty

  copytruncate

  dateext

}

After:

1) create a separate additional logrotate conf file /etc/logrotate.d/logs_erp with content below (increased twice rotate 30 -> 60 and plus delaycompress):

/etc/logrotate.d/logs_erp

/mnt/shared/project/var/log/erp.log 

{

  su www-project www-data

  rotate 60

  daily

  missingok

  compress

  delaycompress

  notifempty

  copytruncate

  dateext

}

2) modify current logrotate conf file /etc/logrotate.d/logs with excluding first 'erp' characters from wildcard:

/mnt/shared/project/var/log/[!e][!r][!p]*.log 

{

  su www-project www-data

  rotate 30

  daily

  missingok

  compress

  notifempty

  copytruncate

  dateext

}

3) run logrotate conf file /etc/logrotate.d/logs_erp by cron additionally at 18.25

under root account

25 18 * * * logrotate -f /etc/logrotate.d/logs_erp

Linux Bash check disk space

df --output=pcent / | tail -n +2 | tr -cd '0-9\n'

output available percent without header and symbol '%' - only number

#!/bin/bash

for mnt in / /var/log /var/www /mnt/shared; do

  chk_avail=`df --output=pcent $mnt | tail -n +2 | tr -cd '0-9\n'`

  flg=`echo $(( 100 - $chk_avail ))`

  if [ $flg -lt 15 ]; then

    echo 'Dataserver lack of free space, only '$flg' percent available in '$mnt' partition' | mail -s 'Dataserver lack of free space' alerts@gmail.com

  fi

done

find for time interval

 find . -maxdepth 1 -newermt '2019-01-01' ! -newermt '2021-02-15' -name "importexport_*" -exec ls -l '{}' \;

Linux Ubuntu 18.04 Use redis for cache pages and PHP sessions storage

apt update -y && apt install redis-server php-redis

/etc/php/7.2/fpm/pool.d/user.conf

php_admin_value[session.save_handler] = redis

php_admin_value[session.save_path] = "tcp://127.0.0.1:6379?persistent=1&weight=1&database=2"

phpinfo should display redis info on the page.

There is link https://github.com/phpredis/phpredis#php-session-handler for save_path parameters

The checking script is 

<?php

  $redisinstance = new Redis();

  $redisinstance->connect("127.0.0.1", 6379);

  $result = $redisinstance->get("test");

  if ($result) {

      echo $result;

  } else {

      echo "No matching key found. Refresh the browser to add it!";

      $redisinstance->set("test", "Successfully retrieved the data!") or die("Couldn't save anything to redis...");

  }

?>

After refreshing page

redis-cli

127.0.0.1:6379> select 0

OK

127.0.0.1:6379> keys "*"

1) "test"

127.0.0.1:6379> get "test"

"Successfully retrieved the data!"

Linux Ububntu 18.04 Use memcached for PHP session storage

https://devdocs.magento.com/guides/v2.3/config-guide/memcache/memcache_ubuntu.html

apt update -y && apt install php-memcached memcached
/etc/php/7.2/fpm/php.ini
session.save_handler = "memcached"
session.save_path = "127.0.0.1:11211"

or /etc/php/7.2/fpm/pool.d/user.conf
php_admin_value[session.save_path] = '127.0.0.1:11211'
php_admin_value[session.save_handler] = 'memcached'

phpinfo should display memcached info on the page.

Check working with a script cache-test.php

Linux Postfix 530 5.7.0 must issue a starttls command first

при отправке ошибка
Решение:
было в master.cf
-o smtpd_tls_security_level=encrypt
помогло
-o smtpd_tls_security_level=may

Linux mariadb doesn't start after upgrading to Debian 10

Mariadb version is 10.3. After upgrading there were errors within service starting:

status
Process: 1232 ExecStartPost=/etc/mysql/debian-start (code=exited, status=203/EXEC)

journal -xe
mariadb.service: Failed at step EXEC spawning /etc/mysql/debian-start: No such file or directory

resolving
https://stackoverflow.com/questions/51317209/mariadb-10-3-8-installation-from-mariadb-repo-missing-files

apt install --reinstall -o Dpkg::Options::="--force-confmiss" mariadb-server-10.3

Configuration file '/etc/mysql/debian-start', does not exist on system.

Installing new config file as you requested.

Configuration file '/etc/mysql/mariadb.conf.d/50-mysqld_safe.cnf', does not exist on system.

Installing new config file as you requested.

Configuration file '/etc/mysql/mariadb.conf.d/50-server.cnf', does not exist on system.

Installing new config file as you requested.

apt install --reinstall -o Dpkg::Options::="--force-confmiss" mysql-common

Configuration file '/etc/mysql/my.cnf', does not exist on system.

Installing new config file as you requested.

Afterwards, there were added requested files.

226 error in LXC container

lxc config set container1 security.nesting true

Linux RabbitMQ-server cluster

https://www.rabbitmq.com/clustering.html
https://kamaok.org.ua/?p=2689
https://blog.flux7.com/blogs/tutorials/how-to-creating-highly-available-message-queues-using-rabbitmq

1) add hostname (do real hostname) between master and slave to /etc/hosts

2) copy /var/lib/rabbitmq/.erlang.cookie from master to slave

systemctl restart rabbitmq-server (on slave)


3) join to cluster

rabbitmqctl stop_app

Stopping rabbit application on node rabbit@<hostname_slave> ...

rabbitmqctl reset

Resetting node rabbit@<hostname_slave> ...

rabbitmqctl join_cluster rabbit@<hostname_slave>

Clustering node rabbit@<hostname_slave> with rabbit@<hostname_master>

rabbitmqctl start_app

Starting node rabbit@<hostname_slave> ...
completed with 0 plugins.


4) check cluster

rabbitmqctl cluster_status

Cluster status of node rabbit@<hostname_slave> ...

[{nodes,[{disc,[rabbit@<hostname_master>,rabbit@<hostname_slave>]}]},

{running_nodes,[rabbit@<hostname_master>,rabbit@<hostname_slave>]},

{cluster_name,<<"rabbit@localhost">>},
{partitions,[]},

{alarms,[{rabbit@<hostname_master>,[]},{rabbit@<hostname_slave>,[]}]}]

5) Configure syncing queues police between master and slave nodes

Run on master

rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

for all vhost

https://stackoverflow.com/questions/38611583/rabbitmq-is-there-a-way-to-apply-policy-on-all-virtual-host-in-cluster-environ

from
https://www.rabbitmq.com/ha.html

ha-sync-mode: manual
- this is the default mode. A new queue mirror will not receive existing messages, it will only receive new messages. The new queue mirror will become an exact replica of the master over time, once consumers have drained messages that only exist on the master. If the master queue fails before all unsychronised messages are drained, those messages will be lost. You can fully synchronise a queue manually, refer to unsynchronised mirrors section for details.

ha-sync-mode: automatic
- a queue will automatically synchronise when a new mirror joins. It is worth reiterating that queue synchronisation is a blocking operation. If queues are small, or you have a fast network between RabbitMQ nodes and the ha-sync-batch-size was optimised, this is a good choice.

for / vhost
rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

Setting policy "ha-all" for pattern "" to "{"ha-mode":"all","ha-sync-mode":"automatic"}" with priority "0" for vhost "/" ...

for vhost1 vhost
rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}' -p vhost1

Setting policy "ha-all" for pattern "" to "{"ha-mode":"all","ha-sync-mode":"automatic"}" with priority "0" for vhost "vhost1" ...

Linux ucarp promote slave to master

Проверяем, что nginx конфиги идентичны на обоих серверах.

https://kamaok.org.ua/?p=1994

# номер группы (число от 1 до 255) одинаковый для всех серверов;
ucarp-vid 1

# отказоустойчивый IP-адрес(VIP)
ucarp-vip 192.168.1.125

# пароль/ключ для шифрования сетевого протокола, одинаковый для серверов в пределах группы;
ucarp-password fahDo7zaze

# управление приоритетом выбора мастера из нескольких кандидатов группы (Чем больше число, тем меньше вероятность, что хост станет Мастером)
ucarp-advskew 1

# интервал оповещения хостов в группе о своем статусе;
ucarp-advbase 1

# Роль в группе при запуске. используется для повышения приоритета при выборе мастера.
ucarp-master yes

# Виртуальный интерфейс
iface eth0:ucarp inet static
address 192.168.1.125
netmask 255.255.255.0

Отсюда следует, 

1. меняем ucarp-advskew 50 для slave и 10 для master
2. ucarp-master yes и ucarp-master no меняем соответсвенно

перегружаем настройки на лету

https://serverfault.com/questions/298776/ucarp-promote-slave-to-master

на мастере запускаем

killall -USR2 ucarp

проверяем ip a s 

Linux nginx configs backup

Буду использовать отдельно созданный для этого ssh key из специально созданной папки

mkdir -p /root/sshkey-for-nginx-backup
chmod 0600 /root/sshkey-for-nginx-backup

ssh-keygen -t rsa -b 4096 -f /root/sshkey-for-nginx-backup/id_rsa -C sshkey-for-nginx-backup@hostname

copy ssh pub key to remote server

apt install zstd

#!/bin/bash

function logging {
    echo $(date +%F_%H-%M) "______" "$1"
}

current_date=`date +%F`

tar -cf - /etc/nginx /etc/letsencrypt | zstd -19 | ssh -p 2202 -i /root/sshkey-for-nginx-backup/id_rsa -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no hostname-backup@1.2.3.4 "cat - > /var/www/backup/hostname/nginx.hostname.$current_date.tar.zst" && logging "Backup was sent successfully ..." || logging "Backup wasn't created ..."

Скрипт будет запускаться по крону каждый день

#Backup nginx configs to backup server

20 5 * * * /usr/local/bin/backup-nginx-config.sh >> /var/log/backup-nginx-config.log 2>&1

Linux Bash parse IPs from log by word

Парсить IP с отозванными сертификатами из лога

Mon Jul 22 05:38:54 2019 1.2.3.4:6252 VERIFY ERROR: depth=0, error=certificate revoked: ...

grep revoked openvpn.log | awk '{ print $6 }' | awk -F":" '{ print $1 }' | sort | uniq -u

awk -F":" разделитель, это парсим 1.2.3.4:6252 для получения только IP
Чтобы использовать uniq, сначала нужно отсортировать sort

Linux grep remove # and white_space

Для удобства чтения конфигов

grep -v '#' redis.conf | grep -v "^[[:space:]]*$"

Linux sudo keep environment variables APP_ENV

https://stackoverflow.com/questions/8633461/how-to-keep-environment-variables-when-using-sudo

Добавить в sudoers.d в пользовательский конфиг

Defaults env_keep += "APP_ENV"

Linux /bin/ls: Argument list too long

Нужно было чистить каталог, но стандартная очистка с созданием списка ls показала ошибку при первом запуске с большим количеством файлов (~400 000)
Решение https://www.in-ulm.de/~mascheck/various/argmax/ , из параграфа
How to avoid the limit in a shell выбрал этот вариант:
find /home/folder -mtime +7 -print0 -type f | xargs -0 rm -rf

Все файлы старше семи дней удаляются

Linux rabbitmq-server add read-only user

Включаем плагины менеджмента

rabbitmq-plugins enable rabbitmq_management
Заводим нового пользователя

rabbitmqctl add_user username usernamePassword

Права по порядку config, write, read Даем полные права только на чтение ".*"

rabbitmqctl set_permissions -p / username "^username.*" "^username.*" ".*"

Чтобы подключаться удаленно, нужно дать доступ

по описанию monitiring самый подходящий для этого

rabbitmqctl set_user_tags radly monitoring