AdminSysGNULinux

Cette page est en lecture seule. Vous pouvez afficher le texte source, mais ne pourrez pas le modifier. Contactez votre administrateur si vous pensez qu'il s'agit d'une erreur.
<!DOCTYPE markdown>
{{tag>Brouillon}}

# Haute dispo cluster failover redhat

Voir aussi :
* OpenSVC
* Paquet resource-agents

Ressources :
* myvip
* fence_node-1
* fence_node-2
* ping
* srvweb
* ClusterMon-External


Liens intros :
* https://www.youtube.com/watch?v=Xl0TQer9uZc
* https://www.youtube.com/watch?v=RbJwluN_qrY

## Installation

Voir :
* http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Clusters_from_Scratch/
* http://clusterlabs.org/doc/en-US/Pacemaker/

### Prérequis

Prérequis
  * Date syncho
  * SELinux désactivé
  * service NetworkManager arrêté 
  * Règles pare-feu
  * Conf ''/etc/hosts''


#### Date synchro (ntp)

Les nœuds doivent avoir la date et l'heure synchronisée (voir NTP)

Vérif
~~~bash
date
~~~

Exemple avec Clush [[cluster_shell_parallele]]
~~~bash
echo date |clush -B -w node-[1-2]
~~~


#### SELinux désactivé

~~~bash
setenforce 0
sed -i.bak "s/SELINUX=enforcing/SELINUX=permissive/g" /etc/selinux/config
~~~

Vérif
~~~bash
sestatus
~~~

#### Service NetworkManager arrêté et désactivé

~~~bash
systemctl stop NetworkManager
systemctl disable NetworkManager
~~~

#### Pare-feu

Si pare-feu activé
~~~bash
firewall-cmd --permanent --add-service=high-availability
firewall-cmd --add-service=high-availability
~~~

Ou

Désactivation du parefeux
~~~bash
systemctl stop firewalld
systemctl disable firewalld
#rpm -e firewalld
~~~

Vérif
~~~bash
iptables -L -n -v
~~~


#### Résolution noms

Chaque nœud doit pouvoir pinguer les autres via son nom.
Il est conseiller d'utiliser ''/etc/hosts'' plutôt que DNS.

''/etc/hosts''
~~~
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
127.0.1.1	   node-1.localdomain
192.168.97.221      node-1.localdomain node-1
192.168.97.222      node-2.localdomain node-2
~~~

### Install

Install paquets
~~~bash
yum install -y pacemaker pcs psmisc policycoreutils-python
~~~


~~~bash
echo "P@ssw0rd" | passwd hacluster --stdin
 
systemctl start pcsd.service
systemctl enable pcsd.service

#unset http_proxy
#export NO_PROXY=localhost,127.0.0.1,node-1,node-2
pcs cluster auth node-1 node-2 #-u hacluster -p passwd
 
#pcs cluster setup --start --name my_cluster node-1 node-2
pcs cluster setup --name my_cluster node-1 node-2

pcs cluster start --all
pcs cluster enable --all
~~~


Le fichier corosync.conf est automatiquement crée

''/etc/corosync/corosync.conf''
~~~c
totem {
    version: 2
    secauth: off
    cluster_name: my_cluster
    transport: udpu
}

node-list {
    node- {
        ring0_addr: node-1
        node-id: 1
    }

    node- {
        ring0_addr: node-2
        node-id: 2
    }
}

quorum {
    provider: corosync_votequorum
    two_node-: 1
}

logging {
    to_logfile: yes
    logfile: /var/log/cluster/corosync.log
    to_syslog: yes
}
~~~


Vérifier la conf de corosync 1
~~~bash
corosync-cfgtool -s
~~~
Doit retourner **no faults** \\
Ne doit pas comporter d’adresse 127.0.0.1

Vérifier la conf de corosync 2
~~~bash
corosync-cmapctl  |grep members
pcs status corosync
~~~

### Configuration

Prevent Resources from Moving after Recovery
~~~bash
pcs resource defaults resource-stickiness=100
~~~

Pas de quorum
~~~bash
#pcs property set no-quorum-policy=ignore
pcs property set no-quorum-policy=freeze
~~~


## Configuration du fencing / stonith

### Test en vue du fencing via iDRAC

Voir https://www.devops.zone/tricks/connecting-ssh-drac-reboot-server/

Tester du fencing
~~~bash
/usr/sbin/fence_drac5 --ip=192.168.96.221 --username=root --password=calvin --ssh -c 'admin1->'
~~~

Test avec OpenManage ''/opt/dell/srvadmin/sbin/racadm''
~~~bash
racadm -r 192.168.96.221 -u root -p calvin get iDRAC.Info
~~~

Test via SSH sur iDRAC
Pour redemarrer le serveur en se connectant en SSH sur la iDRAC
~~~
ssh root@192.168.96.221
racadm serveraction powercycle
~~~

** Si pas de stonith / fence ** sinon la VIP refusera de démarrer
~~~bash
# Si pas de stonith / fence
pcs property set stonith-enabled=false
~~~

### Vérif

~~~bash
crm_verify -LVVV
~~~

### Configuration

~~~bash
# pcs stonith create fence_node-1 fence_drac5 ipaddr=192.168.96.221 login=root passwd=calvin secure=1 cmd_prompt="/admin1->" pcmk_host_list=node-1 stonith-action=poweroff
pcs stonith create fence_node-1 fence_drac5 ipaddr=192.168.96.221 login=root passwd=calvin secure=1 cmd_prompt="/admin1->" pcmk_host_list=node-1 op monitor interval="60s"
pcs stonith create fence_node-2 fence_drac5 ipaddr=192.168.96.222 login=root passwd=calvin secure=1 cmd_prompt="/admin1->" pcmk_host_list=node-2 op monitor interval="60s"

pcs stonith level add 1 node-1 fence_node-1
pcs stonith level add 1 node-2 fence_node-2
~~~

Interdire le suicide (le fencing de soi-même)
~~~bash
pcs constraint location fence_node-1 avoids node-1
pcs constraint location fence_node-2 avoids node-2
~~~

Tester le fencing
~~~bash
#stonith_admin --reboot node-1
pcs stonith fence node-1
~~~



### Ajout ressources

Ajout ressource VIP (adresse IP virtuelle)
~~~bash
pcs resource create myvip IPaddr2 ip=192.168.97.230 cidr_netmask=24 nic=bond0 op monitor interval=30s on-fail=fence
#pcs constraint location myvip prefers node-1=INFINITY
pcs constraint location myvip prefers node-1=100
pcs constraint location myvip prefers node-2=50
#pcs resource meta myvip resource-stickiness=100
~~~

Ajouter ressource ping
~~~bash
pcs resource create ping ocf:pacemaker:ping dampen=5s multiplier=1000 host_list=192.168.97.250 --clone
pcs constraint location myvip rule score=-INFINITY pingd lt 1 or not_defined pingd
~~~

Ajout ressource Apache

Avant il faut configurer http://localhost/server-status
et arrêter le service d'apache sur l'ensemble des nœuds

~~~bash
curl http://localhost/server-status
systemctl stop httpd.service
systemctl disable httpd.service
~~~

~~~bash
pcs resource create srvweb apache configfile="/etc/httpd/conf/httpd.conf" statusurl="http://127.0.0.1/server-status" op monitor interval=1min #--clone
# Le serveur Web toujours sur la VIP
pcs constraint colocation add srvweb with myvip
# D'abord la VIP puis le serveur Web
pcs constraint order myvip then srvweb
~~~

## Manip

Déplacer la VIP
~~~bash
pcs resource move myvip node-1
pcs resource move myvip node-2
~~~

Retour arrière - Déplacer la VIP
~~~bash
#pcs constraint --full |grep prefer
pcs constraint remove cli-prefer-myvip
pcs resource relocate run
~~~

Remise à zero compteur erreurs
~~~bash
#pcs resource failcount reset res1
#crm_resource -P
pcs resource cleanup
~~~

Déplacer toutes les ressources sur le nœud primaire (ignoring resource stickiness)
~~~bash
#pcs resource relocate show
pcs resource relocate run
~~~

Maintenance sur une ressource
~~~bash
#pcs resource update fence_node-1 meta target-role=stopped
#pcs resource update fence_node-1 meta is-managed=false
#pcs resource update fence_node-1 op monitor enabled=false
#pcs resource disable fence_node-1
pcs resource unmanage fence_node-1
~~~

Maintenance générale du cluster
~~~bash
pcs property set maintenance-mode=true
~~~

Fin de maintenance
~~~bash
pcs property set maintenance-mode=false
~~~

Arrêt du cluster
~~~bash
pcs cluster stop --all
pcs cluster disable --all 
~~~

## Diagnostic / Supervision

### Diag Passif

[[http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Clusters_from_Scratch/_verify_corosync_installation.html|Vérif corosync]]
~~~bash
# Check syntax conf
corosync -t

# Check cluster communication
corosync-cfgtool -s

# check the node's network
corosync-cmapctl  |grep members
~~~

Vérif
~~~bash
pcs cluster pcsd-status
pcs cluster verify
pcs status corosync
crm_mon -1 --fail
crm_mon -1Af
journalctl --since yesterday -p err
journalctl -u pacemaker.service --since "2017-02-24 16:00" -p warning
~~~

Script supervision (ces commandes doivent retourner aucune ligne)
~~~bash
LANG=C pcs status |egrep "Stopped|standby|OFFLINE|UNCLEAN|Failed|error"
crm_verify -LVVV
LANG=C pcs resource relocate show |sed -ne '/Transition Summary:/,$p' |grep -v '^Transition Summary:'
crm_mon -1f | grep -q fail-count
~~~

Voir plus haut si (script ''/usr/local/bin/crm_logger.sh'')
~~~bash
tailf  /var/log/messages |grep "ClusterMon-External:"
~~~

Script supervision
Quel nœud est actif
~~~bash
LANG=C crm_resource --resource myvip --locate |cut -d':' -f2 |tr -d ' '
~~~

Le serveur web répond t-il bien en utilisant l'IP de la VIP. (Le code de retour doit-être **0**)
~~~bash
#curl -4 -m 1 --connect-timeout 1 http://192.168.97.230/ > /dev/null 2>&1
curl -4 -m 1 --connect-timeout 1 http://192.168.97.230/cl.html > /dev/null 2>&1
#echo $?
~~~

### ACL

Compte en lecture seule avec les droits de consulter crm_mon \\

Attention : ce compte trouver le mdp iDRAC/Ilo
~~~bash
pcs stonith --full |grep passwd
~~~

Mise en œuvre 
~~~bash
#adduser rouser
#usermod -a -G haclient rouser
usermod -a -G haclient process

pcs property set enable-acl=true
pcs acl role create read-only description="Read access to cluster" read xpath /cib
#pcs acl user create rouser read-only
pcs acl user create process read-only
~~~

~~~bash
#crm_mon --daemonize --as-html /var/www/html/cl.html
~~~

''/usr/local/bin/crm_logger.sh''
~~~bash
#!/bin/sh
# https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/pdf/High_Availability_Add-On_Reference/Red_Hat_Enterprise_Linux-7-High_Availability_Add-On_Reference-en-US.pdf
logger -t "ClusterMon-External" "${CRM_notify_node:-x} ${CRM_notify_rsc:-x} \
${CRM_notify_task:-x} ${CRM_notify_desc:-x} ${CRM_notify_rc:-x} \
${CRM_notify_target_rc:-x} ${CRM_notify_status:-x} ${CRM_notify_recipient:-x}";
exit
~~~

~~~bash
chmod 755 /usr/local/bin/crm_logger.sh
chown root.root /usr/local/bin/crm_logger.sh
~~~

~~~bash
pcs resource create ClusterMon-External ClusterMon update=10000 user=process extra_options="-E /usr/local/bin/crm_logger.sh --watch-fencing" htmlfile=/var/www/html/cl.html pidfile=/tmp/crm_mon-external.pid op monitor on-fail="restart" interval="60" clone
~~~

Colocation - page de monitoting toujours actif sur la VIP \\
**Seulement nécessaire si ressource non clonée**
~~~bash
pcs constraint colocation add ClusterMon-External with myvip
~~~

Test
~~~bash
curl 192.168.97.230/cl.html
~~~
Voir https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Reference/s1-eventnotification-HAAR.html

### Diag Actif

En cas de pb
~~~bash
pcs resource debug-start resource_id
~~~

## Ajout 2em interface pour le heartbeat 


Redundant Ring Protocol (RRP)
rrp_mode
If set to active, Corosync uses both interfaces actively.
If set to passive, Corosync sends messages alternatively over the available networks. 

Avant de modifier la conf, on passe le cluster en mode maintenance :
~~~bash
pcs property set maintenance-mode=true
~~~

''/etc/hosts''
~~~
192.168.21.10      node1
192.168.22.10     node1b
192.168.21.11      node2
192.168.22.11     node2b
~~~

On ajoute **rrp_mode** et **ring1_addr** 
''/etc/corosync/corosync.conf''
~~~c
totem {
    rrp_mode: active
}
nodelist {
    node {
        ring0_addr: node1
        ring1_addr: node1b
        nodeid: 1
    }

    node {
        ring0_addr: node2
        ring1_addr: node2b
        nodeid: 2
    }
}
~~~

~~~bash
pcs cluster reload corosync
pcs cluster status corosync
corosync-cfgtool -s
pcs property unset maintenance-mode
~~~


## Reprise sur incident

~~~bash
#crm_resource -P
pcs resource cleanup
pcs resource relocate run
#pcs cluster start --all
~~~

## Crash-tests

Test 1 Crash brutal
~~~bash
echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger
~~~

Test 2 Coupure électrique : Débranchement du câble

Test 3 Coupure réseaux
~~~bash
ifdown bond0
~~~

Test 4 Perte du ping de la passerelle sur l'un des nœud
~~~bash
iptables -A OUTPUT -d 192.168.97.250/32 -p icmp -j REJECT
~~~

Test 5 Fork bomb, nœud ne répond plus, sauf au ping

Fork bomb
~~~bash
:(){ :|:& };:
~~~

Test 6 Perte connexion iDRAC :  Débranchement du câble



## Nettoyage - effacer

~~~bash
pcs cluster stop  --force #--all
pcs cluster destroy  --force #--all

systemctl stop pcsd
systemctl stop corosync
systemctl stop pacemaker

yum remove -y pcsd corosync pacemaker
userdel hacluster

rm -rf /dev/shm/qb-*-data /dev/shm/qb-*-header

rm -rf /etc/corosync
rm -rf /var/lib/corosync
rm -rf /var/lib/pcsd
rm -rf /var/lib/pacemaker

rm -rf /var/log/cluster/
rm -rf /var/log/pcsd/
rm -f /var/log/pacemaker.log*
~~~
 


## Erreurs

### 1 Erreur Dell hardware

~~~
UEFI0081: Memory size has changed from the last time the system was started. No action is required if memory was added or removed. 
~~~

http://www.dell.com/support/manuals/fr/fr/frbsdt1/integrated-dell-remote-access-cntrllr-8-with-lifecycle-controller-v2.00.00.00/eemi_13g-v1/UEFI-Event-Messages?guid=GUID-C1C6F253-F8EF-43BF-B8ED-1A9B2A910AC4&lang=en-us

### 2 Test fork-bomb

~~~
error: Integration Timer (I_INTEGRATED) just popped in state S_INTEGRATION! (180000ms)
~~~


## Autres

### Pour voir / vérifier les "property"

~~~bash
#pcs property set symmetric-cluster=true
pcs property
~~~

### Ressources

Lister
~~~bash
pcs resource standards
~~~
~~~
ocf
lsb
service
systemd
stonith
~~~

~~~bash
pcs resource providers
~~~

~~~
heartbeat
openstack
pacemaker
~~~

Lister les agents : Exemple
~~~bash
pcs resource agents systemd
pcs resource agents ocf:heartbeat
~~~

Timeout par défaut pour les ressources
~~~bash
pcs resource op defaults timeout=240s
~~~

Stopper toutes les ressources
~~~bash
pcs property set stop-all-resources=true
~~~

~~~bash
pcs property unset stop-all-resources
~~~

ocf:pacemaker:ping \\
''/usr/lib/ocf/resource.d/pacemaker/ping''

ocf:heartbeat:apache \\
''/usr/lib/ocf/resource.d/heartbeat/apache''

~~~bash
egrep '^#.*OCF_RESKEY' /usr/lib/ocf/resource.d/heartbeat/apache
export OCF_ROOT=/usr/lib/ocf/
/usr/lib/ocf/resource.d/heartbeat/apache meta-data
~~~

Autre
Lister toutes les ressources
~~~bash
crm_resource --list
~~~

Dump CIB (Cluster Information Base)
~~~bash
pcs cluster cib
pcs cluster cib cib-dump.xml
~~~

Ajout d'une ressource service
~~~bash
pcs resource create CRON systemd:crond
#pcs resource op add CRON start interval=0s timeout=1800s
~~~

UPDATE
~~~bash
pcs resource update ClusterMon-External  htmlfile='/tmp/cl.html'
~~~

UNSET
~~~bash
pcs resource update ClusterMon-External  htmlfile=
~~~
### Stonith

~~~bash
pcs property list --all |grep stonith
~~~

Confirmer que le nœud est bien arrêté. \\
Attention, si ce n'est pas le cas risque de pb
~~~bash
pcs stonith confirm node2
~~~

### Failcount

~~~bash
crm_mon --failcounts

pcs resource failcount show resource_id
pcs resource failcount reset resource_id
~~~

Actualisation de l’état, et remise à zéro du "failcount"
~~~bash
pcs resource cleanup resource_id
~~~

### Install depuis zero

~~~bash
echo "P@ssw0rd" |passwd hacluster --stdin
systemctl start pcsd.service
systemctl enable pcsd.service
pcs cluster auth -u hacluster -p P@ssw0rd 8si-pms-pps-srv-1 8si-pms-pps-srv-2
		
pcs cluster setup --name my_cluster 8si-pms-pps-srv-1 8si-pms-pps-srv-2
pcs cluster start --all
pcs cluster enable --all

pcs resource defaults resource-stickiness=100
pcs property set no-quorum-policy=freeze

pcs stonith create fence_8si-pms-pps-srv-1 fence_drac5 ipaddr=172.18.202.230 login=root passwd=calvin secure=1 cmd_prompt="/admin1->" pcmk_host_list=8si-pms-pps-srv-1 op monitor interval="60s"
pcs stonith create fence_8si-pms-pps-srv-2 fence_drac5 ipaddr=172.18.202.231 login=root passwd=calvin secure=1 cmd_prompt="/admin1->" pcmk_host_list=8si-pms-pps-srv-2 op monitor interval="60s"
 
pcs stonith level add 1 8si-pms-pps-srv-1 fence_8si-pms-pps-srv-1
pcs stonith level add 1 8si-pms-pps-srv-2 fence_8si-pms-pps-srv-2

pcs constraint location fence_8si-pms-pps-srv-1 avoids 8si-pms-pps-srv-1
pcs constraint location fence_8si-pms-pps-srv-2 avoids 8si-pms-pps-srv-2

pcs resource create myvip IPaddr2 ip=172.18.202.226 cidr_netmask=24 nic=bond0 op monitor interval=30s #on-fail=fence
pcs constraint location myvip prefers 8si-pms-pps-srv-1=100
pcs constraint location myvip prefers 8si-pms-pps-srv-2=50
#pcs resource meta myvip resource-stickiness=60

# l'utilisateur process doit appartenir au groupe haclient
#usermod -a -G haclient process
pcs property set enable-acl=true
pcs acl role create read-only description="Read access to cluster" read xpath /cib
pcs acl user create process read-only

pcs resource create ClusterMon-External ClusterMon update=10000 user=process extra_options="-E /usr/local/bin/crm_logger.sh --watch-fencing" htmlfile=/var/www/html/cl.html pidfile=/tmp/crm_mon-external.pid op monitor on-fail="restart" interval="60" clone

pcs resource create appmgr systemd:appmgr
pcs constraint colocation add appmgr with myvip
~~~

------------------

Voir aussi :
* Généralité
** https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Configuration_Example_-_Oracle_HA_on_Cluster_Suite/sample_2_node.html
* keepalived
** http://tecadmin.net/ip-failover-setup-using-keepalived-on-centos-redhat-6/#
* Ricci Luci Ccs cman **ccs, ricci and luci are deprecated**
** http://www.tecmint.com/cman-multi-node-cluster-setup-in-linux/
** http://www.thegeekstuff.com/2015/03/linux-redhat-cluster
* Css
** http://www.thegeekstuff.com/2015/03/redhat-cluster-failover/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+TheGeekStuff+
** https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Cluster_Administration/s1-config-failover-domain-cli-CA.html
* Autres
** https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Configuration_Example_-_Oracle_HA_on_Cluster_Suite/cold_failover.html
** http://www.cyberciti.biz/faq/rhel-centos-fedora-keepalived-lvs-cluster-configuration/

Fencing
* https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Reference/ch-fencing-HAAR.html
* https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Overview/s1-fencing-HAAO.html
* http://www.linuxtopia.org/online_books/rhel6/rhel_6_cluster_admin/rhel_6_cluster_s1-config-failover-domain-cli-CA.html

Cluster
* https://www.guillaume-leduc.fr/lequilibrage-de-charge-sous-linux-avec-lvs-theorie.html
* http://www.ultramonkey.org/papers/lvs_tutorial/html/
* http://connect.ed-diamond.com/GNU-Linux-Magazine/GLMF-097/Cluster-haute-disponibilite-avec-equilibrage-de-charge