{{tag>Brouillon}} # Haute dispo cluster failover redhat Voir aussi : * OpenSVC * Paquet resource-agents Ressources : * myvip * fence_node-1 * fence_node-2 * ping * srvweb * ClusterMon-External Liens intros : * https://www.youtube.com/watch?v=Xl0TQer9uZc * https://www.youtube.com/watch?v=RbJwluN_qrY ## Installation Voir : * http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Clusters_from_Scratch/ * http://clusterlabs.org/doc/en-US/Pacemaker/ ### Prérequis Prérequis * Date syncho * SELinux désactivé * service NetworkManager arrêté * Règles pare-feu * Conf ''/etc/hosts'' #### Date synchro (ntp) Les nœuds doivent avoir la date et l'heure synchronisée (voir NTP) Vérif ~~~bash date ~~~ Exemple avec Clush [[cluster_shell_parallele]] ~~~bash echo date |clush -B -w node-[1-2] ~~~ #### SELinux désactivé ~~~bash setenforce 0 sed -i.bak "s/SELINUX=enforcing/SELINUX=permissive/g" /etc/selinux/config ~~~ Vérif ~~~bash sestatus ~~~ #### Service NetworkManager arrêté et désactivé ~~~bash systemctl stop NetworkManager systemctl disable NetworkManager ~~~ #### Pare-feu Si pare-feu activé ~~~bash firewall-cmd --permanent --add-service=high-availability firewall-cmd --add-service=high-availability ~~~ Ou Désactivation du parefeux ~~~bash systemctl stop firewalld systemctl disable firewalld #rpm -e firewalld ~~~ Vérif ~~~bash iptables -L -n -v ~~~ #### Résolution noms Chaque nœud doit pouvoir pinguer les autres via son nom. Il est conseiller d'utiliser ''/etc/hosts'' plutôt que DNS. ''/etc/hosts'' ~~~ 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.1.1 node-1.localdomain 192.168.97.221 node-1.localdomain node-1 192.168.97.222 node-2.localdomain node-2 ~~~ ### Install Install paquets ~~~bash yum install -y pacemaker pcs psmisc policycoreutils-python ~~~ ~~~bash echo "P@ssw0rd" | passwd hacluster --stdin systemctl start pcsd.service systemctl enable pcsd.service #unset http_proxy #export NO_PROXY=localhost,127.0.0.1,node-1,node-2 pcs cluster auth node-1 node-2 #-u hacluster -p passwd #pcs cluster setup --start --name my_cluster node-1 node-2 pcs cluster setup --name my_cluster node-1 node-2 pcs cluster start --all pcs cluster enable --all ~~~ Le fichier corosync.conf est automatiquement crée ''/etc/corosync/corosync.conf'' ~~~c totem { version: 2 secauth: off cluster_name: my_cluster transport: udpu } node-list { node- { ring0_addr: node-1 node-id: 1 } node- { ring0_addr: node-2 node-id: 2 } } quorum { provider: corosync_votequorum two_node-: 1 } logging { to_logfile: yes logfile: /var/log/cluster/corosync.log to_syslog: yes } ~~~ Vérifier la conf de corosync 1 ~~~bash corosync-cfgtool -s ~~~ Doit retourner **no faults** \\ Ne doit pas comporter d’adresse 127.0.0.1 Vérifier la conf de corosync 2 ~~~bash corosync-cmapctl |grep members pcs status corosync ~~~ ### Configuration Prevent Resources from Moving after Recovery ~~~bash pcs resource defaults resource-stickiness=100 ~~~ Pas de quorum ~~~bash #pcs property set no-quorum-policy=ignore pcs property set no-quorum-policy=freeze ~~~ ## Configuration du fencing / stonith ### Test en vue du fencing via iDRAC Voir https://www.devops.zone/tricks/connecting-ssh-drac-reboot-server/ Tester du fencing ~~~bash /usr/sbin/fence_drac5 --ip=192.168.96.221 --username=root --password=calvin --ssh -c 'admin1->' ~~~ Test avec OpenManage ''/opt/dell/srvadmin/sbin/racadm'' ~~~bash racadm -r 192.168.96.221 -u root -p calvin get iDRAC.Info ~~~ Test via SSH sur iDRAC Pour redemarrer le serveur en se connectant en SSH sur la iDRAC ~~~ ssh root@192.168.96.221 racadm serveraction powercycle ~~~ ** Si pas de stonith / fence ** sinon la VIP refusera de démarrer ~~~bash # Si pas de stonith / fence pcs property set stonith-enabled=false ~~~ ### Vérif ~~~bash crm_verify -LVVV ~~~ ### Configuration ~~~bash # pcs stonith create fence_node-1 fence_drac5 ipaddr=192.168.96.221 login=root passwd=calvin secure=1 cmd_prompt="/admin1->" pcmk_host_list=node-1 stonith-action=poweroff pcs stonith create fence_node-1 fence_drac5 ipaddr=192.168.96.221 login=root passwd=calvin secure=1 cmd_prompt="/admin1->" pcmk_host_list=node-1 op monitor interval="60s" pcs stonith create fence_node-2 fence_drac5 ipaddr=192.168.96.222 login=root passwd=calvin secure=1 cmd_prompt="/admin1->" pcmk_host_list=node-2 op monitor interval="60s" pcs stonith level add 1 node-1 fence_node-1 pcs stonith level add 1 node-2 fence_node-2 ~~~ Interdire le suicide (le fencing de soi-même) ~~~bash pcs constraint location fence_node-1 avoids node-1 pcs constraint location fence_node-2 avoids node-2 ~~~ Tester le fencing ~~~bash #stonith_admin --reboot node-1 pcs stonith fence node-1 ~~~ ### Ajout ressources Ajout ressource VIP (adresse IP virtuelle) ~~~bash pcs resource create myvip IPaddr2 ip=192.168.97.230 cidr_netmask=24 nic=bond0 op monitor interval=30s on-fail=fence #pcs constraint location myvip prefers node-1=INFINITY pcs constraint location myvip prefers node-1=100 pcs constraint location myvip prefers node-2=50 #pcs resource meta myvip resource-stickiness=100 ~~~ Ajouter ressource ping ~~~bash pcs resource create ping ocf:pacemaker:ping dampen=5s multiplier=1000 host_list=192.168.97.250 --clone pcs constraint location myvip rule score=-INFINITY pingd lt 1 or not_defined pingd ~~~ Ajout ressource Apache Avant il faut configurer http://localhost/server-status et arrêter le service d'apache sur l'ensemble des nœuds ~~~bash curl http://localhost/server-status systemctl stop httpd.service systemctl disable httpd.service ~~~ ~~~bash pcs resource create srvweb apache configfile="/etc/httpd/conf/httpd.conf" statusurl="http://127.0.0.1/server-status" op monitor interval=1min #--clone # Le serveur Web toujours sur la VIP pcs constraint colocation add srvweb with myvip # D'abord la VIP puis le serveur Web pcs constraint order myvip then srvweb ~~~ ## Manip Déplacer la VIP ~~~bash pcs resource move myvip node-1 pcs resource move myvip node-2 ~~~ Retour arrière - Déplacer la VIP ~~~bash #pcs constraint --full |grep prefer pcs constraint remove cli-prefer-myvip pcs resource relocate run ~~~ Remise à zero compteur erreurs ~~~bash #pcs resource failcount reset res1 #crm_resource -P pcs resource cleanup ~~~ Déplacer toutes les ressources sur le nœud primaire (ignoring resource stickiness) ~~~bash #pcs resource relocate show pcs resource relocate run ~~~ Maintenance sur une ressource ~~~bash #pcs resource update fence_node-1 meta target-role=stopped #pcs resource update fence_node-1 meta is-managed=false #pcs resource update fence_node-1 op monitor enabled=false #pcs resource disable fence_node-1 pcs resource unmanage fence_node-1 ~~~ Maintenance générale du cluster ~~~bash pcs property set maintenance-mode=true ~~~ Fin de maintenance ~~~bash pcs property set maintenance-mode=false ~~~ Arrêt du cluster ~~~bash pcs cluster stop --all pcs cluster disable --all ~~~ ## Diagnostic / Supervision ### Diag Passif [[http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Clusters_from_Scratch/_verify_corosync_installation.html|Vérif corosync]] ~~~bash # Check syntax conf corosync -t # Check cluster communication corosync-cfgtool -s # check the node's network corosync-cmapctl |grep members ~~~ Vérif ~~~bash pcs cluster pcsd-status pcs cluster verify pcs status corosync crm_mon -1 --fail crm_mon -1Af journalctl --since yesterday -p err journalctl -u pacemaker.service --since "2017-02-24 16:00" -p warning ~~~ Script supervision (ces commandes doivent retourner aucune ligne) ~~~bash LANG=C pcs status |egrep "Stopped|standby|OFFLINE|UNCLEAN|Failed|error" crm_verify -LVVV LANG=C pcs resource relocate show |sed -ne '/Transition Summary:/,$p' |grep -v '^Transition Summary:' crm_mon -1f | grep -q fail-count ~~~ Voir plus haut si (script ''/usr/local/bin/crm_logger.sh'') ~~~bash tailf /var/log/messages |grep "ClusterMon-External:" ~~~ Script supervision Quel nœud est actif ~~~bash LANG=C crm_resource --resource myvip --locate |cut -d':' -f2 |tr -d ' ' ~~~ Le serveur web répond t-il bien en utilisant l'IP de la VIP. (Le code de retour doit-être **0**) ~~~bash #curl -4 -m 1 --connect-timeout 1 http://192.168.97.230/ > /dev/null 2>&1 curl -4 -m 1 --connect-timeout 1 http://192.168.97.230/cl.html > /dev/null 2>&1 #echo $? ~~~ ### ACL Compte en lecture seule avec les droits de consulter crm_mon \\ Attention : ce compte trouver le mdp iDRAC/Ilo ~~~bash pcs stonith --full |grep passwd ~~~ Mise en œuvre ~~~bash #adduser rouser #usermod -a -G haclient rouser usermod -a -G haclient process pcs property set enable-acl=true pcs acl role create read-only description="Read access to cluster" read xpath /cib #pcs acl user create rouser read-only pcs acl user create process read-only ~~~ ~~~bash #crm_mon --daemonize --as-html /var/www/html/cl.html ~~~ ''/usr/local/bin/crm_logger.sh'' ~~~bash #!/bin/sh # https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/pdf/High_Availability_Add-On_Reference/Red_Hat_Enterprise_Linux-7-High_Availability_Add-On_Reference-en-US.pdf logger -t "ClusterMon-External" "${CRM_notify_node:-x} ${CRM_notify_rsc:-x} \ ${CRM_notify_task:-x} ${CRM_notify_desc:-x} ${CRM_notify_rc:-x} \ ${CRM_notify_target_rc:-x} ${CRM_notify_status:-x} ${CRM_notify_recipient:-x}"; exit ~~~ ~~~bash chmod 755 /usr/local/bin/crm_logger.sh chown root.root /usr/local/bin/crm_logger.sh ~~~ ~~~bash pcs resource create ClusterMon-External ClusterMon update=10000 user=process extra_options="-E /usr/local/bin/crm_logger.sh --watch-fencing" htmlfile=/var/www/html/cl.html pidfile=/tmp/crm_mon-external.pid op monitor on-fail="restart" interval="60" clone ~~~ Colocation - page de monitoting toujours actif sur la VIP \\ **Seulement nécessaire si ressource non clonée** ~~~bash pcs constraint colocation add ClusterMon-External with myvip ~~~ Test ~~~bash curl 192.168.97.230/cl.html ~~~ Voir https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Reference/s1-eventnotification-HAAR.html ### Diag Actif En cas de pb ~~~bash pcs resource debug-start resource_id ~~~ ## Ajout 2em interface pour le heartbeat Redundant Ring Protocol (RRP) rrp_mode If set to active, Corosync uses both interfaces actively. If set to passive, Corosync sends messages alternatively over the available networks. Avant de modifier la conf, on passe le cluster en mode maintenance : ~~~bash pcs property set maintenance-mode=true ~~~ ''/etc/hosts'' ~~~ 192.168.21.10 node1 192.168.22.10 node1b 192.168.21.11 node2 192.168.22.11 node2b ~~~ On ajoute **rrp_mode** et **ring1_addr** ''/etc/corosync/corosync.conf'' ~~~c totem { rrp_mode: active } nodelist { node { ring0_addr: node1 ring1_addr: node1b nodeid: 1 } node { ring0_addr: node2 ring1_addr: node2b nodeid: 2 } } ~~~ ~~~bash pcs cluster reload corosync pcs cluster status corosync corosync-cfgtool -s pcs property unset maintenance-mode ~~~ ## Reprise sur incident ~~~bash #crm_resource -P pcs resource cleanup pcs resource relocate run #pcs cluster start --all ~~~ ## Crash-tests Test 1 Crash brutal ~~~bash echo 1 > /proc/sys/kernel/sysrq echo c > /proc/sysrq-trigger ~~~ Test 2 Coupure électrique : Débranchement du câble Test 3 Coupure réseaux ~~~bash ifdown bond0 ~~~ Test 4 Perte du ping de la passerelle sur l'un des nœud ~~~bash iptables -A OUTPUT -d 192.168.97.250/32 -p icmp -j REJECT ~~~ Test 5 Fork bomb, nœud ne répond plus, sauf au ping Fork bomb ~~~bash :(){ :|:& };: ~~~ Test 6 Perte connexion iDRAC : Débranchement du câble ## Nettoyage - effacer ~~~bash pcs cluster stop --force #--all pcs cluster destroy --force #--all systemctl stop pcsd systemctl stop corosync systemctl stop pacemaker yum remove -y pcsd corosync pacemaker userdel hacluster rm -rf /dev/shm/qb-*-data /dev/shm/qb-*-header rm -rf /etc/corosync rm -rf /var/lib/corosync rm -rf /var/lib/pcsd rm -rf /var/lib/pacemaker rm -rf /var/log/cluster/ rm -rf /var/log/pcsd/ rm -f /var/log/pacemaker.log* ~~~ ## Erreurs ### 1 Erreur Dell hardware ~~~ UEFI0081: Memory size has changed from the last time the system was started. No action is required if memory was added or removed. ~~~ http://www.dell.com/support/manuals/fr/fr/frbsdt1/integrated-dell-remote-access-cntrllr-8-with-lifecycle-controller-v2.00.00.00/eemi_13g-v1/UEFI-Event-Messages?guid=GUID-C1C6F253-F8EF-43BF-B8ED-1A9B2A910AC4&lang=en-us ### 2 Test fork-bomb ~~~ error: Integration Timer (I_INTEGRATED) just popped in state S_INTEGRATION! (180000ms) ~~~ ## Autres ### Pour voir / vérifier les "property" ~~~bash #pcs property set symmetric-cluster=true pcs property ~~~ ### Ressources Lister ~~~bash pcs resource standards ~~~ ~~~ ocf lsb service systemd stonith ~~~ ~~~bash pcs resource providers ~~~ ~~~ heartbeat openstack pacemaker ~~~ Lister les agents : Exemple ~~~bash pcs resource agents systemd pcs resource agents ocf:heartbeat ~~~ Timeout par défaut pour les ressources ~~~bash pcs resource op defaults timeout=240s ~~~ Stopper toutes les ressources ~~~bash pcs property set stop-all-resources=true ~~~ ~~~bash pcs property unset stop-all-resources ~~~ ocf:pacemaker:ping \\ ''/usr/lib/ocf/resource.d/pacemaker/ping'' ocf:heartbeat:apache \\ ''/usr/lib/ocf/resource.d/heartbeat/apache'' ~~~bash egrep '^#.*OCF_RESKEY' /usr/lib/ocf/resource.d/heartbeat/apache export OCF_ROOT=/usr/lib/ocf/ /usr/lib/ocf/resource.d/heartbeat/apache meta-data ~~~ Autre Lister toutes les ressources ~~~bash crm_resource --list ~~~ Dump CIB (Cluster Information Base) ~~~bash pcs cluster cib pcs cluster cib cib-dump.xml ~~~ Ajout d'une ressource service ~~~bash pcs resource create CRON systemd:crond #pcs resource op add CRON start interval=0s timeout=1800s ~~~ UPDATE ~~~bash pcs resource update ClusterMon-External htmlfile='/tmp/cl.html' ~~~ UNSET ~~~bash pcs resource update ClusterMon-External htmlfile= ~~~ ### Stonith ~~~bash pcs property list --all |grep stonith ~~~ Confirmer que le nœud est bien arrêté. \\ Attention, si ce n'est pas le cas risque de pb ~~~bash pcs stonith confirm node2 ~~~ ### Failcount ~~~bash crm_mon --failcounts pcs resource failcount show resource_id pcs resource failcount reset resource_id ~~~ Actualisation de l’état, et remise à zéro du "failcount" ~~~bash pcs resource cleanup resource_id ~~~ ### Install depuis zero ~~~bash echo "P@ssw0rd" |passwd hacluster --stdin systemctl start pcsd.service systemctl enable pcsd.service pcs cluster auth -u hacluster -p P@ssw0rd 8si-pms-pps-srv-1 8si-pms-pps-srv-2 pcs cluster setup --name my_cluster 8si-pms-pps-srv-1 8si-pms-pps-srv-2 pcs cluster start --all pcs cluster enable --all pcs resource defaults resource-stickiness=100 pcs property set no-quorum-policy=freeze pcs stonith create fence_8si-pms-pps-srv-1 fence_drac5 ipaddr=172.18.202.230 login=root passwd=calvin secure=1 cmd_prompt="/admin1->" pcmk_host_list=8si-pms-pps-srv-1 op monitor interval="60s" pcs stonith create fence_8si-pms-pps-srv-2 fence_drac5 ipaddr=172.18.202.231 login=root passwd=calvin secure=1 cmd_prompt="/admin1->" pcmk_host_list=8si-pms-pps-srv-2 op monitor interval="60s" pcs stonith level add 1 8si-pms-pps-srv-1 fence_8si-pms-pps-srv-1 pcs stonith level add 1 8si-pms-pps-srv-2 fence_8si-pms-pps-srv-2 pcs constraint location fence_8si-pms-pps-srv-1 avoids 8si-pms-pps-srv-1 pcs constraint location fence_8si-pms-pps-srv-2 avoids 8si-pms-pps-srv-2 pcs resource create myvip IPaddr2 ip=172.18.202.226 cidr_netmask=24 nic=bond0 op monitor interval=30s #on-fail=fence pcs constraint location myvip prefers 8si-pms-pps-srv-1=100 pcs constraint location myvip prefers 8si-pms-pps-srv-2=50 #pcs resource meta myvip resource-stickiness=60 # l'utilisateur process doit appartenir au groupe haclient #usermod -a -G haclient process pcs property set enable-acl=true pcs acl role create read-only description="Read access to cluster" read xpath /cib pcs acl user create process read-only pcs resource create ClusterMon-External ClusterMon update=10000 user=process extra_options="-E /usr/local/bin/crm_logger.sh --watch-fencing" htmlfile=/var/www/html/cl.html pidfile=/tmp/crm_mon-external.pid op monitor on-fail="restart" interval="60" clone pcs resource create appmgr systemd:appmgr pcs constraint colocation add appmgr with myvip ~~~ ------------------ Voir aussi : * Généralité ** https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Configuration_Example_-_Oracle_HA_on_Cluster_Suite/sample_2_node.html * keepalived ** http://tecadmin.net/ip-failover-setup-using-keepalived-on-centos-redhat-6/# * Ricci Luci Ccs cman **ccs, ricci and luci are deprecated** ** http://www.tecmint.com/cman-multi-node-cluster-setup-in-linux/ ** http://www.thegeekstuff.com/2015/03/linux-redhat-cluster * Css ** http://www.thegeekstuff.com/2015/03/redhat-cluster-failover/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+TheGeekStuff+ ** https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Cluster_Administration/s1-config-failover-domain-cli-CA.html * Autres ** https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Configuration_Example_-_Oracle_HA_on_Cluster_Suite/cold_failover.html ** http://www.cyberciti.biz/faq/rhel-centos-fedora-keepalived-lvs-cluster-configuration/ Fencing * https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Reference/ch-fencing-HAAR.html * https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Overview/s1-fencing-HAAO.html * http://www.linuxtopia.org/online_books/rhel6/rhel_6_cluster_admin/rhel_6_cluster_s1-config-failover-domain-cli-CA.html Cluster * https://www.guillaume-leduc.fr/lequilibrage-de-charge-sous-linux-avec-lvs-theorie.html * http://www.ultramonkey.org/papers/lvs_tutorial/html/ * http://connect.ed-diamond.com/GNU-Linux-Magazine/GLMF-097/Cluster-haute-disponibilite-avec-equilibrage-de-charge