Notes stockage CEPH

Voir :

https://www.ionos.fr/digitalguide/serveur/know-how/la-solution-ceph/

https://documentation.suse.com/ses/6/single-html/ses-admin/index.html#salt-adding-nodes

https://groupes.renater.fr/wiki/ceph/_media/public/2017-06_ecole_ete_ceph_-_lundi_mardi_fschaer.pdf

https://blog.ippon.fr/2020/01/27/ceph-technologie-de-stockage-haute-disponibilite-en-action/

http://www.aube.gouv.fr/content/download/18662/124383/file/ceph.pdf

http://xstra.u-strasbg.fr/lib/exe/fetch.php?media=stockage:presentation_ceph_geiger_20171005.pdf

https://docs.ceph.com/docs/master/start/intro/

https://linuxfr.org/news/installation-de-ceph-jewel-sous-debian-jessie

https://ceph.com/install/

https://www.lemagit.fr/conseil/Comprendre-les-bases-de-larchitecture-de-Ceph

https://connect.ed-diamond.com/GNU-Linux-Magazine/GLMF-179/Presentation-et-installation-du-systeme-de-stockage-reparti-Ceph

https://www.objectif-libre.com/wp-content/uploads/2020/04/Plan-formation-Cloud-Ceph-1.pdf

Voir aussi :

SeaweedFS

ceph status ceph-deploy admin serveur

30 TB30 TB30 TB——-- 90 TB

DFS100 TBSMBWindows DFS

Lexique

OSD (pour Object Storage Daemon), un disque

Etapes :Monter un CEPHMonter un Samba

DFS
Droits lectures pour tout auth

Synchro les données

U:\Services\Direction des Etudes Gty\Modification\Inventaire DE-SdT\Inventaire 2020

http://people.redhat.com/bhubbard/nature/nature-new/glossary/#term-node

Prérequis

MatérielVoir : https://docs.ceph.com/en/latest/start/hardware-recommendations/

Logiciel ;http://people.redhat.com/bhubbard/nature/nature-new/start/quick-start-preflight/

Liste :

Réseau (Ceph préconise l’utilisation de 2 interfaces réseau)

/etc/hosts

NTP

Ceph deploy user (with passwordless sudo privileges)

SSH passwordless

Sudo tty (Si requiretty ⇒ Defaults:ceph !requiretty)

SELinux

Composants

MDS (Meta Data Server)

Consomme du CPU et de la RAM (1 Gi de mémoire par instance). Utile que si l'on planifie d'utiliser CephFS.

Monitors

For small clusters, 1-2 GB is generally sufficient

OSD (Object Storage Daemon)

Du côté de la mémoire, 512 Mi par instance sont suffisants sauf lors de la récupération où 1 Gi de mémoire par Ti de données et par instance est conseillé.

Installation

Voir :

https://docs.ceph.com/en/latest/install/manual-deployment/

https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/installation_guide_for_ubuntu/storage_cluster_installation

https://dokk.org/documentation/ceph/v0.91/start/quick-ceph-deploy/

Les users CEPH ne doivent pas être des utilisateurs standard, mais des utilisateurs de services eux même chargés d’une gestion fine des droitsCeph préconise l’utilisation de 2 interfaces réseau

echo "deb http://ftp.debian.org/debian buster-backports main" >> /etc/apt/sources.list.d/backports.list apt-get update apt-get install -t buster-backports ceph
zcat /usr/share/doc/ceph/sample.ceph.conf.gz > /etc/ceph/ceph.conf

# uuidgen
67274814-239f-4a05-8415-ed04df45876c

[global] ### http://docs.ceph.com/docs/master/rados/configuration/general-config-ref/ fsid = 67274814-239f-4a05-8415-ed04df45876c # use `uuidgen` to generate your own UUID public network = 192.168.56.0/24 cluster network = 192.168.56.0/24 # Replication level, number of data copies. # Type: 32-bit Integer # (Default: 3) osd pool default size = 2 ## Replication level in degraded state, less than 'osd pool default size' value. # Sets the minimum number of written replicas for objects in the # pool in order to acknowledge a write operation to the client. If # minimum is not met, Ceph will not acknowledge the write to the # client. This setting ensures a minimum number of replicas when # operating in degraded mode. # Type: 32-bit Integer # (Default: 0), which means no particular minimum. If 0, minimum is size - (size / 2). ;osd pool default min size = 2 osd pool default min size = 1 [mon] ### http://docs.ceph.com/docs/master/rados/configuration/mon-config-ref/ ### http://docs.ceph.com/docs/master/rados/configuration/mon-osd-interaction/ # The IDs of initial monitors in a cluster during startup. # If specified, Ceph requires an odd number of monitors to form an # initial quorum (e.g., 3). # Type: String # (Default: None) mon initial members = kub1,kub2,kub3 [mon.kub1] host = kub1 mon addr = 192.168.56.21:6789 [mon.kub2] host = kub2 mon addr = 192.168.56.22:6789 [mon.kub3] host = kub3 mon addr = 192.168.56.23:6789
ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'
[client.admin] key = AQBGuWRfchSlDRAA3/bTmiPTLLN0w4JdVOxpDQ== caps mds = "allow" caps mon = "allow *" caps osd = "allow *"
ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *' ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
systemctl enable ceph.target systemctl start ceph.target systemctl enable ceph-mon@$(hostname -s) systemctl start ceph-mon@$(hostname -s) systemctl status ceph-mon@$(hostname -s).service

# ceph health detail
HEALTH_WARN 3 monitors have not enabled msgr2
MON_MSGR2_NOT_ENABLED 3 monitors have not enabled msgr2
mon.kub1 is not bound to a msgr2 port, only v1:192.168.56.21:6789/0
mon.kub2 is not bound to a msgr2 port, only v1:192.168.56.22:6789/0
mon.kub3 is not bound to a msgr2 port, only v1:192.168.56.23:6789/0
# ceph mon enable-msgr2
# ceph health detail
HEALTH_OK

OSD

Voir : https://wiki.nix-pro.com/view/CEPH_deployment_guide

ceph-volume remplace ceph-disk

ceph-volume inventory ceph-volume inventory /dev/sdb ceph-volume lvm batch --bluestore /dev/sda /dev/sdb /dev/sdc CEPH_VOLUME_DEBUG=1 ceph-volume inventory /dev/sdb ceph-volume lvm zap /dev/sdb --destroy
ceph-osd -i 0 --mkfs --mkkey --osd-uuid 13b2da5a-033f-4d58-b106-2f0212df6438 chown -R ceph:ceph /var/lib/ceph ceph auth list ceph auth add osd.0 osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-0/keyring
[osd.0] key = AQBizGxGhJcwJxAAHhOGHXQuCUTktxNszj62aQ==
ceph --cluster ceph osd crush add-bucket kub1 host ceph osd crush move kub1 root=default chown -R ceph:ceph /var/lib/ceph ceph --cluster ceph osd crush add osd.0 1.0 host=kub1 ceph-volume raw prepare --bluestore --data /dev/sdb1 systemctl start ceph-osd@1

CephFS

cd /etc/ceph sudo mkcephfs -a -c /etc/ceph/ceph.conf -k ceph.keyring

Administration

Ceph health

ceph -s ceph mon_status -f json-pretty ceph -w ceph df ceph health detail ceph -n client.admin --keyring=/etc/ceph/ceph.client.admin.keyring health ceph pg dump ceph pg X.Y query ceph pgdump_stuck inactive

OSD

ceph osd tree watch ceph osd pool stats ceph osd map

Supression OSD

ceph osd crush reweight osd.XX 0. # Passage du poids de l’OSD à 0 ceph osd out XX # Marquage de l’OSD comme non disponible au cluster # 1er mouvement de données, ~10To rebalancés #stop ceph-osd id=XX systemctl stop ceph-osd@XX.service # arrêt de l'exécution de l’OSD sur le serveur ceph osd crush remove osd.XX # Sortie logique de l’OSD du cluster # 2nd mouvement de données (non prévu), ~10To rebalancés ceph auth del osd.{osd-num} # suppression des clés d’authentification de l’OSD au cluster ceph osd rm {osd-num} # suppression définitive de l’OSD du cluster #ceph-volume lvm zap /dev/sdb --destroy

Autres

ceph mgr module l ceph mgr module enable plop

Client

Voir :

https://manjusri.ucsc.edu/2017/08/30/luminous-on-pulpos/

https://manjusri.ucsc.edu/2017/08/21/kraken-on-pulpos/

mount -t ceph 128.114.86.4:6789:/ /mnt/pulpos -o name=admin,secretfile=/etc/ceph/admin.secret

128.114.86.4:6789,128.114.86.5:6789,128.114.86.2:6789:/ /mnt/pulpos ceph name=admin,secretfile=/etc/ceph/admin.secret,noatime,_netdev 0 2

ceph-fuse -m 128.114.86.4:6789 /mnt/pulpos