Uncategorized

Install Ceph 15 (Octopus) Storage Cluster on Ubuntu 20.04

- by jason
By
Josphat Mutai
–
11657
1
 

I’ve been planning on putting together an article covering installation of Ceph Storage Cluster on Ubuntu 20.04 Linux servers and this is the delivery day. Ceph is software defined storage solution designed for building distributed storage clusters on commodity hardware. The requirements for building Ceph Storage Cluster on Ubuntu 20.04 will depend largely on the desired use case.

This setup is not for running mission critical intense write applications. You may need to consult official project documentation for such requirements especially on networking and storage hardware. Below are the standard Ceph components that will be configured in this installation guide:

  • Ceph MON – Monitor Server
  • Ceph MDS – Metadata Servers
  • Ceph MGR – Ceph Manager daemon
  • Ceph OSDs – The object storage daemons

Install Ceph Storage Cluster on Ubuntu 20.04

Before you begin the deployment of Ceph Storage Cluster on Ubuntu 20.04 Linux servers you need to prepare the servers needed. Below is a picture of my servers ready for setup.

 
prepare ceph servers

As seen in the picture, my Lab have the following server names and IP addresses.

Server Hostname Server IP Address Ceph components Server Specs
ceph-mon-01 172.16.20.10 Ceph MON, MGR,MDS 8gb ram, 4vpcus
ceph-mon-02 172.16.20.11 Ceph MON, MGR,MDS 8gb ram, 4vpcus
ceph-mon-03 172.16.20.12 Ceph MON, MGR,MDS 8gb ram, 4vpcus
ceph-osd-01 172.16.20.13 Ceph OSD 16gb ram, 8vpcus
ceph-osd-02 172.16.20.14 Ceph OSD 16gb ram, 8vpcus
ceph-osd-03 172.16.20.15 Ceph OSD 16gb ram, 8vpcus

Step 1: Prepare first Monitor node

The ceph component used for deployment is Cephadm. Cephadm deploys and manages a Ceph cluster by connection to hosts from the manager daemon via SSH to add, remove, or update Ceph daemon containers.

Login to your first Monitor node:

$ ssh root@ceph-mon-01 
Warning: Permanently added 'ceph-mon-01,172.16.20.10' (ECDSA) to the list of known hosts.
Enter passphrase for key '/var/home/jkmutai/.ssh/id_rsa': 
Welcome to Ubuntu 20.04 LTS (GNU/Linux 5.4.0-33-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

Last login: Tue Jun  2 20:36:36 2020 from 172.16.20.10
root@ceph-mon-01:~#

Update /etc/hosts file with the entries for all the IP addresses and hostnames.

# vim /etc/hosts

127.0.0.1 localhost

# Ceph nodes
172.16.20.10  ceph-mon-01
172.16.20.11  ceph-mon-02
172.16.20.12  ceph-mon-03
172.16.20.13  ceph-osd-01
172.16.20.14  ceph-osd-02
172.16.20.15  ceph-osd-03

Update and upgrade OS:

sudo apt update && sudo apt -y upgrade
sudo systemctl reboot

Install Ansible and other basic utilities:

sudo apt update
sudo apt -y install software-properties-common git curl vim bash-completion ansible

Confirm Ansible has been installed.

$ ansible --version
ansible 2.9.6
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.8.2 (default, Apr 27 2020, 15:53:34) [GCC 9.3.0]

Ensure /usr/local/bin path is added to PATH.

echo "PATH=\$PATH:/usr/local/bin" >>~/.bashrc
source ~/.bashrc

Check your current PATH:

$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/bin

Generate SSH keys:

$ ssh-keygen -t rsa -b 4096 -N '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:3gGoZCVsA6jbnBuMIpnJilCiblaM9qc5Xk38V7lfJ6U root@ceph-admin
The key's randomart image is:
+---[RSA 4096]----+
| ..o. . |
|. +o . |
|. .o.. . |
|o .o .. . . |
|o%o.. oS . o .|
|@+*o o… .. .o |
|O oo . ….. .E o|
|o+.oo. . ..o|
|o .++ . |
+----[SHA256]-----+

Install Cephadm:

curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
chmod +x cephadm
sudo mv cephadm  /usr/local/bin/

Confirm cephadm is available for use locally:

$ cephadm --help

Step 2: Update all Ceph nodes and push ssh public key

With the first Mon node configured, create an ansible playbook to update all nodes, and push ssh public key and update /etc/hosts file in all nodes.

cd ~/
vim prepare-ceph-nodes.yml

Modify below contents to set correct timezone and add to the file.

---
- name: Prepare ceph nodes
  hosts: ceph_nodes
  become: yes
  become_method: sudo
  vars:
    ceph_admin_user: cephadmin
  tasks:
    - name: Set timezone
      timezone:
        name: Africa/Nairobi

    - name: Update system
      apt:
        name: "*"
        state: latest
        update_cache: yes

    - name: Install common packages
      apt:
        name: [vim,git,bash-completion,wget,curl,chrony]
        state: present
        update_cache: yes

    - name: Set authorized key taken from file to root user
      authorized_key:
        user: root
        state: present
        key: "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
   
    - name: Install Docker
      shell: |
        curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
        echo "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" > /etc/apt/sources.list.d/docker-ce.list
        apt update
        apt install -qq -y docker-ce docker-ce-cli containerd.io

    - name: Reboot server after update and configs
      reboot:

Create inventory file.

$ vim hosts
[ceph_nodes]
ceph-mon-01
ceph-mon-02
ceph-mon-03
ceph-osd-01
ceph-osd-02
ceph-osd-03

Save key passphrase if you use one.

$ eval `ssh-agent -s` && ssh-add ~/.ssh/id_rsa_jmutai 
Agent pid 3275
Enter passphrase for /root/.ssh/id_rsa_jmutai: 
Identity added: /root/.ssh/id_rsa_jkmutai (/root/.ssh/id_rsa_jmutai)

Configure ssh:

tee -a ~/.ssh/config<<EOF
Host *
    UserKnownHostsFile /dev/null
    StrictHostKeyChecking no
    IdentitiesOnly yes
    ConnectTimeout 0
    ServerAliveInterval 300
EOF

Execute Playbook:

# As root user with  default ssh key:
$ ansible-playbook -i hosts prepare-ceph-nodes.yml --user root

# As root user with password:
$ ansible-playbook -i hosts prepare-ceph-nodes.yml --user root --ask-pass

# As sudo user with password - replace ubuntu with correct username
$ ansible-playbook -i hosts prepare-ceph-nodes.yml --user ubuntu --ask-pass --ask-become-pass

# As sudo user with ssh key and sudo password - replace ubuntu with correct username
$ ansible-playbook -i hosts prepare-ceph-nodes.yml --user ubuntu --ask-become-pass

# As sudo user with ssh key and passwordless sudo - replace ubuntu with correct username
$ ansible-playbook -i hosts prepare-ceph-nodes.yml --user ubuntu --ask-become-pass

# As sudo or root user with custom key
$ ansible-playbook -i hosts prepare-ceph-nodes.yml --private-key /path/to/private/key <options>

In my case I’ll run:

$ ansible-playbook -i hosts prepare-ceph-nodes.yml --private-key ~/.ssh/id_rsa_jkmutai

Execution output:

 ansible-playbook -i hosts prepare-ceph-nodes.yml --private-key ~/.ssh/id_rsa_jmutai 

PLAY [Prepare ceph nodes] ******************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************
ok: [ceph-mon-03]
ok: [ceph-mon-02]
ok: [ceph-mon-01]
ok: [ceph-osd-01]
ok: [ceph-osd-02]
ok: [ceph-osd-03]

TASK [Update system] ***********************************************************************************************************************************
changed: [ceph-mon-01]
changed: [ceph-mon-02]
changed: [ceph-mon-03]
changed: [ceph-osd-02]
changed: [ceph-osd-01]
changed: [ceph-osd-03]

TASK [Install common packages] *************************************************************************************************************************
changed: [ceph-mon-02]
changed: [ceph-mon-01]
changed: [ceph-osd-02]
changed: [ceph-osd-01]
changed: [ceph-mon-03]
changed: [ceph-osd-03]

TASK [Add ceph admin user] *****************************************************************************************************************************
changed: [ceph-osd-02]
changed: [ceph-mon-02]
changed: [ceph-mon-01]
changed: [ceph-mon-03]
changed: [ceph-osd-01]
changed: [ceph-osd-03]

TASK [Create sudo file] ********************************************************************************************************************************
changed: [ceph-mon-02]
changed: [ceph-osd-02]
changed: [ceph-mon-01]
changed: [ceph-osd-01]
changed: [ceph-mon-03]
changed: [ceph-osd-03]

TASK [Give ceph admin user passwordless sudo] **********************************************************************************************************
changed: [ceph-mon-02]
changed: [ceph-mon-01]
changed: [ceph-osd-02]
changed: [ceph-osd-01]
changed: [ceph-mon-03]
changed: [ceph-osd-03]

TASK [Set authorized key taken from file to ceph admin] ************************************************************************************************
changed: [ceph-mon-01]
changed: [ceph-osd-01]
changed: [ceph-mon-03]
changed: [ceph-osd-02]
changed: [ceph-mon-02]
changed: [ceph-osd-03]

TASK [Set authorized key taken from file to root user] *************************************************************************************************
changed: [ceph-mon-01]
changed: [ceph-mon-02]
changed: [ceph-mon-03]
changed: [ceph-osd-01]
changed: [ceph-osd-02]
changed: [ceph-osd-03]

TASK [Install Docker] **********************************************************************************************************************************
changed: [ceph-mon-01]
changed: [ceph-mon-02]
changed: [ceph-osd-02]
changed: [ceph-osd-01]
changed: [ceph-mon-03]
changed: [ceph-osd-03]

TASK [Reboot server after update and configs] **********************************************************************************************************
changed: [ceph-osd-01]
changed: [ceph-mon-02]
changed: [ceph-osd-02]
changed: [ceph-mon-01]
changed: [ceph-mon-03]
changed: [ceph-osd-03]

PLAY RECAP *********************************************************************************************************************************************
ceph-mon-01                : ok=9    changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
ceph-mon-02                : ok=9    changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
ceph-mon-03                : ok=9    changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
ceph-osd-01                : ok=9    changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
ceph-osd-02                : ok=9    changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
ceph-osd-03                : ok=9    changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Test ssh as Ceph admin user created on the nodes:

$ ssh cephadmin@ceph-mon-02
Warning: Permanently added 'ceph-mon-02,172.16.20.11' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 20.04 LTS (GNU/Linux 5.4.0-28-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage


The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.

cephadmin@ceph-mon-01:~$ sudo su -
root@ceph-mon-01:~# logout

cephadmin@ceph-mon-01:~$ exit
logout
Connection to ceph-mon-01 closed.

Configure /etc/hosts

Update /etc/hosts on all nodes if you don’t active DNS configured for hostnames on all cluster servers.

Here is the playbook to modify:

$ vim update-hosts.yml
---
- name: Prepare ceph nodes
  hosts: ceph_nodes
  become: yes
  become_method: sudo
  tasks:
    - name: Clean /etc/hosts file
      copy:
        content: ""
        dest: /etc/hosts

    - name: Update /etc/hosts file
      blockinfile:
        path: /etc/hosts
        block: |
           127.0.0.1     localhost
           172.16.20.10  ceph-mon-01
           172.16.20.11  ceph-mon-02
           172.16.20.12  ceph-mon-03
           172.16.20.13  ceph-osd-01
           172.16.20.14  ceph-osd-02
           172.16.20.15  ceph-osd-03

Running playbook:

$ ansible-playbook -i hosts update-hosts.yml --private-key ~/.ssh/id_rsa_jmutai 

PLAY [Prepare ceph nodes] ******************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************
ok: [ceph-mon-01]
ok: [ceph-osd-02]
ok: [ceph-mon-03]
ok: [ceph-mon-02]
ok: [ceph-osd-01]
ok: [ceph-osd-03]

TASK [Clean /etc/hosts file] ***************************************************************************************************************************
changed: [ceph-mon-02]
changed: [ceph-mon-01]
changed: [ceph-osd-01]
changed: [ceph-osd-02]
changed: [ceph-mon-03]
changed: [ceph-osd-03]

TASK [Update /etc/hosts file] **************************************************************************************************************************
changed: [ceph-mon-02]
changed: [ceph-mon-01]
changed: [ceph-osd-01]
changed: [ceph-osd-02]
changed: [ceph-mon-03]
changed: [ceph-osd-03]

PLAY RECAP *********************************************************************************************************************************************
ceph-mon-01                : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
ceph-mon-02                : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
ceph-mon-03                : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
ceph-osd-01                : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
ceph-osd-02                : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
ceph-osd-03                : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Confirm:

$ ssh cephadmin@ceph-mon-01
$ cat /etc/hosts
# BEGIN ANSIBLE MANAGED BLOCK
127.0.0.1      localhost
172.16.20.10   ceph-mon-01
172.16.20.11   ceph-mon-02
172.16.20.12   ceph-mon-03
172.16.20.13   ceph-osd-01
172.16.20.14   ceph-osd-02
172.16.20.15   ceph-osd-03
# END ANSIBLE MANAGED BLOCK

Step 3: Deploy Ceph 15 (Octopus) Storage Cluster on Ubuntu 20.04

To bootstrap a new Ceph Cluster on Ubuntu 20.04, you need the first monitor address – IP or hostname.

sudo mkdir -p /etc/ceph
cephadm bootstrap \
  --mon-ip ceph-mon-01 \
  --initial-dashboard-user admin \
  --initial-dashboard-password Str0ngAdminP@ssw0rd

Execution output:

INFO:cephadm:Verifying podman|docker is present...
INFO:cephadm:Verifying lvm2 is present...
INFO:cephadm:Verifying time synchronization is in place...
INFO:cephadm:Unit chrony.service is enabled and running
INFO:cephadm:Repeating the final host check...
INFO:cephadm:podman|docker (/usr/bin/docker) is present
INFO:cephadm:systemctl is present
INFO:cephadm:lvcreate is present
INFO:cephadm:Unit chrony.service is enabled and running
INFO:cephadm:Host looks OK
INFO:root:Cluster fsid: 8dbf2eda-a513-11ea-a3c1-a534e03850ee
INFO:cephadm:Verifying IP 172.16.20.10 port 3300 ...
INFO:cephadm:Verifying IP 172.16.20.10 port 6789 ...
INFO:cephadm:Mon IP 172.16.20.10 is in CIDR network 172.31.1.1
INFO:cephadm:Pulling latest docker.io/ceph/ceph:v15 container...
INFO:cephadm:Extracting ceph user uid/gid from container image...
INFO:cephadm:Creating initial keys...
INFO:cephadm:Creating initial monmap...
INFO:cephadm:Creating mon...
INFO:cephadm:Waiting for mon to start...
INFO:cephadm:Waiting for mon...
INFO:cephadm:mon is available
INFO:cephadm:Assimilating anything we can from ceph.conf...
INFO:cephadm:Generating new minimal ceph.conf...
INFO:cephadm:Restarting the monitor...
INFO:cephadm:Setting mon public_network...
INFO:cephadm:Creating mgr...
INFO:cephadm:Wrote keyring to /etc/ceph/ceph.client.admin.keyring
INFO:cephadm:Wrote config to /etc/ceph/ceph.conf
INFO:cephadm:Waiting for mgr to start...
INFO:cephadm:Waiting for mgr...
INFO:cephadm:mgr not available, waiting (1/10)...
INFO:cephadm:mgr not available, waiting (2/10)...
INFO:cephadm:mgr not available, waiting (3/10)...
INFO:cephadm:mgr not available, waiting (4/10)...
INFO:cephadm:mgr is available
INFO:cephadm:Enabling cephadm module...
INFO:cephadm:Waiting for the mgr to restart...
INFO:cephadm:Waiting for Mgr epoch 5...
INFO:cephadm:Mgr epoch 5 is available
INFO:cephadm:Setting orchestrator backend to cephadm...
INFO:cephadm:Generating ssh key...
INFO:cephadm:Wrote public SSH key to to /etc/ceph/ceph.pub
INFO:cephadm:Adding key to root@localhost's authorized_keys...
INFO:cephadm:Adding host ceph-mon-01...
INFO:cephadm:Deploying mon service with default placement...
INFO:cephadm:Deploying mgr service with default placement...
INFO:cephadm:Deploying crash service with default placement...
INFO:cephadm:Enabling mgr prometheus module...
INFO:cephadm:Deploying prometheus service with default placement...
INFO:cephadm:Deploying grafana service with default placement...
INFO:cephadm:Deploying node-exporter service with default placement...
INFO:cephadm:Deploying alertmanager service with default placement...
INFO:cephadm:Enabling the dashboard module...
INFO:cephadm:Waiting for the mgr to restart...
INFO:cephadm:Waiting for Mgr epoch 13...
INFO:cephadm:Mgr epoch 13 is available
INFO:cephadm:Generating a dashboard self-signed certificate...
INFO:cephadm:Creating initial admin user...
INFO:cephadm:Fetching dashboard port number...
INFO:cephadm:Ceph Dashboard is now available at:

	     URL: https://ceph-mon-01:8443/
	    User: admin
	Password: Str0ngAdminP@ssw0rd

INFO:cephadm:You can access the Ceph CLI with:

	sudo /usr/local/bin/cephadm shell --fsid 8dbf2eda-a513-11ea-a3c1-a534e03850ee -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring

INFO:cephadm:Please consider enabling telemetry to help improve Ceph:

	ceph telemetry on

For more information see:

	https://docs.ceph.com/docs/master/mgr/telemetry/

INFO:cephadm:Bootstrap complete.

Install Ceph tools.

cephadm add-repo --release octopus
cephadm install ceph-common

Add extra monitors if you have them.

--- Copy Ceph SSH key ---
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-mon-02
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-mon-03

--- Label the nodes with mon ---
ceph orch host label add ceph-mon-01 mon
ceph orch host label add ceph-mon-02 mon
ceph orch host label add ceph-mon-03 mon

--- Add nodes to the cluster ---
ceph orch host add ceph-mon-02
ceph orch host add ceph-mon-03

--- Apply configs ---
ceph orch apply mon ceph-mon-02
ceph orch apply mon ceph-mon-03

View a list of hosts and labels.

# ceph orch host ls

HOST         ADDR         LABELS  STATUS  
ceph-mon-01  ceph-mon-01  mon             
ceph-mon-02  ceph-mon-02  mon             
ceph-mon-03  ceph-mon-03  mon

Containers running:

# docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS               NAMES
7d666ae63232        prom/alertmanager          "/bin/alertmanager -…"   3 minutes ago       Up 3 minutes                            ceph-8dbf2eda-a513-11ea-a3c1-a534e03850ee-alertmanager.ceph-mon-01
4e7ccde697c7        prom/prometheus:latest     "/bin/prometheus --c…"   3 minutes ago       Up 3 minutes                            ceph-8dbf2eda-a513-11ea-a3c1-a534e03850ee-prometheus.ceph-mon-01
9fe169a3f2dc        ceph/ceph-grafana:latest   "/bin/sh -c 'grafana…"   8 minutes ago       Up 8 minutes                            ceph-8dbf2eda-a513-11ea-a3c1-a534e03850ee-grafana.ceph-mon-01
c8e99deb55a4        prom/node-exporter         "/bin/node_exporter …"   8 minutes ago       Up 8 minutes                            ceph-8dbf2eda-a513-11ea-a3c1-a534e03850ee-node-exporter.ceph-mon-01
277f0ef7dd9d        ceph/ceph:v15              "/usr/bin/ceph-crash…"   9 minutes ago       Up 9 minutes                            ceph-8dbf2eda-a513-11ea-a3c1-a534e03850ee-crash.ceph-mon-01
9de7a86857aa        ceph/ceph:v15              "/usr/bin/ceph-mgr -…"   10 minutes ago      Up 10 minutes                           ceph-8dbf2eda-a513-11ea-a3c1-a534e03850ee-mgr.ceph-mon-01.qhokxo
d116bc14109c        ceph/ceph:v15              "/usr/bin/ceph-mon -…"   10 minutes ago      Up 10 minutes                           ceph-8dbf2eda-a513-11ea-a3c1-a534e03850ee-mon.ceph-mon-01

Step 4: Deploy Ceph OSDs

Install the cluster’s public SSH key in the new OSD nodes root user’s authorized_keys file:

ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-osd-01
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-osd-02
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-osd-03

Tell Ceph that the new node is part of the cluster:

--- Add hosts to cluster ---
ceph orch host add ceph-osd-01
ceph orch host add ceph-osd-02
ceph orch host add ceph-osd-03

--- Give new nodes labels ---

ceph orch host label add  ceph-osd-01 osd
ceph orch host label add  ceph-osd-02 osd
ceph orch host label add  ceph-osd-03 osd

View all devices on storage nodes:

# ceph orch device ls
HOST         PATH      TYPE   SIZE  DEVICE                           AVAIL  REJECT REASONS  
ceph-mon-01  /dev/sda  hdd   76.2G  QEMU_HARDDISK_drive-scsi0-0-0-0  False  locked          
ceph-mon-02  /dev/sda  hdd   76.2G  QEMU_HARDDISK_drive-scsi0-0-0-0  False  locked          
ceph-mon-03  /dev/sda  hdd   76.2G  QEMU_HARDDISK_drive-scsi0-0-0-0  False  locked          
ceph-osd-01  /dev/sdb  hdd   50.0G  HC_Volume_5680482                True                   
ceph-osd-01  /dev/sda  hdd   76.2G  QEMU_HARDDISK_drive-scsi0-0-0-0  False  locked          
ceph-osd-02  /dev/sdb  hdd   50.0G  HC_Volume_5680484                True                   
ceph-osd-02  /dev/sda  hdd   76.2G  QEMU_HARDDISK_drive-scsi0-0-0-0  False  locked          
ceph-osd-03  /dev/sdb  hdd   50.0G  HC_Volume_5680483                True                   
ceph-osd-03  /dev/sda  hdd   76.2G  QEMU_HARDDISK_drive-scsi0-0-0-0  False  locked

A storage device is considered available if all of the following conditions are met:

  • The device must have no partitions.
  • The device must not have any LVM state.
  • The device must not be mounted.
  • The device must not contain a file system.
  • The device must not contain a Ceph BlueStore OSD.
  • The device must be larger than 5 GB.

Tell Ceph to consume any available and unused storage device:

# ceph orch daemon add osd ceph-osd-01:/dev/sdb
Created osd(s) 0 on host 'ceph-osd-01'

# ceph orch daemon add osd ceph-osd-02:/dev/sdb
Created osd(s) 1 on host 'ceph-osd-02'

# ceph orch daemon add osd ceph-osd-03:/dev/sdb
Created osd(s) 1 on host 'ceph-osd-03'

Check ceph status:

# ceph -s
  cluster:
    id:     8dbf2eda-a513-11ea-a3c1-a534e03850ee
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum ceph-mon-01 (age 23m)
    mgr: ceph-mon-01.qhokxo(active, since 22m), standbys: ceph-mon-03.rhhvzc
    osd: 3 osds: 3 up (since 36s), 3 in (since 36s)
 
  data:
    pools:   1 pools, 1 pgs
    objects: 1 objects, 0 B
    usage:   3.0 GiB used, 147 GiB / 150 GiB avail
    pgs:     1 active+clean

Step 5: Access Ceph Dashboard

Ceph Dashboard is now available at the Address of active MGR server.

# ceph -s

For the this will be:

URL: https://ceph-mon-01:8443/
User: admin
Password: Str0ngAdminP@ssw0rd

Login with the credentials to access Ceph management dashboard.

ceph login

Enjoy management of Ceph Storage Cluster on Ubuntu 20.04 using Cephadm and Containers. Our next articles will cover on adding additional OSDs, removing them, configuring RGW e.t.c. Stay connected for updates.

Install Ceph 15 (Octopus) Storage Cluster on Ubuntu 20.04

 

Related Posts

compare two directory trees with rsync

linux man ip command

CentOS7 Change repo end of life

About jason

View all posts by jason →