Mandatory Assignment

• 梳理 ceph 的个组件及其功能

• 基于 ceph-deploy 部署 ceph 集群

• 梳理块存储，文件存储以及对象存储的使用场景

• 基于 ceph 块存储实现块设备挂载及使用

• 基于 cephFS 实现多主机数据共享

What is ceph?

Ceph is an open source, distributed, scaled-out, software-defined storage system that can provide block, object, and file storage. The clusters of Ceph are designed in order to run on any hardware with the help of an algorithm called CRUSH (Controlled Replication Under Scalable Hashing).

The CRUSH algorithm enables the client to independently computes where data should be written to or read from. By deriving this metadata dynamically, there is no need to manage a centralized table.

Servers can perform a CRUSH lookup very quickly; moreover, a smaller computing load can be distributed across cluster nodes, leveraging the power of distributed storage. This allows Ceph can quickly scale to hundreds of petabytes without the risk of bottlenecks and the associated single points of failure.

Ceph storage type

Ceph is a true unified storage solution that provides block, file, and object services from a single unified software defined backend. Ceph provides three main types of storage:

• Block storage via the RADOS Block Device (RBD)

• File storage via CephFS

• Object storage via RADOS Gateway, which provides S3 and Swift-compatible storage.

As the following picture show:

You are free to run Ceph on any hardware that matches Ceph’s requirements. This is a major development in the storage industry, which has typically suffered from strict vendor lock-in. Also Ceph prefers consistency as per the CAP theorem, and will try at all costs to make protecting your data a higher priority than availability in the event of a partition.

Ceph history

• 2003–2007: Ceph was developed at University of California by Sage Weil in 2003 as a part of his PhD project. Then it was open sourced in 2006 under a LGPL to serve as a reference implementation and research platform. Lawrence Livermore National Laboratory supported Sage’s early followup work from 2003 to 2007.

• 2007–2011: DreamHost supported Ceph development from 2007 to 2011. During this period the core components of Ceph gained stability and reliability, new features were implemented, and the road map for the future was drawn.

• 2012 — Current: In 2012 Sage Weil founded Inktank to enable the widespread adoption of Ceph. In 2014 Red Hat agreed to acquire Inktank.

Active releases of Ceph are showing below:

Ceph benefit

Ceph comes with the following benefits:

• Scalability: It supports rapid scaling using CRUSH algorithm. Ceph overcomes all scalability challenges and provides truly flexible, resilient storage at scale.

• Reliability: Ceph comes with self-managing and self-healing features, it monitor and manager daemon processes coordinate to increase reliability and data vavilability across the entire system.

• Performance: As a pure SDS (Software-defined Storage system), it is designed to maximize effectiveness and performance, regardless of the underline infrastructure.

Ceph core components

RADOS and OSD

In Ceph, the core storage layer is called RADOS (Reliable Autonomous Distributed Object Store), it provides an object store on which the higher-level storage protocols are built. RADOS is:

• HA with no single point of failure (SPoF)

• Reliable and resilient

• Self-healing and Self-managing

• Adaptive

• Scalable

The RADOS layer consists of a number of object storage daemons (OSDs) and each OSD is completely independent and forms peer-to-peer relationships in the entire cluster. Then each OSD is typically mapped to a single disk, in contrast to the traditional approach of presenting a number of disks combined into a single device via a RAID controller to the OS.

MONS

The other key component is the MONs (monitors). The name here is very misleading, because MONs not only do cluster monitoring, but act as arbiters, traffic cops, and physicians for the cluster as a whole.

Monitors are responsible for forming a cluster quorum via the use of Paxos. Monitors use cluster maps to provide cluster state, and these cluster maps are used by both Ceph cluster components and clients to describe the cluster topology and enable data to be safely stored in the right location.

Manager

Manager is responsible for Ceph cluster configuration and statistics. In order to track the state of every single object in the cluster, Ceph hashes the underlying object names to place objects into a number of placement groups.

Then it use CRUSH algorithm to place the placement groups onto the OSDs. This significantly reduces the computation power needed for tracking millions of objects to a matter of tracking a much more manageable number of placement groups, normally measured in thousands. It responsibilities include:

• Management of drive and chassis status/locator LEDs

• Creating and managing a map of clients such as rbd-mirror and RADOS Gateway,

• Holistic management of Ceph’s scrubs

• Richer management of reweighting and rebalancing operations.

• Integration with external inventory systems such as RackTables, NetBox, HP SIM, and Cisco UCS Manager

• Interface for monitoring/metrics systems such as Nagios, Icinga, Graphite, and Prometheus

Ceph core services

Ceph RADOS Block Device (RBD)

The RBD service presents block (also known as volume) storage in a fashion that with traditional HDD/SDD applications can consume with little or no adjustment. It breaks up block-based application data into small chunks, and these small chunks are stored as objects within RADOS. RBD orchestrates the storage of the objects in virtual block devices

Ceph RADOS Gateway (RGW)

Ceph Object Gateway is an object storage interface built on top of librados to provide applications with a RESTful gateway to Ceph Storage Clusters. Ceph Object Storage supports two interfaces:

• S3-compatible: Provides object storage functionality with an interface that is compatible with a large subset of the Amazon S3 RESTful API.

• Swift-compatible: Provides object storage functionality with an interface that is compatible with a large subset of the OpenStack Swift API.

Under the hood, Ceph Object Gateway uses RGW (An HTTP server) for interacting with underline storage.

Ceph FS

The Ceph filesystem is a POSIX-compliant file system built on top of RADOS. CephFS endeavors to provide a state-of-the-art, multi-use, highly available, and performant file store for a variety of applications, including traditional use-cases like shared home directories, HPC scratch space, and distributed workflow shared storage.

In CephFS, file metadata is stored in a separate RADOS pool from the file data and served via MDS (Metadata server). Access to data is coordinated through the cluster of MDS which serve as authorities for the state of the distributed metadata cache cooperatively maintained by clients and MDS.

Ceph Librados

Based upon RADOS, the librados API enables you to create your own interface to the Ceph Storage Cluster. With librados, you can interact with the two types of Ceph daemons:

• The Ceph Monitor, which maintains a master copy of the cluster map.

• The Ceph OSD Daemon (OSD), which stores data as objects on a storage node.

Ceph Use Cases

• OpenStack/KVM: Ceph is the perfect match for providing storage to an OpenStack environment or KVM virtualization.

• K8s

• Large bulk block storage

• Object storage: Ceph currently has one of the best compatibility records for matching the S3 API

• Web farm servers

• NAS

• Big Data

Ceph Implementation Suggestions

The following are some general rules for deploying a successful Ceph cluster:

• Minimum 10G networking

• Don’t use the no barrier mount option with filestore

• Don’t configure pools with a size of two or a min_size of one

• Don’t use consumer SSDs

• Don’t use raid controllers in write back mode without battery protection

• Do have an agreed backup and recovery plan

Ceph Install

Different install methods

There are several different ways to install Ceph, such as:

• Cephadm: Installs and manages a Ceph cluster using containers and systemd, with tight integration with the CLI and dashboard GUI.

• Rook: Deploys and manages Ceph clusters running in K8s, while also enabling management of storage resources and provisioning via K8s APIs.

• ceph-ansible: Deploys and manages Ceph clusters using Ansible.

• ceph-salt: Installs Ceph using Salt and cephadm.

• jaas.ai/ceph-mon: Installs Ceph using Juju.

• github.com/openstack/puppet-ceph: Installs Ceph via Puppet.

• Manual: Ceph can also be installed manually.

If you want to install Ceph cluster stand alone, the recommended way is Cephadm, if you want to install Ceph cluster in K8s, then Rook is the recommended way.

Cephadm

Cephadm is a utility that is used to manage a Ceph cluster. It does not rely on external configuration tools like Ansible, Rook, or Salt. However, those external configuration tools can be used to automate operations not performed by cephadm itself.

Cephadm manages the full lifecycle of a Ceph cluster. This includes bootstrapping process by creating a tiny Ceph cluster on a single node then expand the cluster, adding hosts and provisioning Ceph daemons and services. The management of this lifecycle can be performed either via the Ceph command-line interface (CLI) or via the dashboard (GUI).

Ceph Cluster Installation

Requirements:

• Python 3

• Systemd

• Podman or Docker for running containers

• Time synchronization (such as chrony or NTP)

• LVM2 for provisioning storage devices

Download cephadm

There are two ways to install cephadm:

• curl-based installation

• distribution-specific installation

• We will use curl-based installation here:

• Download cephadm

$ curl --silent --remote-name --location https://github.com/ceph/ceph/raw/quincy/src/cephadm/cephadm

Move it to /usr/local/bin, make sure this path is in your environment $PATH

$ mv cephadm /usr/local/bin/$ which cephadm/usr/local/bin/cephadm

Bootstrap a new ceph cluster

The first step in creating a new Ceph cluster is running the cephadm bootstrap command on the Ceph cluster’s first host. The --mon-ip is the host IP:

$ cephadm bootstrap --mon-ip 10.221.36.156 --allow-fqdn-hostname

We have to allow feqn-hostname here, since by default AWS EC2 has ip-10-221-36-156.ec2.internal as hostname. Otherwise you will run into the following error:

ERROR: hostname is a fully qualified domain name (ip-10-221-36-156.ec2.internal); either fix (e.g., "sudo hostname ip-10-229-36-156" or similar) or pass --allow-fqdn-hostname

The command run output:

$ cephadm bootstrap --mon-ip 10.221.36.156 --allow-fqdn-hostnameVerifying podman|docker is present...Verifying lvm2 is present...Verifying time synchronization is in place......Pulling container image quay.io/ceph/ceph:v17...Ceph version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)Extracting ceph user uid/gid from container image......Ceph Dashboard is now available at:             URL: https://ip-10-221-36-156.ec2.internal:8443/            User: admin        Password: ehpzvrbf9zEnabling client.admin keyring and conf on hosts with "admin" labelSaving cluster configuration to /var/lib/ceph/0e7905be-61ff-11ed-86b2-12a743769881/config directoryEnabling autotune for osd_memory_targetYou can access the Ceph CLI as following in case of multi-cluster or non-default config:        sudo /usr/local/bin/cephadm shell --fsid 0e7905be-61ff-11ed-86b2-12a743769881 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyringOr, if you are only running a single cluster on this host:        sudo /usr/local/bin/cephadm shellPlease consider enabling telemetry to help improve Ceph:        ceph telemetry onFor more information see:        https://docs.ceph.com/docs/master/mgr/telemetry/

Now you should be able to visit Ceph UI from the above url: https://ip-10-221-36-156.ec2.internal:8443/

It will ask you to update the default password at first time login, once you update your password and re-login, you should see the following UI:

The landing dashboard should look like:

You can check number of hosts in the cluster:

Check services:

Check monitors: