How does cache tiering work under Ceph?

In the past, with SATA SSDs as cache storage, the performance improvement from its tiering was insignificant. Nowadays, the cost of NVMe SSDs has dropped sharply and the transfer performance of NVMe SSDs is much higher than that of HDDs. Therefore, it was necessary to find out whether NVMe SSDs as a cache tier significantly accelerate an HDD pool.

To test the effectiveness of an NVMe cache, a demo scenario was designed that could pick up the performance of an HDD-based storage pool.

Mars500 Cluster

Mars400 Cluster

• 24 x OSD on NVMe SSD (3x Ambedded Mars500 appliances)
• 24x OSD on HDD (3x Ambedded Mars400 appliances)
• HDD and NVMe servers are located in separated CRUSH roots.

Test Celents

• 2 x physical servers. 2x 25Gb network card
• Each server runs 7x VMs.
• Each VM has 4x core and 8 GB memory

Setup the Cache Tier by Ambedded UVS manager

 1. Create a base pool by using the HDD osd.

 2. Create an NVMe pool using the NVMe SSD osd.

 3. Add the NVMe pool as the cache tier of the HDD pool. 

 

During the cache test, the engineers observed the pool statistics using the ceph osd pool stats command. Flushing, evicting and promoting activities took place in the cache and base pools. The pool statistics were recorded at different times during the cache test.

Data was written to the cache 
pool cache id 84 
 client io 21 MiB/s wr, 0 op/s rd, 5.49k op/s wr 

pool mars400_rbd id 86 
 nothing is going on

Cache was doing promote and evict 
pool cache id 84 
 client io 42 MiB/s wr, 0 op/s rd, 10.79k op/s wr 
 cache tier io 179 MiB/s evict, 17 op/s promote 

pool mars400_rbd id 86 
 client io 0 B/s rd, 1.4 MiB/s wr, 18 op/s rd, 358 op/s wr

Cache was flusing 
pool cache id 84 
 client io 3.2 GiB/s rd, 830 op/s rd, 0 op/s wr 
 cache tier io 238 MiB/s flush, 14 op/s promote, 1 PGs flushing 

pool mars400_rbd id 86 
 client io 126 MiB/s rd, 232 MiB/s wr, 44 op/s rd, 57 op/s wr

PG was evicting 
pool cache id 84 
 client io 2.6 GiB/s rd, 0 B/s wr, 663 op/s rd, 0 op/s wr 
 cache tier io 340 MiB/s flush, 2.7 MiB/s evict, 21 op/s promote, 1 PGs evicting (full) 

pool mars400_rbd id 86 
 client io 768 MiB/s rd, 344 MiB/s wr, 212 op/s rd, 86 op/s wr

PG Flushing and client IO direct to base pool. (clients were writing data) 
pool cache id 84 
 client io 0 B/s wr, 0 op/s rd, 1 op/s wr 
 cache tier io 515 MiB/s flush, 7.7 MiB/s evict, 1 PGs flushin 

pool mars400_rbd id 86 
 client io 613 MiB/s wr, 0 op/s rd, 153 op/s wr

After the endurance test, the experts let the cluster rest for a few hours and repeated the 4 kB random write test. They got much better performance afterwards. This was because the cache space was released for the new write.

After this test, they were also confident that using the NVMe pool as the cache tier of an HDD pool could provide a significant performance improvement.

It should be noted, however, that the performance of cache tiering cannot be guaranteed. Performance depends on the current cache hits, and the same performance cannot be achieved by repeating tests with the same configuration and workloads.

Conclusion: If an application requires consistent performance, it should have a pure NVMe SSD pool.