Network volumes in Kubernetes provide great flexibility, but still, nothing beats local volumes from direct-attached storage in the sense of database performance.
I want to explore ways to deploy both Percona Kubernetes Operators (Percona Kubernetes Operator for Percona XtraDB Cluster and Percona Kubernetes Operator for Percona Server for MongoDB) using local volumes, both on the bare-metal deployments or in cloud deployments.
Simple Ways
There are two ways available out of the box to deploy using local storage, which you can use immediately.
You can specify in cluster deployment yaml volume specification, using either hostPath:
1 2 3 4 | volumeSpec: hostPath: path: /data type: Directory |
Or emptyDir (which will be equal somewhat to ephemeral storage in EC2):
1 2 | volumeSpec: emptyDir: {} |
While this will work, it is a very rigid way to force local storage, and it is not really the “Kubernetes way”, as we will lose the capability to manage data volumes. We want to see a more uniform way with Persistent Volumes and Persistent Volumes Claims.
Persistent Volumes
Recognizing limitations with hostPath and emptyDir, Kubernetes introduced Local Persistent Volumes.
Unfortunately, while this will work for the simple deployment of single pods, it does not work with dynamically created volumes which we need for Operators. We need the support of Dynamic Volume Provisioning.
There are several projects to combine Dynamic Volume Provisioning with Local Persistent Volumes, but I did not have much success with them, and the only project which worked for me is OpenEBS.
OpenEBS provides much more than just Local Persistent Volumes, but in this blog, I want to touch only OpenEBS Local PV Hostpath.
OpenEBS Local PV Hostpath
This is actually quite simple, and this is what I like about OpenEBS. First, we will define storage classes:
For the local nvme storage:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: localnvme annotations: openebs.io/cas-type: local cas.openebs.io/config: | - name: StorageType value: hostpath - name: BasePath value: /data/openebs provisioner: openebs.io/local reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer |
And for the local ssd:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: localssd annotations: openebs.io/cas-type: local cas.openebs.io/config: | - name: StorageType value: hostpath - name: BasePath value: /mnt/data/ebs provisioner: openebs.io/local reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer |
And after that we can deploy Operators using StorageClass, for example in cluster deployment yaml:
1 2 3 4 5 6 7 | volumeSpec: persistentVolumeClaim: storageClassName: localnvme accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 200Gi |
The cluster will be deployed using localnvme StorageClass, and using space on /data/openebs.
Now we can observe used volumes with kubectl get pv:
1 2 3 4 5 6 7 | NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-325e11ff-9082-432c-b484-c0c7a3d1c949 200Gi RWO Delete Bound pxc/datadir-cluster2-pxc-2 localssd 2d18h pvc-3f940d14-8363-450e-a238-a9ff09bee5bc 200Gi RWO Delete Bound pxc/datadir-cluster3-pxc-1 localnvme 43h pvc-421cbf46-73a6-45f1-811e-1fca279fe22d 200Gi RWO Delete Bound pxc/datadir-cluster2-pxc-1 localssd 2d18h pvc-53ee7345-bc53-4808-97d6-2acf728a57d7 200Gi RWO Delete Bound pxc/datadir-cluster3-pxc-0 localnvme 43h pvc-b98eca4a-03b0-4b5f-a152-9b45a35767c6 200Gi RWO Delete Bound pxc/datadir-cluster2-pxc-0 localssd 2d18h pvc-fbe499a9-ecae-4196-b29e-c4d35d69b381 200Gi RWO Delete Bound pxc/datadir-cluster3-pxc-2 localnvme 43h |
There we can see data volumes from two deployed clusters. OpenEBS Local PV Hostpath is now my go-to method to deploy clusters with the local storage.
