Currently, Installer supports the following Storage Class, providing persistent storage service for KubeSphere (more storage classes will be supported soon).
The versions of storage systems and corresponding CSI plugins in the table listed below have been well tested.
| Name | Version | Reference |
|---|---|---|
| Ceph RBD Server | v0.94.10 | For development and testing, refer to Install Ceph Storage Server for details. Please refer to Ceph Documentation for production. |
| Ceph RBD Client | v12.2.5 | Before installing KubeSphere, you need to configure the corresponding parameters in common.yaml. Please refer to Ceph RBD |
| GlusterFS Server | v3.7.6 | For development and testing, refer to Deploying GlusterFS Storage Server for details. Please refer to Gluster Documentation or Gluster Documentation for production. Note you need to install Heketi Manager (V3.0.0). |
| GlusterFS Client | v3.12.10 | Before installing KubeSphere, you need to configure the corresponding parameters in common.yaml. Please refer to GlusterFS |
| NFS Client | v3.1.0 | Before installing KubeSphere, you need to configure the corresponding parameters in common.yaml. Make sure you have prepared NFS storage server. Please see NFS Client |
| QingCloud-CSI | v0.2.0.1 | You need to configure the corresponding parameters in common.yaml before installing KubeSphere. Please refer to QingCloud CSI for details |
| NeonSAN-CSI | v0.3.0 | Before installing KubeSphere, you need to configure the corresponding parameters in common.yaml. Make sure you have prepared QingStor NeonSAN storage server. Please see Neonsan-CSI |
Note: You are only allowed to set ONE default storage classes in the cluster. To specify a default storage class, make sure there is no default storage class already exited in the cluster.
After preparing the storage server, you need to refer to the parameters description in the following table. Then modify the corresponding configurations in conf/common.yaml accordingly.
The following describes the storage configuration in common.yaml.
Note: Local Volume is configured as the default storage class in
common.yamlby default. If you are going to set other storage class as the default, disable the Local Volume and modify the configuration for other storage class.
A Local Volume represents a mounted local storage device such as a disk, partition or directory. Local volumes can only be used as a statically created PersistentVolume. We recommend you to use Local volume in testing or development only since it is quick and easy to install KubeSphere without the struggle to set up persistent storage server. Refer to following table for the definition in conf/common.yaml.
| Local volume | Description |
|---|---|
| local_volume_provisioner_enabled | Whether to use Local as the persistent storage, defaults to true |
| local_volume_provisioner_storage_class | Storage class name, default value：local |
| local_volume_is_default_class | Whether to set Local as the default storage class, defaults to true. |
An NFS volume allows an existing NFS (Network File System) share to be mounted into your Pod. NFS can be configured in conf/common.yaml. Note you need to prepare NFS server in advance.
| NFS | Description |
|---|---|
| nfs_client_enable | Whether to use NFS as the persistent storage, defaults to false |
| nfs_client_is_default_class | Whether to set NFS as default storage class, defaults to false. |
| nfs_server | The NFS server address, either IP or Hostname |
| nfs_path | NFS shared directory, which is the file directory shared on the server, see Kubernetes Documentation |
| nfs_vers3_enabled | Specifies which version of the NFS protocol to use, defaults to false which means v4. True means v4 |
| nfs_archiveOnDelete | Archive PVC when deleting. It will automatically remove data from oldPath when it sets to false |
The open source Ceph RBD distributed storage system can be configured to use in conf/common.yaml. You need to prepare Ceph storage server in advance. Please refer to Kubernetes Documentation for more details.
| Ceph_RBD | Description |
|---|---|
| ceph_rbd_enabled | Whether to use Ceph RBD as the persistent storage, defaults to false |
| ceph_rbd_storage_class | Storage class name |
| ceph_rbd_is_default_class | Whether to set Ceph RBD as default storage class, defaults to false |
| ceph_rbd_monitors | Ceph monitors, comma delimited. This parameter is required, which depends on Ceph RBD server parameters |
| ceph_rbd_admin_id | Ceph client ID that is capable of creating images in the pool. Defaults to “admin” |
| ceph_rbd_admin_secret | Admin_id's secret, secret name for "adminId". This parameter is required. The provided secret must have type “kubernetes.io/rbd” |
| ceph_rbd_pool | Ceph RBD pool. Default is “rbd” |
| ceph_rbd_user_id | Ceph client ID that is used to map the RBD image. Default is the same as adminId |
| ceph_rbd_user_secret | Secret for User_id, it is required to create this secret in namespace which used rbd image |
| ceph_rbd_fsType | fsType that is supported by Kubernetes. Default: "ext4" |
| ceph_rbd_imageFormat | Ceph RBD image format, “1” or “2”. Default is “1” |
| ceph_rbd_imageFeatures | This parameter is optional and should only be used if you set imageFormat to “2”. Currently supported features are layering only. Default is “”, and no features are turned on |
Note:
The ceph secret, which is created in storage class, like "cephrbdadminsecret" and "cephrbdusersecret", is retrieved using following command in Ceph storage server.
ceph auth get-key client.adminGlusterFS is a scalable network filesystem suitable for data-intensive tasks such as cloud storage and media streaming. You need to prepare GlusterFS storage server in advance. Please refer to Kubernetes Documentation for further information.
| GlusterFS（It requires glusterfs cluster which is managed by heketi） | Description |
|---|---|
| glusterfs_provisioner_enabled | Whether to use GlusterFS as the persistent storage, defaults to false |
| glusterfs_provisioner_storage_class | Storage class name |
| glusterfs_is_default_class | Whether to set GlusterFS as default storage class, defaults to false |
| glusterfs_provisioner_restauthenabled | Gluster REST service authentication boolean that enables authentication to the REST server |
| glusterfs_provisioner_resturl | Gluster REST service/Heketi service url which provision gluster volumes on demand. The general format should be "IP address:Port" and this is a mandatory parameter for GlusterFS dynamic provisioner |
| glusterfs_provisioner_clusterid | Optional, for example, 630372ccdc720a92c681fb928f27b53f is the ID of the cluster which will be used by Heketi when provisioning the volume. It can also be a list of clusterids |
| glusterfs_provisioner_restuser | Gluster REST service/Heketi user who has access to create volumes in the Gluster Trusted Pool |
| glusterfs_provisioner_secretName | Optional, identification of Secret instance that contains user password to use when talking to Gluster REST service, Installer will automatically create this secret in kube-system |
| glusterfs_provisioner_gidMin | The minimum value of GID range for the storage class |
| glusterfs_provisioner_gidMax | The maximum value of GID range for the storage class |
| glusterfs_provisioner_volumetype | The volume type and its parameters can be configured with this optional value, for example: ‘Replica volume’: volumetype: replicate:3 |
| jwt_admin_key | "jwt.admin.key" field is from "/etc/heketi/heketi.json" in Heketi server |
Attention：
Please note:
"glusterfs_provisioner_clusterid"could be returned from glusterfs server by running the following command:
export HEKETI_CLI_SERVER=http://localhost:8080
heketi-cli cluster listQingCloud Block Storage is supported in KubeSphere as the persistent storage service. If you would like to experience dynamic provisioning when creating volume, we recommend you to use it as your persistent storage solution. KubeSphere integrates QingCloud-CSI, and allows you to use various block storage services of QingCloud. With simple configuration, you can quickly expand, clone PVCs and view the topology of PVCs, create/delete snapshot, as well as restore volume from snapshot.
QingCloud-CSI plugin has implemented the standard CSI. You can easily create and manage different types of volumes in KubeSphere, which are provided by QingCloud. The corresponding PVCs will created with ReadWriteOnce access mode and mounted to running Pods.
QingCloud-CSI supports create the following five types of volume in QingCloud:
| QingCloud-CSI | Description |
|---|---|
| qingcloud_csi_enabled | Whether to use QingCloud-CSI as the persistent storage volume, defaults to false |
| qingcloud_csi_is_default_class | Whether to set QingCloud-CSI as default storage class, defaults to false |
| qingcloud_access_key_id , qingcloud_secret_access_key |
Please obtain it from QingCloud Console |
| qingcloud_zone | Zone should be the same as the zone where the Kubernetes cluster is installed, and the CSI plugin will operate on the storage volumes for this zone. For example: zone can be set to these values, such as sh1a (Shanghai 1-A), sh1b (Shanghai 1-B), pek2 (Beijing 2), pek3a (Beijing 3-A), pek3b (Beijing 3-B), pek3c (Beijing 3-C), gd1 (Guangdong 1), gd2a (Guangdong 2-A), ap1 (Asia Pacific 1), ap2a (Asia Pacific 2-A) |
| type | The type of volume in QingCloud platform. In QingCloud platform, 0 represents high performance volume. 3 represents super high performance volume. 1 or 2 represents high capacity volume depending on cluster‘s zone, see QingCloud Documentation |
| maxSize, minSize | Limit the range of volume size in GiB |
| stepSize | Set the increment of volumes size in GiB |
| fsType | The file system of the storage volume, which supports ext3, ext4, xfs. The default is ext4 |
The NeonSAN-CSI plugin supports the enterprise-level distributed storage QingStor NeonSAN as the persistent storage solution. You need prepare the NeonSAN server, then configure the NeonSAN-CSI plugin to connect to its storage server in conf/common.yaml. Please refer to NeonSAN-CSI Reference for further information.
| NeonSAN | Description |
|---|---|
| neonsan_csi_enabled | Whether to use NeonSAN as the persistent storage, defaults to false |
| neonsan_csi_is_default_class | Whether to set NeonSAN-CSI as the default storage class, defaults to false |
| Neonsan_csi_protocol | transportation protocol, user must set the option, such as TCP or RDMA |
| neonsan_server_address | NeonSAN server address |
| neonsan_cluster_name | NeonSAN server cluster name |
| neonsan_server_pool | A comma separated list of pools. Tell plugin to manager these pools. User must set the option, the default value is kube |
| neonsan_server_replicas | NeonSAN image replica count. Default: 1 |
| neonsan_server_stepSize | set the increment of volumes size in GiB. Default: 1 |
| neonsan_server_fsType | The file system to use for the volume. Default: ext4 |