diff --git a/docs/examples/kafka/confluent/kafka-confluent-license.yaml b/docs/examples/kafka/confluent/kafka-confluent-license.yaml new file mode 100644 index 0000000000..88d5910650 --- /dev/null +++ b/docs/examples/kafka/confluent/kafka-confluent-license.yaml @@ -0,0 +1,7 @@ +apiVersion: v1 +kind: Secret +metadata: + name: kafka-confluent-license + namespace: demo +stringData: + license: diff --git a/docs/examples/kafka/confluent/kafka-confluent.yaml b/docs/examples/kafka/confluent/kafka-confluent.yaml new file mode 100644 index 0000000000..9438218827 --- /dev/null +++ b/docs/examples/kafka/confluent/kafka-confluent.yaml @@ -0,0 +1,19 @@ +apiVersion: kubedb.com/v1 +kind: Kafka +metadata: + name: kafka-confluent + namespace: demo +spec: + version: confluent-8.3.2 + license: + secretName: kafka-confluent-license + replicas: 3 + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: standard + deletionPolicy: WipeOut diff --git a/docs/guides/kafka/concepts/kafka.md b/docs/guides/kafka/concepts/kafka.md index bbc7c47a09..66470edbe3 100644 --- a/docs/guides/kafka/concepts/kafka.md +++ b/docs/guides/kafka/concepts/kafka.md @@ -227,6 +227,36 @@ configuration: connection.uri=mongodb://mongo-user:mongo-password@mongo-host:27017 ``` +### spec.license + +`spec.license` is an optional field used to provide an enterprise license for a licensed Kafka distribution, such as [Confluent Server](/docs/guides/kafka/concepts/kafkaversion.md#specdistribution) (`KafkaVersion.spec.distribution: Confluent`). It references a Secret containing the license key. This field has no effect for the default `KubeDB` distribution. + +- `spec.license.secretName` is a required field that specifies the name of the Secret (in the same namespace as the Kafka object) that holds the license key. +- `spec.license.key` is an optional field that specifies the key inside the Secret's data that holds the license value. Defaults to `license`. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: kafka-confluent-license + namespace: demo +stringData: + license: +--- +apiVersion: kubedb.com/v1 +kind: Kafka +metadata: + name: kafka-confluent + namespace: demo +spec: + version: confluent-8.3.2 + license: + secretName: kafka-confluent-license + ... +``` + +Without a valid license, Confluent Server runs on a 30-day trial and then stops working, so `spec.license` should be set before the trial period expires. + ### spec.tieredStorage `spec.tieredStorage` is an optional field that specifies the tiered storage configuration for Kafka. Tiered storage allows Kafka to offload older data to cheaper storage solutions like S3, Azure Blob Storage, GCS, or local storage, while keeping recent data on faster local storage. Tiered storage helps in reducing the cost of storage and improves the performance of Kafka by keeping the frequently accessed data on local storage. Following are the fields of `spec.tieredStorage`: - `provider` ( `string` | `""` ) - is a field that specifies the tiered storage provider. Supported providers are `s3`, `azure`, `gcs`, and `local`. diff --git a/docs/guides/kafka/concepts/kafkaversion.md b/docs/guides/kafka/concepts/kafkaversion.md index 11ce7b9241..44ab25f4f7 100644 --- a/docs/guides/kafka/concepts/kafkaversion.md +++ b/docs/guides/kafka/concepts/kafkaversion.md @@ -78,6 +78,27 @@ We use official Apache Kafka release tar files to build docker images for suppor The default value of this field is `false`. If `spec.deprecated` is set to `true`, KubeDB operator will skip processing this CRD object and will add a event to the CRD object specifying that the DB version is deprecated. +### spec.distribution + +`spec.distribution` is an optional field that specifies the Kafka distribution this `KafkaVersion` uses. Supported values are: + +- `KubeDB` (default) - kubedb's own image, built from the official Apache Kafka release. +- `Confluent` - [Confluent Server](https://docs.confluent.io/platform/current/installation/docker/image-reference.html) (Enterprise), referenced directly from Confluent's own image registry rather than rehosted under KubeDB's registry, since Confluent Server's license is commercial and doesn't permit redistribution. + +A `Kafka` object using a `Confluent` distribution `KafkaVersion` must have `spec.license` set (see [Kafka CRD](/docs/guides/kafka/concepts/kafka.md#speclicense)) once Confluent's 30-day trial period expires. + +```yaml +apiVersion: catalog.kubedb.com/v1alpha1 +kind: KafkaVersion +metadata: + name: confluent-8.3.2 +spec: + distribution: Confluent + db: + image: docker.io/confluentinc/cp-server:8.3.2 + ... +``` + ### spec.db.image `spec.db.image` is a required field that specifies the docker image which will be used to create PetSet by KubeDB operator to create expected Kafka database. diff --git a/docs/guides/kafka/confluent/_index.md b/docs/guides/kafka/confluent/_index.md new file mode 100644 index 0000000000..c7b414fcba --- /dev/null +++ b/docs/guides/kafka/confluent/_index.md @@ -0,0 +1,10 @@ +--- +title: Confluent Enterprise +menu: + docs_{{ .version }}: + identifier: kf-confluent-kafka + name: Confluent Enterprise + parent: kf-kafka-guides + weight: 27 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/kafka/confluent/confluent.md b/docs/guides/kafka/confluent/confluent.md new file mode 100644 index 0000000000..b848bbce57 --- /dev/null +++ b/docs/guides/kafka/confluent/confluent.md @@ -0,0 +1,147 @@ +--- +title: Confluent Enterprise Kafka +menu: + docs_{{ .version }}: + identifier: kf-confluent-kafka-docs + name: Overview + parent: kf-confluent-kafka + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Confluent Enterprise Kafka + +KubeDB can run [Confluent Server](https://docs.confluent.io/platform/current/installation/docker/image-reference.html) (Confluent's Enterprise Kafka distribution) instead of KubeDB's own Apache Kafka based image. This tutorial will show you how to deploy a Kafka cluster on the `Confluent` distribution and provide it with an enterprise license. + +## Before You Begin + +At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +Now, install the KubeDB operator in your cluster following the steps [here](/docs/setup/install/_index.md). + +To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create namespace demo +namespace/demo created + +$ kubectl get namespace +NAME STATUS AGE +demo Active 9s +``` + +> Note: YAML files used in this tutorial are stored in [examples/kafka/confluent/](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/kafka/confluent) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Confluent distribution KafkaVersion + +Running Confluent Server requires a `KafkaVersion` catalog object whose `spec.distribution` is set to `Confluent`, pointing `spec.db.image` at Confluent's own image directly (KubeDB doesn't rehost Confluent Server, since its license is commercial and doesn't permit redistribution). KubeDB ships a `confluent-8.3.2` `KafkaVersion` by default: + +```bash +$ kubectl get kafkaversion confluent-8.3.2 -o yaml +``` + +```yaml +apiVersion: catalog.kubedb.com/v1alpha1 +kind: KafkaVersion +metadata: + name: confluent-8.3.2 +spec: + distribution: Confluent + db: + image: docker.io/confluentinc/cp-server:8.3.2 + version: 8.3.2 + ... +``` + +See the [KafkaVersion concept](/docs/guides/kafka/concepts/kafkaversion.md#specdistribution) page for details. + +## Create License Secret + +Confluent Server runs on a 30-day trial without a license, then stops working. Create a Secret containing your enterprise license key: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: kafka-confluent-license + namespace: demo +stringData: + license: +``` + +Apply the secret: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/kafka/confluent/kafka-confluent-license.yaml +secret/kafka-confluent-license created +``` + +## Create a Confluent Enterprise Kafka + +Here is an example `Kafka` CR that uses the `confluent-8.3.2` `KafkaVersion` and the license Secret created above: + +```yaml +apiVersion: kubedb.com/v1 +kind: Kafka +metadata: + name: kafka-confluent + namespace: demo +spec: + version: confluent-8.3.2 + license: + secretName: kafka-confluent-license + replicas: 3 + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: standard + deletionPolicy: WipeOut +``` + +Here, + +- `spec.version` is set to `confluent-8.3.2`, the `Confluent` distribution `KafkaVersion`. +- `spec.license.secretName` points at the Secret holding the enterprise license key. See the [Kafka concept](/docs/guides/kafka/concepts/kafka.md#speclicense) page for details. + +Apply the yaml: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/kafka/confluent/kafka-confluent.yaml +kafka.kubedb.com/kafka-confluent created +``` + +KubeDB operator watches for `Kafka` objects using Kubernetes API. When a `Kafka` object is created, KubeDB operator will create a new PetSet running Confluent Server, along with the other Kubernetes resources like Service, Secret etc. required to run the Kafka cluster. + +```bash +$ kubectl get kafka -n demo -w +NAME TYPE VERSION STATUS AGE +kafka-confluent kubedb.com/v1 confluent-8.3.2 Provisioning 2s +kafka-confluent kubedb.com/v1 confluent-8.3.2 Provisioning 4s +. +. +kafka-confluent kubedb.com/v1 confluent-8.3.2 Ready 112s +``` + +## Cleanup + +To clean up the resources created by this tutorial, run the following commands: + +```bash +$ kubectl delete -n demo kafka kafka-confluent +$ kubectl delete -n demo secret kafka-confluent-license +$ kubectl delete ns demo +``` + +## Next Steps + +- Detail concepts of [Kafka object](/docs/guides/kafka/concepts/kafka.md). +- Detail concepts of [KafkaVersion object](/docs/guides/kafka/concepts/kafkaversion.md). +- Deploy your first Kafka database with KubeDB by following the guide [here](/docs/guides/kafka/quickstart/kafka/index.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md).