Utilizing Kraft based Kafka in Kubernetes Deployment

[heisenbergs-uncertainty]

Hi,

I have been working on updating the helm chart to provide more options around configuring the charts to replace some of the external packages with chart dependencies along with additional options to disable the dependency if you already have the required dependency deployed. The goal is to get the helm chart deployment working in a similar way to the Grafana Oncall Helm Chart.

The external dependency I am wondering about is the Kafka and Zookeeper deployments. Would it be possible in the current state to substitute the current Kafka implementation utilizing ZooKeeper with a Kafka implementation based on Kraft? I'm not sure what the internal dependencies of Streampipes is on ZooKeeper when utilizing Kafka as the message broker. Would implementing a Kraft based system break internal functionality or would it be as simple as modifying the environment parameters based on a value like "kafka.mode"?

Hopefully that makes sense. Let me know if I need to clarify at all!

[dominikriemer]

Hi @what-is-a-theostack thanks for starting the discussion!
I think it's a good idea to introduce chart dependencies to our helm charts.
For Kafka, we have already experimented with an upgrade to 3.x and Kraft. There was only one issue concerning the live preview of pipeline (I think due to slower partition reassignment), but this shouldn't be a problem anymore with the most recent Kafka versions and some improvements we added to the pipeline preview.

So I guess we can target a replacement of Kafka/Zookeeper with Kafka/Kraft for the next release and I don't expect any incompatibilities in the internal messaging system.

Do you plan to contribute the helm improvements to our k8s installer?

[heisenbergs-uncertainty]

I would like to! It's currently for personal projects but I would love to contribute to the project! Big fan of this software.

I'm personally using Kafka so that dependency along with couchdb and influxdb should be included in the first go as well. Eventually I will look at updating the nats and pulsar dependencies + monitoring once I have the core dependencies locked down.

[heisenbergs-uncertainty]

Also my goal is to preserve the zookeeper functionality while just including the ability to use Kraft deployments. So people who are using zookeeper currently won't be forced to upgrade to Kraft just to continue using the helm chart with new releases

[dominikriemer]

Sounds good! I did a few quick tests and it seems there are no major code changes needed. So I'll open an issue to enable Kraft as default for new installations and then we can discuss if there are any additional things we need to take care of.

[heisenbergs-uncertainty]

Sounds good! I will take a look at the contributing documentation to see what I need to do to contribute.

[dominikriemer]

@what-is-a-theostack I see that you asked something about service authentication but it seems the comment got deleted - just want to ask if you were you able to solve the issue?

[heisenbergs-uncertainty]

I was able to figure it out! I was having authentication issues with the JWT Validation functions but it turned out to be old credentials hanging out in the couchdb. Reset couchdb and it resolved.

Although, I have found that the extensions service and the backend service seem to have a lot of trouble staying healthy in the kubernetes setup. The logs show the extensions service bouncing back and forth between healthy and unhealthy and registering and de-registering with the backend. The pods end up racking up quite a bit of restarts over time and I can't really figure out what the root cause is.

This behavior is present in the current helm chart as well so I'm confident it's not due to my implementation.

[heisenbergs-uncertainty]

Well I think I figured this out as well haha. I've been developing locally with minikube and it turns out the issue was due to the resources not having enough dedicated RAM. Increased the memory and the restarts stopped.

Sorry about the spam. Still getting a hang of the landscape.