Monitoring and Analytics Built into Streampipes

[heisenbergs-uncertainty]

I just wanted to put this idea out there to see what kind of interest there would be in working on building analytics and monitoring into the main UI of Streampipes.

I know we are able to collect analytics and monitor in Grafana but with the dash-boarding capability of Streampipes it would be pretty cool to build observability and monitoring into the platform.

I had some ideas on it such as:

• Kafka topic sizes, maybe even adding some exposure to all the kafka topics and enabling something similar to RedPanda UI
• Adapter throughput - stuff like events/sec, average event size, etc
• Ability to enable monitoring on processors to see more information on these
• Status of adapters - add the ability to monitor whether an adapter is connected or not
  • This might be a separate issue from observability and just a general feature of adapters I was thinking of.
• Anything else people can think of

I would be willing to work on some kind of POC or beta version of this but I wanted to see if you guys think its a viable feature before I started as I think it would be a significant amount of work but could be very valuable.

[tenthe]

Hi @heisenbergs-uncertainty,

Thanks for bringing this up — I also see a lot of value in improving monitoring and observability in StreamPipes.

Like you mentioned, I also think we should first align on what kind of monitoring functionality should be built directly into StreamPipes, and what should be delegated to external tools like Grafana, before jumping into implementation.

There are already some built-in monitoring features that might serve as a good starting point. For example, when you go to Connect or Pipelines and click on Show details, you'll see:

• A metrics tab
• A logs tab showing output from the extension service

Here are some existing classes that could serve as a starting point:

• AbstractMonitoringResource
  • AdapterMonitoringResource
  • PipelineMonitoring
• ExtensionsLogProvider

We could consider extending these components to support some of your ideas. I particularly like the idea of notifying when an adapter stops emitting data — for example via email — as an early step. That could be quite helpful operationally.

Let’s maybe collect and prioritize use cases (e.g., adapter throughput, Kafka topic sizes, processor-level metrics), and then see how far we can get with extending the existing functionality before introducing new components.

What do you think?

[heisenbergs-uncertainty]

I really like the way you laid this out.

I think it would be important to keep with the Streampipes theme of abstract and extendable resources that allow for easy replication and customization so that we don't limit the application of monitoring.

I think it would also be important to provide some sort of switch for enabling monitoring and configuring the monitoring. Maybe up front we could start with a general switch that allows turning it on and eventually extend it where it can be toggled/configured at a pipeline element level. Hopefully that makes sense. I'm thinking of a similar manner to how kubernetes implements resource limits.

What would be the process of collecting and prioritizing use cases?

[tenthe]

My suggestion would be to collect them here in this discussion and then create issues to be impltmented