Is There an Existing Flow to Send Data to REST API and Process Response?

[Sugandhab]

Hi team,

I’m exploring a use case where I need to send incoming weather data to an external REST API (AI simulator) and use its response to continue processing within the same pipeline (e.g., store prediction in a database).

Before I go ahead and build a custom processor, I wanted to check:
👉 Is there any existing processor or flow in StreamPipes that supports sending data to a REST API and handling the response as part of the event stream?

Would appreciate any pointers or examples if something similar exists.

Thanks in advance!

[bossenti]

Hey @Sugandhab

For your use case StreamPipes Functions is probably the way to go.
Here is one example from our Python SDK. Of course, functions are also available in the Java SDK if that suits you more.

Let me here what you think about it and if you need some help to get it started 🙂

[Sugandhab]

I understand that we can use functions for such scenarios but how do we connect these functions to pipelines?
also do we have any Java sdk examples available so that I can run through it and can understand end to end process.

[dominikriemer]

Hi @Sugandhab if you want to create a pipeline element which calls a REST service and transforms the event stream with the response from the service, have a look at the SDK tutorial [1] and our own pipeline elements [2]. I recommend to start with the Maven archetype which generates the code skeleton for you.

[1] https://streampipes.apache.org/docs/extend-tutorial-data-processors/
[2] e.g., https://github.com/apache/streampipes/blob/dev/streampipes-extensions/streampipes-processors-enricher-jvm/src/main/java/org/apache/streampipes/processors/enricher/jvm/processor/limitsenrichment/QualityControlLimitsEnrichmentProcessor.java

[tenthe]

Hi @Sugandhab,

the function would be a separate service containing dedicated code, capable of subscribing to event streams and creating new ones.
If you want to reuse your code in a pipeline, you need to implement a data processor.

In general:

• For complex tasks or tasks that have a fixed data stream, use a Function, because it provides a dedicated solution.
• For more flexibility and generic tasks, Processors work better, as they can be used in different pipelines and configured by user input.

Here’s the documentation for Java Functions: Link

I hope this helps!