Feature Request: Native StreamPipes HA / Failover Support for Edge Deployments

[amanchaudhary-95]

Problem Description

This is in addition to #3932
In industrial IIoT environments, the edge system running StreamPipes is often a single point of failure. If the edge device goes down due to hardware failure, network outage, or system crash, the entire data acquisition pipeline breaks.

Currently, StreamPipes does not provide native high availability (HA) or failover capability between multiple StreamPipes instances running at the edge. Each instance operates independently, and there is no built-in mechanism for:

• automatic failover
• cluster coordination
• shared state
• synchronized adapter/pipeline execution
• seamless continuation of data ingestion

This results in potential data loss, pipeline downtime, and reduced reliability for industrial use cases.

Requested Feature: StreamPipes HA / Failover Mode

Proposed Scenario (Real-World Use Case)

A StreamPipes instance runs on an edge device connected to a PLC. If this edge device fails (hardware crash, power issue, OS failure), data ingestion stops completely. If StreamPipes could run in a clustered/HA configuration, another StreamPipes node could immediately take over:

• reconnect to the PLC
• resume ingestion
• maintain pipeline continuity
• prevent data loss

Combining this with edge-side buffering (inside the adapter) would deliver a robust, fault-tolerant architecture for industrial IIoT.

Why This Matters:

Industrial automation environments require:

• Zero data loss
• Continuous monitoring
• 24×7 reliability
• Redundant systems

StreamPipes already provides excellent modular pipelines, adapters, and extensibility. Adding HA/failover capabilities would make StreamPipes production-ready for harsh industrial scenarios and significantly increase adoption.