reverse proxy for ui - nginx

[jormus]

Hey @dominikriemer
is there a mayor change between version 0.95 and 0.97 for ingress / nginx rules?
i cant get the live data on adapters and streampipes working anymore.
i see errors on icon loading, and no responses for live data in the browser console.

should we document ingress rules to run this project behind reverse proxy?

Thx. !

[heisenbergs-uncertainty]

Hey I'm not Dominik but I upgraded from 0.95 to 0.97.

I don't think there are any significant differences in the reverse proxy logic or for the reverse proxy itself.

It could be environmental settings if you are using kubernetes that are not being set correctly. It also could be improper registration of the adapter service which I've had happen to me.

How are you deploying? If you have logs from the backend and from the extensions service I can try and diagnose better

[dominikriemer]

That's right, in general there are no differences in the reverse proxy configuration between the two versions.
But in 0.97.0, we replaced a long polling-based approach in the live preview with a streaming approach (application/octet-stream).

So depending on the ingress or the way you route the requests, there might be some changes needed although we didn't require any changes in the Nginx config or the Traefik configuration in our internal K8s setup.

Icon loading is probably a different issue, do you have a volume for the core service defined? By default, StreamPipes stores assets such as icons and documentation in ${user.home}/.streampipes. The location can be changed by providing the env variable SP_CORE_ASSET_BASE_DIR.

[jormus]

I don't know what exactly has fixed my issue today, should have reported earlier, I ended up redeploying the complete stack, and reverting back to a simpler config map.
I post the nginx rules tomorrow, when I'm behind my office laptop, could maybe help somebody else.
@heisenbergs-uncertainty I deploy with argo ci/cd and k8s manifests, if u are intrested in the yamls feel free to ask.

[jormus]

ok , it seems when I redeploy the backend service the issue is back, all http500 errors on icon loading. strange.
@dominikriemer I did not create volume mount for the assets. It seems something where the icons are not installed on re deploying the backend service, what I initially did to try solve an opc-ua issue where the backend keeps trying connecting with a server with wrong creds even after I deleted the adapter. (bug report?)
after uninstalling processors / adapters / sinks, and reinstalling them, all icons are working again.
so, its safe to say ingress is not the problem.
is it best practice to use the SP_CORE_ASSET_BASE_DIR env variable?

[heisenbergs-uncertainty]

How are you deploying? It's hard to tell if I don't know what you are doing.

It sounds like the extensions service is not properly registering with the backend.

I have found in kubernetes that if you redeploy the backend, you need to restart your extensions afterwards. There seems to be an issue in kubernetes where the extensions does not successfully unregister and re-register with the new backend and need to be restarted AFTER the backend has already fully restarted

[heisenbergs-uncertainty]

Just saw that it's k8s.

Follow my process that I outlined above.

The current helm chart does not always successfully restart the backend service due to multiple poss accessing the same Persistent Volume Claim. When this happens, the extensions service will complete its restart before the backend and register with the old service and will prevent the two services from correctly coordinating.

Here are the steps I take when I redeploy if I have issues with the extensions service:

• Redeploy with new updates
• kubectl get rs -n ${streampipes_namespace}
• Delete the old record set. It will be whichever one is older.
• Delete the new backend pod to remove the volume error
• Restart the extensions service

If this works then you can repeat this in the future.

I am currently working on a new helm chart and seeing if I can resolve this issue. I started it a long time ago but haven't had the time since I had a baby shortly after opening the PR.

Hopefully I can get back to it soon as the helm chart is very outdated in its current state.

You should also note that I don't use the official chart but use my own chart.

[jormus]

hey, @heisenbergs-uncertainty thx for the info
For first testing i had a look at the hel(m) charts and the docker compose, and deployed a stack like we would use the compose up commands.
I'm really testing the capability's first , before going into depth of the platform architecture.
I need some specific points to be working and tested, because i have quite some devices / datapoints to connect .
is there already committed code for your overhaul? I will sit together with my IT infra guys later to discuss this platform, thzy will figure out the best way how this would fit in our current eco system, if it gets a go.

i keep u updated, maybe we will refactor the deployments as well, would be good to share this with the community then.