Re-run workflow jobs fail to scale runners

[jbuettnerbild]

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

v0.27.5

Helm Chart Version

0.23.5

CertManager Version

v1.13.1

Deployment Method

Helm

cert-manager installation

i confirm that i installed the cert-manager correctly

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  annotations:
    meta.helm.sh/release-name: action-runner-deployment-dev
    meta.helm.sh/release-namespace: github-system
  creationTimestamp: "2025-02-20T10:44:37Z"
  generation: 124789
  labels:
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: action-runner-deployment-dev
    helm.toolkit.fluxcd.io/namespace: github-system
  name: action-runner-deployment-dev
  namespace: github-system
  resourceVersion: "1328457561"
  uid: 47bba4d5-9114-4334-b570-a3840774ef80
spec:
  effectiveTime: "2025-11-14T10:58:52Z"
  replicas: 1
  selector: null
  template:
    metadata:
      annotations:
        karpenter.sh/do-not-disrupt: "true"
        karpenter.sh/do-not-evict: "true"
    spec:
      dockerdContainerResources:
        requests:
          cpu: "4"
          memory: 4Gi
      dockerdWithinRunnerContainer: true
      env:
       ...
       group: editorial
      image: spring-docker.jfrog.io/summerwind/actions-runner-dind:v2.321.0-ubuntu-20.04-8b36ea9
      imagePullPolicy: IfNotPresent
      labels:
      - self-hosted-editorial
      - self-hosted-editorial-dev
      nodeSelector:
        nmt.de/workergroup_name: gh
      organization: spring-media
      resources:
        requests:
          cpu: "4"
          memory: 4Gi
      securityContext:
        runAsUser: 1000
      serviceAccountName: action-runner-deployment-dev
      tolerations:
      - effect: NoSchedule
        key: workergroup_name
        operator: Equal
        value: gh
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  annotations:
    meta.helm.sh/release-name: action-runner-deployment-dev
    meta.helm.sh/release-namespace: github-system
  creationTimestamp: "2025-02-20T10:44:37Z"
  generation: 14148
  labels:
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: action-runner-deployment-dev
    helm.toolkit.fluxcd.io/namespace: github-system
  name: action-runner-deployment-dev
  namespace: github-system
  resourceVersion: "1328457509"
  uid: fb786ab0-9718-49c6-9e8d-9777fc9fef41
spec:
  capacityReservations:
  - effectiveTime: "2025-11-14T10:58:52Z"
    expirationTime: "2025-11-14T11:28:52Z"
    replicas: 1
  maxReplicas: 15
  minReplicas: 0
  scaleDownDelaySecondsAfterScaleOut: 300
  scaleTargetRef:
    kind: RunnerDeployment
    name: action-runner-deployment-dev
  scaleUpTriggers:
  - duration: 30m
    githubEvent:
      workflowJob: {}

To Reproduce

1. Configure HorizontalRunnerAutoscaler with scaleUpTriggers using workflowJob events
2. Run a GitHub Actions workflow with multiple jobs where some jobs fail and others succeed
3. In GitHub UI, click "Re-run failed jobs" (not "Re-run all jobs")
4. Observe that no runners are scaled up and the re-run jobs remain queued indefinitely

Important: The issue occurs when more jobs were completed successfully than failed, as all completed jobs send workflow_job.completed events simultaneously with the new workflow_job.queued event.

Describe the bug

When re-running failed jobs from a GitHub Actions workflow (without re-running all jobs), the Actions Runner Controller (ARC) webhook server receives the workflow_job.queued event but fails to scale up runners because multiple completed events from previously successful jobs in the same workflow run immediately cancel the capacity reservation.

Log Evidence
When failed jobs are re-run at 09:35:11Z, the webhook server receives all events within the same second:

Queued event (new job):

09:35:11 scaled action-runner-deployment-dev by 1
workflowJob.ID: 55390717820 (queued)
workflowJob.labels: ["self-hosted-editorial-dev"]

Completed events (from previous successful jobs in same workflow run):

09:35:11 scaled action-runner-deployment-dev by -1
workflowJob.ID: 55390717921 (completed, self-hosted-editorial-dev)

09:35:11 scaled action-runner-deployment-dev by -1
workflowJob.ID: 55390717848 (completed, self-hosted-editorial-dev)

09:35:11 scaled action-runner-deployment-dev by -1
workflowJob.ID: 55390717785 (completed, self-hosted-editorial-dev)

09:35:11 scaled action-runner-deployment-dev by -1
workflowJob.ID: 55390717933 (completed, self-hosted-editorial-dev)

09:35:11 scaled action-runner-deployment-dev by -1
workflowJob.ID: 55390717991 (completed, self-hosted-editorial-dev)

Final capacity reservation calculation:

09:35:12 Patching hra action-runner-deployment-dev for capacityReservations update
{"before": 0, "expired": 0, "added": 1, "completed": 5, "after": 0} 

Result: Net capacity = +1 -5 = -4 capped at 0 → No runner is started, workflow job remains queued indefinitely.

Root Cause
The webhook server processes all events (1x queued + 5x completed) within the same second and calculates the net capacity reservation before patching the HRA. The completed events are from jobs that were already completed in a previous workflow run (same workflowJob.runID: 19269836845), but GitHub re-sends these completed webhooks when "Re-run failed jobs" is triggered.

The webhook server treats all events equally and performs arithmetic on capacity reservations:

• Adds +1 for the new queued event
• Subtracts -1 for each completed event (5x)
• Results in net capacity of 0

Describe the expected behavior

The webhook server should either:

1. Track workflowJob.ID to prevent completed events from already-finished jobs affecting new queued events
2. Ignore completed events that were sent as part of a workflow re-run (same workflowJob.runID but older workflowJob.ID)
3. Process queued events with higher priority or separate from completed events
4. Maintain a grace period where new queued events are protected from immediate cancellation by completed events

Whole Controller Logs

actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 0 HRAs by key	{"key": "spring-media/lean-preview-server"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found some runner groups are managed by ARC	{"event": "workflow_job", "hookID": "403136573", "delivery": "369cfff0-c13d-11f0-9a49-73451605fcb8", "workflowJob.status": "queued", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "queued", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717820, "groups": "RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}, RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	groups	{"event": "workflow_job", "hookID": "403136573", "delivery": "369cfff0-c13d-11f0-9a49-73451605fcb8", "workflowJob.status": "queued", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "queued", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717820, "groups": "RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}, RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 2 HRAs by key	{"key": "spring-media/group/editorial"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	job scale up target found	{"event": "workflow_job", "hookID": "403136573", "delivery": "369cfff0-c13d-11f0-9a49-73451605fcb8", "workflowJob.status": "queued", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "queued", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717820, "enterprise": "axelspringer", "organization": "spring-media", "repository": "lean-preview-server", "key": "spring-media/group/editorial"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	INFO	controllers.webhookbasedautoscaler	scaled action-runner-deployment-dev by 1	{"event": "workflow_job", "hookID": "403136573", "delivery": "369cfff0-c13d-11f0-9a49-73451605fcb8", "workflowJob.status": "queued", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "queued", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717820}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 0 HRAs by key	{"key": "spring-media/lean-preview-server"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found some runner groups are managed by ARC	{"event": "workflow_job", "hookID": "403136573", "delivery": "36918e40-c13d-11f0-8389-49eb6889155b", "workflowJob.status": "completed", "workflowJob.labels": ["ubuntu-latest"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717569, "groups": "RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}, RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	groups	{"event": "workflow_job", "hookID": "403136573", "delivery": "36918e40-c13d-11f0-8389-49eb6889155b", "workflowJob.status": "completed", "workflowJob.labels": ["ubuntu-latest"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717569, "groups": "RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}, RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 2 HRAs by key	{"key": "spring-media/group/editorial"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 2 HRAs by key	{"key": "spring-media/group/editorial"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	no repository/organizational/enterprise runner found	{"event": "workflow_job", "hookID": "403136573", "delivery": "36918e40-c13d-11f0-8389-49eb6889155b", "workflowJob.status": "completed", "workflowJob.labels": ["ubuntu-latest"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717569, "repository": "spring-media/lean-preview-server", "organization": "spring-media", "enterprise": "axelspringer"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Scale target not found. If this is unexpected, ensure that there is exactly one repository-wide or organizational runner deployment that matches this webhook event. If --watch-namespace is set ensure this is configured correctly.	{"event": "workflow_job", "hookID": "403136573", "delivery": "36918e40-c13d-11f0-8389-49eb6889155b", "workflowJob.status": "completed", "workflowJob.labels": ["ubuntu-latest"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717569}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 0 HRAs by key	{"key": "spring-media/lean-preview-server"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found some runner groups are managed by ARC	{"event": "workflow_job", "hookID": "403136573", "delivery": "369537c0-c13d-11f0-924c-218e4813d67f", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717636, "groups": "RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}, RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	groups	{"event": "workflow_job", "hookID": "403136573", "delivery": "369537c0-c13d-11f0-924c-218e4813d67f", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717636, "groups": "RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}, RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 2 HRAs by key	{"key": "spring-media/group/editorial"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	job scale up target found	{"event": "workflow_job", "hookID": "403136573", "delivery": "369537c0-c13d-11f0-924c-218e4813d67f", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717636, "enterprise": "axelspringer", "organization": "spring-media", "repository": "lean-preview-server", "key": "spring-media/group/editorial"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	INFO	controllers.webhookbasedautoscaler	scaled action-runner-deployment by -1	{"event": "workflow_job", "hookID": "403136573", "delivery": "369537c0-c13d-11f0-924c-218e4813d67f", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717636}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 0 HRAs by key	{"key": "spring-media/lean-preview-server"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found some runner groups are managed by ARC	{"event": "workflow_job", "hookID": "403136573", "delivery": "36ad0580-c13d-11f0-9240-21a6bae2e175", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717921, "groups": "RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}, RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	groups	{"event": "workflow_job", "hookID": "403136573", "delivery": "36ad0580-c13d-11f0-9240-21a6bae2e175", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717921, "groups": "RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}, RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 2 HRAs by key	{"key": "spring-media/group/editorial"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	job scale up target found	{"event": "workflow_job", "hookID": "403136573", "delivery": "36ad0580-c13d-11f0-9240-21a6bae2e175", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717921, "enterprise": "axelspringer", "organization": "spring-media", "repository": "lean-preview-server", "key": "spring-media/group/editorial"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	INFO	controllers.webhookbasedautoscaler	scaled action-runner-deployment-dev by -1	{"event": "workflow_job", "hookID": "403136573", "delivery": "36ad0580-c13d-11f0-9240-21a6bae2e175", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717921}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 0 HRAs by key	{"key": "spring-media/lean-preview-server"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found some runner groups are managed by ARC	{"event": "workflow_job", "hookID": "403136573", "delivery": "36a58b70-c13d-11f0-8674-5fd99cc19cc8", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717848, "groups": "RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}, RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	groups	{"event": "workflow_job", "hookID": "403136573", "delivery": "36a58b70-c13d-11f0-8674-5fd99cc19cc8", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717848, "groups": "RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}, RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 2 HRAs by key	{"key": "spring-media/group/editorial"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	job scale up target found	{"event": "workflow_job", "hookID": "403136573", "delivery": "36a58b70-c13d-11f0-8674-5fd99cc19cc8", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717848, "enterprise": "axelspringer", "organization": "spring-media", "repository": "lean-preview-server", "key": "spring-media/group/editorial"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	INFO	controllers.webhookbasedautoscaler	scaled action-runner-deployment-dev by -1	{"event": "workflow_job", "hookID": "403136573", "delivery": "36a58b70-c13d-11f0-8674-5fd99cc19cc8", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717848}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 0 HRAs by key	{"key": "spring-media/lean-preview-server"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found some runner groups are managed by ARC	{"event": "workflow_job", "hookID": "403136573", "delivery": "369a8ef0-c13d-11f0-9c88-59d8274de87b", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717785, "groups": "RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}, RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	groups	{"event": "workflow_job", "hookID": "403136573", "delivery": "369a8ef0-c13d-11f0-9c88-59d8274de87b", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717785, "groups": "RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}, RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 2 HRAs by key	{"key": "spring-media/group/editorial"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	job scale up target found	{"event": "workflow_job", "hookID": "403136573", "delivery": "369a8ef0-c13d-11f0-9c88-59d8274de87b", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717785, "enterprise": "axelspringer", "organization": "spring-media", "repository": "lean-preview-server", "key": "spring-media/group/editorial"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	INFO	controllers.webhookbasedautoscaler	scaled action-runner-deployment-dev by -1	{"event": "workflow_job", "hookID": "403136573", "delivery": "369a8ef0-c13d-11f0-9c88-59d8274de87b", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717785}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 0 HRAs by key	{"key": "spring-media/lean-preview-server"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found some runner groups are managed by ARC	{"event": "workflow_job", "hookID": "403136573", "delivery": "36ae3e00-c13d-11f0-8e34-a6543708a794", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717933, "groups": "RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}, RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	groups	{"event": "workflow_job", "hookID": "403136573", "delivery": "36ae3e00-c13d-11f0-8e34-a6543708a794", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717933, "groups": "RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}, RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 2 HRAs by key	{"key": "spring-media/group/editorial"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	job scale up target found	{"event": "workflow_job", "hookID": "403136573", "delivery": "36ae3e00-c13d-11f0-8e34-a6543708a794", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717933, "enterprise": "axelspringer", "organization": "spring-media", "repository": "lean-preview-server", "key": "spring-media/group/editorial"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	INFO	controllers.webhookbasedautoscaler	scaled action-runner-deployment-dev by -1	{"event": "workflow_job", "hookID": "403136573", "delivery": "36ae3e00-c13d-11f0-8e34-a6543708a794", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717933}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 0 HRAs by key	{"key": "spring-media/lean-preview-server"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found some runner groups are managed by ARC	{"event": "workflow_job", "hookID": "403136573", "delivery": "36b4a6a0-c13d-11f0-879c-a1462b511ef3", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717991, "groups": "RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}, RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	groups	{"event": "workflow_job", "hookID": "403136573", "delivery": "36b4a6a0-c13d-11f0-879c-a1462b511ef3", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717991, "groups": "RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}, RunnerGroup{Scope:Organization, Kind:Custom, Name:editorial}"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	Found 2 HRAs by key	{"key": "spring-media/group/editorial"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	DEBUG	controllers.webhookbasedautoscaler	job scale up target found	{"event": "workflow_job", "hookID": "403136573", "delivery": "36b4a6a0-c13d-11f0-879c-a1462b511ef3", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717991, "enterprise": "axelspringer", "organization": "spring-media", "repository": "lean-preview-server", "key": "spring-media/group/editorial"}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:11Z	INFO	controllers.webhookbasedautoscaler	scaled action-runner-deployment-dev by -1	{"event": "workflow_job", "hookID": "403136573", "delivery": "36b4a6a0-c13d-11f0-879c-a1462b511ef3", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted-editorial-dev"], "repository.name": "lean-preview-server", "repository.owner.login": "spring-media", "repository.owner.type": "Organization", "enterprise.slug": "axelspringer", "action": "completed", "workflowJob.runID": 19269836845, "workflowJob.ID": 55390717991}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:12Z	DEBUG	controllers.webhookbasedautoscaler	Patching hra action-runner-deployment-dev for capacityReservations update	{"before": 0, "expired": 0, "added": 1, "completed": 5, "after": 0}
actions-runner-controller-github-webhook-server-b5d8df855-c7wpd github-webhook-server 2025-11-14T09:35:12Z	DEBUG	controllers.webhookbasedautoscaler	Patching hra action-runner-deployment for capacityReservations update	{"before": 0, "expired": 0, "added": 0, "completed": 1, "after": 0}

Whole Runner Pod Logs

not needed for this here.

Additional Context

Workaround Attempted
Setting scaleDownDelaySecondsAfterScaleOut: 300 does not solve the issue, as it only prevents runner pod scale-down, not the capacity reservation cancellation by completed events.

GitHub Webhook Configuration

• Webhook events: workflow_job (all action types)
• GitHub does not support filtering webhook events by action type (queued, completed, etc.)
• All webhook deliveries show 200 OK responses
• Events are delivered correctly to the webhook server

[jbuettnerbild]