test: normalize newline handling in kubeadm YAML #22021

bobsira · 2025-12-01T18:44:25Z

In the 54 tests I've listed at the end, all the tests are failing because the golden YAML and the generated YAML differ in invisible whitespace (almost certainly line endings) on Windows, not because the actual kubeadm config content is wrong. Also the tests run successfully when I run them locally on my windows machine but fail when running on GitHub action runner

The test failures appear as the snippet below

FAIL: TestGenerateKubeadmYAMLDNS (0.16s)
    --- FAIL: TestGenerateKubeadmYAMLDNS/dns_v1.34 (0.02s)
        kubeadm_test.go:207: unexpected diff:
            --- Expected
            +++ Got

It appears that on GitHub Actions, on windows images, git commonly checks out text files with CRLF (\r\n), while: generated YAML from Go uses LF (\n). Also we might have different comparison in tests, raw string differences, no normalization. This might be producing byte-level mismatches, leading to false negative failures in CI, despite correct behavior locally.

I've added and applied a shared normalizeNewlines() helper to ensure comparisons are based on actual content differences

There are 54 tests in total that I aim to fix with this change:

kubeadm DNS tests (6)

TestGenerateKubeadmYAMLDNS/dns_v1.34
TestGenerateKubeadmYAMLDNS/dns_v1.33
TestGenerateKubeadmYAMLDNS/dns_v1.32
TestGenerateKubeadmYAMLDNS/dns_v1.31
TestGenerateKubeadmYAMLDNS/dns_v1.30
TestGenerateKubeadmYAMLDNS/dns_v1.29

kubeadm YAML tests (48)
For v1.34

TestGenerateKubeadmYAML/default_v1.34
TestGenerateKubeadmYAML/containerd_v1.34
TestGenerateKubeadmYAML/crio_v1.34
TestGenerateKubeadmYAML/options_v1.34
TestGenerateKubeadmYAML/crio-options-gates_v1.34
TestGenerateKubeadmYAML/containerd-api-port_v1.34
TestGenerateKubeadmYAML/containerd-pod-network-cidr_v1.34
TestGenerateKubeadmYAML/image-repository_v1.34

For v1.33

TestGenerateKubeadmYAML/default_v1.33
TestGenerateKubeadmYAML/containerd_v1.33
TestGenerateKubeadmYAML/crio_v1.33
TestGenerateKubeadmYAML/options_v1.33
TestGenerateKubeadmYAML/crio-options-gates_v1.33
TestGenerateKubeadmYAML/containerd-api-port_v1.33
TestGenerateKubeadmYAML/containerd-pod-network-cidr_v1.33
TestGenerateKubeadmYAML/image-repository_v1.33

For v1.32

TestGenerateKubeadmYAML/default_v1.32
TestGenerateKubeadmYAML/containerd_v1.32
TestGenerateKubeadmYAML/crio_v1.32
TestGenerateKubeadmYAML/options_v1.32
TestGenerateKubeadmYAML/crio-options-gates_v1.32
TestGenerateKubeadmYAML/containerd-api-port_v1.32
TestGenerateKubeadmYAML/containerd-pod-network-cidr_v1.32
TestGenerateKubeadmYAML/image-repository_v1.32

For v1.31

TestGenerateKubeadmYAML/default_v1.31
TestGenerateKubeadmYAML/containerd_v1.31
TestGenerateKubeadmYAML/crio_v1.31
TestGenerateKubeadmYAML/options_v1.31
TestGenerateKubeadmYAML/crio-options-gates_v1.31
TestGenerateKubeadmYAML/containerd-api-port_v1.31
TestGenerateKubeadmYAML/containerd-pod-network-cidr_v1.31
TestGenerateKubeadmYAML/image-repository_v1.31

For v1.30

TestGenerateKubeadmYAML/default_v1.30
TestGenerateKubeadmYAML/containerd_v1.30
TestGenerateKubeadmYAML/crio_v1.30
TestGenerateKubeadmYAML/options_v1.30
TestGenerateKubeadmYAML/crio-options-gates_v1.30
TestGenerateKubeadmYAML/containerd-api-port_v1.30
TestGenerateKubeadmYAML/containerd-pod-network-cidr_v1.30
TestGenerateKubeadmYAML/image-repository_v1.30

For v1.29

TestGenerateKubeadmYAML/default_v1.29
TestGenerateKubeadmYAML/containerd_v1.29
TestGenerateKubeadmYAML/crio_v1.29
TestGenerateKubeadmYAML/options_v1.29
TestGenerateKubeadmYAML/crio-options-gates_v1.29
TestGenerateKubeadmYAML/containerd-api-port_v1.29
TestGenerateKubeadmYAML/containerd-pod-network-cidr_v1.29
TestGenerateKubeadmYAML/image-repository_v1.29

k8s-ci-robot · 2025-12-01T18:44:34Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bobsira
Once this PR has been reviewed and has the lgtm label, please assign prezha for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-12-01T18:44:34Z

Hi @bobsira. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

minikube-bot · 2025-12-01T18:48:03Z

Can one of the admins verify this patch?

ComradeProgrammer · 2025-12-01T19:10:36Z

/ok-to-test

nirs

Lets understand why it fails first. I assume that the got string may have windows newlines in some cases, so if we normalize it the issue should be fixed.

nirs · 2025-12-01T20:37:55Z

pkg/minikube/bootstrapper/bsutil/kubeadm_test.go

+// normalizeNewlines converts CRLF -> LF and enforces a single trailing newline.
+func normalizeNewlines(s string) string {
+	s = strings.ReplaceAll(s, "\r\n", "\n")
+	return strings.TrimRight(s, "\n") + "\n"


I don't think we need the last trim - mostly likely the issue is generating yaml using windows newlines (CR LF) when running in the github runner, and replacing "\r\n" with "\n" will fix the issue.

nirs · 2025-12-01T20:41:55Z

pkg/minikube/bootstrapper/bsutil/kubeadm_test.go

 				if err != nil {
 					t.Fatalf("unable to read testdata: %v", err)
 				}
+				normalizedExpected := normalizeNewlines(string(expected))


This comes from line 198:

expected, err := os.ReadFile(fmt.Sprintf("testdata/%s/%s.yaml", version, tc.name))

os.ReadFile() cannot change newlines, it will break code computing a checksum of the bytes. You can check it here:
https://cs.opensource.google/go/go/+/refs/tags/go1.25.4:src/os/file.go;l=867

So what we get in expected is the same on any OS, and we don't need to normalize it.

nirs · 2025-12-01T20:44:16Z

pkg/minikube/bootstrapper/bsutil/kubeadm_test.go

 					t.Fatalf("unable to read testdata: %v", err)
 				}
+				normalizedExpected := normalizeNewlines(string(expected))
+				normalizedGot := normalizeNewlines(string(got))


This comes from line 188:

got, err := GenerateKubeadmYAML(cfg, cfg.Nodes[0], runtime)

This may generate windows newlines somehow. If this is the issue, replacing "\r\n" with "\n" will fix the issue.

You can verify this this is the issue by searching for "\r\n" before we compare and logging the result.

…YAML diffs

nirs · 2025-12-02T14:40:00Z

/ok-to-test

minikube-pr-bot · 2025-12-02T15:46:53Z

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 22021 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 41.1s    │ 40.9s                  │
│ enable ingress │ 15.9s    │ 16.0s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube (PR 22021) start: 42.8s 39.2s 40.4s 40.4s 41.5s
Times for minikube start: 40.4s 44.5s 42.1s 39.1s 39.7s

Times for minikube ingress: 15.7s 16.2s 16.2s 15.7s 15.7s
Times for minikube (PR 22021) ingress: 16.2s 15.7s 15.8s 16.2s 16.2s

docker driver with docker runtime

┌───────────────────┬──────────┬────────────────────────┐
│      COMMAND      │ MINIKUBE │ MINIKUBE  ( PR 22021 ) │
├───────────────────┼──────────┼────────────────────────┤
│ minikube start    │ 21.1s    │ 21.2s                  │
│ ⚠️  enable ingress │ 11.2s    │ 16.6s ⚠️                │
└───────────────────┴──────────┴────────────────────────┘

Times for minikube ingress: 10.6s 10.6s 11.6s 11.6s 11.6s
Times for minikube (PR 22021) ingress: 10.6s 10.6s 40.6s 10.6s 10.6s

Times for minikube start: 22.0s 20.6s 19.4s 20.4s 23.1s
Times for minikube (PR 22021) start: 19.7s 19.6s 24.1s 23.1s 19.4s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 22021 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 20.1s    │ 19.1s                  │
│ enable ingress │ 20.1s    │ 20.3s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 18.5s 22.4s 18.3s 18.6s 22.5s
Times for minikube (PR 22021) start: 20.0s 18.3s 21.5s 18.7s 16.8s

Times for minikube ingress: 20.1s 20.1s 20.1s 20.1s 20.1s
Times for minikube (PR 22021) ingress: 21.1s 20.1s 20.1s 20.1s 20.1s

bobsira · 2025-12-02T19:25:24Z

We’re applying a Git-only fix to stop newline inconsistencies in CI by adding a .gitattributes rule:

**/testdata/** text eol=lf

This tells Git to:

Treat all files in any testdata/ folder as text
Force LF (\n) line endings on checkout, even on GitHub Windows runners

This change is being tracked in this new PR -> #22026

minikube-pr-bot · 2025-12-02T20:50:39Z

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment	Test Name	Flake Rate
Docker_Linux_containerd (9 failed)	TestFunctionalNewestKubernetes/Versionv1.35.0-beta.0/parallel/DashboardCmd(gopogh)	Unknown
Docker_Linux_containerd (9 failed)	TestFunctionalNewestKubernetes/Versionv1.35.0-beta.0/parallel/ServiceCmdConnect(gopogh)	Unknown
Docker_Linux_containerd (9 failed)	TestFunctionalNewestKubernetes/Versionv1.35.0-beta.0/parallel/PersistentVolumeClaim(gopogh)	Unknown
Docker_Linux_containerd (9 failed)	TestFunctionalNewestKubernetes/Versionv1.35.0-beta.0/parallel/MySQL(gopogh)	Unknown
Docker_Linux_containerd (9 failed)	TestFunctionalNewestKubernetes/Versionv1.35.0-beta.0/parallel/TunnelCmd/serial/WaitService/Setup(gopogh)	Unknown
Docker_Linux_containerd (9 failed)	TestFunctionalNewestKubernetes/Versionv1.35.0-beta.0/parallel/TunnelCmd/serial/AccessDirect(gopogh)	Unknown
Docker_Linux_containerd (9 failed)	TestFunctional/parallel/DashboardCmd(gopogh)	0.00% (chart)
Docker_Linux_containerd (9 failed)	TestFunctional/parallel/PersistentVolumeClaim(gopogh)	0.00% (chart)
Docker_Linux_containerd (9 failed)	TestFunctional/parallel/MySQL(gopogh)	0.00% (chart)
Docker_Linux (2 failed)	TestFunctionalNewestKubernetes/Versionv1.35.0-beta.0/parallel/DashboardCmd(gopogh)	Unknown
Docker_Linux (2 failed)	TestFunctionalNewestKubernetes/Versionv1.35.0-beta.0/parallel/MySQL(gopogh)	Unknown
KVM_Linux (9 failed)	TestISOImage/PersistentMounts//data(gopogh)	Unknown
KVM_Linux (9 failed)	TestISOImage/PersistentMounts//var/lib/docker(gopogh)	Unknown
KVM_Linux (9 failed)	TestISOImage/PersistentMounts//var/lib/cni(gopogh)	Unknown
KVM_Linux (9 failed)	TestISOImage/PersistentMounts//var/lib/kubelet(gopogh)	Unknown
KVM_Linux (9 failed)	TestISOImage/PersistentMounts//var/lib/minikube(gopogh)	Unknown
KVM_Linux (9 failed)	TestISOImage/PersistentMounts//var/lib/toolbox(gopogh)	Unknown
KVM_Linux (9 failed)	TestISOImage/PersistentMounts//var/lib/boot2docker(gopogh)	Unknown
KVM_Linux (9 failed)	TestISOImage/VersionJSON(gopogh)	Unknown
KVM_Linux (9 failed)	TestISOImage/eBPFSupport(gopogh)	Unknown

Besides the following environments also have failed tests:

KVM_Linux_crio: 3 failed (gopogh)
Docker_Linux_containerd_arm64: 25 failed (gopogh)
Docker_Linux_crio_arm64: 63 failed (gopogh)
Docker_Linux_crio: 48 failed (gopogh)

To see the flake rates of all tests by environment, click here.

k8s-ci-robot · 2025-12-03T00:14:27Z

@bobsira: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
integration-docker-crio-linux-x86-64	`714686c`	link	true	`/test integration-docker-crio-linux-x86-64`
integration-docker-containerd-linux-x86-64	`714686c`	link	true	`/test integration-docker-containerd-linux-x86-64`
integration-docker-docker-linux-x86-64	`714686c`	link	true	`/test integration-docker-docker-linux-x86-64`
integration-kvm-containerd-linux-x86-64	`714686c`	link	true	`/test integration-kvm-containerd-linux-x86-64`
integration-kvm-docker-linux-x86-64	`714686c`	link	true	`/test integration-kvm-docker-linux-x86-64`
integration-none-docker-linux-x86-64	`714686c`	link	true	`/test integration-none-docker-linux-x86-64`
integration-kvm-crio-linux-x86-64	`714686c`	link	true	`/test integration-kvm-crio-linux-x86-64`
integration-docker-docker-linux-arm64	`714686c`	link	true	`/test integration-docker-docker-linux-arm64`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

normalize YAML diffs to avoid CRLF noise on Windows

6801890

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Dec 1, 2025

k8s-ci-robot requested review from ComradeProgrammer and nirs December 1, 2025 18:44

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 1, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 1, 2025

This comment has been minimized.

Sign in to view

nirs reviewed Dec 1, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

test(bsutil): add CRLF diagnostics and normalize only got in kubeadm …

714686c

…YAML diffs

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Dec 2, 2025

This comment has been minimized.

Sign in to view

test: normalize newline handling in kubeadm YAML #22021

Are you sure you want to change the base?

test: normalize newline handling in kubeadm YAML #22021

Conversation

bobsira commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Dec 1, 2025

Uh oh!

k8s-ci-robot commented Dec 1, 2025

Uh oh!

minikube-bot commented Dec 1, 2025

Uh oh!

ComradeProgrammer commented Dec 1, 2025

Uh oh!

This comment has been minimized.

nirs left a comment

Choose a reason for hiding this comment

Uh oh!

nirs Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

nirs Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

nirs Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

nirs commented Dec 2, 2025

Uh oh!

This comment has been minimized.

minikube-pr-bot commented Dec 2, 2025

Uh oh!

bobsira commented Dec 2, 2025

Uh oh!

This comment has been minimized.

minikube-pr-bot commented Dec 2, 2025

Uh oh!

k8s-ci-robot commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

bobsira commented Dec 1, 2025 •

edited

Loading