[WIP] Bootstrap etcd from consistent_index

[ahrtr]

Link to #20187

Still need to do some performance comparison (bootstrap time)

Note that this PR can also workaround #20967. Once users run into the issue, they just need to create a single-member cluster first (with --force-new-cluster), and add other members later. cc @thechristschn

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahrtr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ahrtr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

❌ Patch coverage is 77.22772% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.12%. Comparing base (7bbb4bd) to head (e644406).

Files with missing lines	Patch %	Lines
etcdutl/etcdutl/common.go	0.00%	5 Missing ⚠️
server/etcdserver/api/membership/storev2.go	16.66%	5 Missing ⚠️
server/etcdserver/bootstrap.go	90.74%	2 Missing and 3 partials ⚠️
server/storage/wal/wal.go	80.00%	4 Missing ⚠️
etcdutl/etcdutl/migrate_command.go	0.00%	2 Missing ⚠️
server/storage/storage.go	66.66%	1 Missing ⚠️
server/storage/util.go	83.33%	1 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
server/etcdserver/server.go	82.25% <100.00%> (+0.67%)	⬆️
server/storage/wal/testing/waltesting.go	60.86% <100.00%> (ø)
server/verify/verify.go	41.66% <100.00%> (+21.66%)	⬆️
server/storage/storage.go	62.79% <66.66%> (+9.84%)	⬆️
server/storage/util.go	83.56% <83.33%> (+1.98%)	⬆️
etcdutl/etcdutl/migrate_command.go	0.00% <0.00%> (ø)
server/storage/wal/wal.go	58.07% <80.00%> (+0.18%)	⬆️
etcdutl/etcdutl/common.go	0.00% <0.00%> (-46.16%)	⬇️
server/etcdserver/api/membership/storev2.go	58.82% <16.66%> (+5.88%)	⬆️
server/etcdserver/bootstrap.go	66.82% <90.74%> (+2.07%)	⬆️

... and 33 files with indirect coverage changes

@@            Coverage Diff             @@
##             main   #20981      +/-   ##
==========================================
+ Coverage   68.56%   69.12%   +0.56%     
==========================================
  Files         422      422              
  Lines       34841    34783      -58     
==========================================
+ Hits        23889    24045     +156     
+ Misses       9537     9343     -194     
+ Partials     1415     1395      -20     

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7bbb4bd...e644406. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[serathius]

The final change will be large and hard review, please plan for smaller steps so we can merge early.
For example this change of changing Open(lg, dirpath, snap) to Open(lg, dirpath, consistentIndex) seems like a correct independent change.

[ahrtr]

I will try to make the workflows green first. Afterwards, I will think about how to break down the PR, to make the review easier.

[serathius]

Sounds good, please consider marking the PR as draft until it's ready for review. You can still run tests via retest command.

[ahrtr]

marking the PR as draft

Some workflows will be ignored in draft mode.

[ahrtr]

Just added prefix [WIP] to title of the PR

[serathius]

Works for me. Thanks!

[ahrtr]

#20187 (comment)

need to revisit test cases below in next step,

• TestBreakConsistentIndexNewerThanSnapshot

[k8s-ci-robot]

@ahrtr: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-etcd-grpcproxy-integration-amd64	e644406	link	true	/test pull-etcd-grpcproxy-integration-amd64
pull-etcd-grpcproxy-integration-arm64	e644406	link	true	/test pull-etcd-grpcproxy-integration-arm64
pull-etcd-robustness-amd64	e644406	link	true	/test pull-etcd-robustness-amd64
pull-etcd-robustness-arm64	e644406	link	true	/test pull-etcd-robustness-arm64
pull-etcd-coverage-report	e644406	link	true	/test pull-etcd-coverage-report

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.