Skip to content

[next-gen] data inconsistency in table after simulated kill tikv-worker repeatedly #64666

@Lily2025

Description

@Lily2025

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、run sysbench
2、add index for two tables
3、simulated kill all tikv-worker repeatedly

  • name: "ha_tikv_worker_all_kill"
    warmUpTime: 1m
    faultType: kill
    selector: "workloadnode-tikv-worker(all)"
    period: "@every 3m"
    faultDuration: 1m
    faultTotalRunTime: 30m
    ddl:
    addIndex:
    - db: "partitioninsert_1647963896275"
    table: "sbtest1"
    timeout: 60m
    times: 7
    index_col_name: "c"
    job_type: "ingest"
    admin_check: true
    - db: "partitioninsert_1647963896275"
    table: "sbtest2"
    timeout: 60m
    times: 7
    index_col_name: "c"
    job_type: "ingest"
    admin_check: true

2. What did you expect to see? (Required)

data inconsistency in table if add index successfully

3. What did you see instead (Required)

data inconsistency in table after simulated kill tidb-worker repeatedly

mysql> admin show ddl jobs;
+--------+---------------------------------------------------+------------+-------------+----------------------+-----------+----------+-----------+----------------------------+----------------------------+----------------------------+---------------+----------+
| JOB_ID | DB_NAME | TABLE_NAME | JOB_TYPE | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | CREATE_TIME | START_TIME | END_TIME | STATE | COMMENTS |
+--------+---------------------------------------------------+------------+-------------+----------------------+-----------+----------+-----------+----------------------------+----------------------------+----------------------------+---------------+----------+
| 7408 | partitioninsert_1647963896275 | sbtest2 | add index | public | 4144 | 3889 | 0 | 2025-11-25 06:29:25.478000 | 2025-11-25 06:29:25.478000 | 2025-11-25 07:22:51.128000 | synced | |

case logs
admin check failed (Error 8223 (HY000): data inconsistency in table: sbtest2, index: index_test_1764052165225, handle: 5402730, index-values:"" != record-values:"handle: 5402730, values: [KindString 16960352122-53417868865-19761453644-60931175938-11266666891-26482692461-21719450064-35170464548-23376004353-52869623519]")
operatorLogs:
[2025-11-25 14:29:25] ###### start adding index
ALTER TABLE sbtest2 ADD INDEX index_test_1764052165225(c)
[2025-11-25 14:29:25] ###### wait for ddl job finish
[2025-11-25 15:22:53] ###### ddl job finished
select job_id, db_name, table_name, job_type, create_time, start_time, end_time, state, query from information_schema.ddl_jobs where query = 'ALTER TABLE sbtest2 ADD INDEX index_test_1764052165225(c)'
jobId: 7408, job type: add index, state: synced, comments:
add index done, it takes: 53m27.992383292s
[2025-11-25 15:22:53] ###### start admin check

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v9.0.0-beta.2.pre-810-gbf57e58
Edition: Enterprise
Git Commit Hash: bf57e58
Git Branch: HEAD
UTC Build Time: 2025-11-24 03:31:04
GoVersion: go1.23.12
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Enterprise Extension Commit Hash: dc633aae52eb11b4e3549e3a7ef5b0ed14e159b1
Kernel Type: Next Generation

Metadata

Metadata

Labels

affects-8.1This bug affects the 8.1.x(LTS) versions.affects-8.5This bug affects the 8.5.x(LTS) versions.component/DXFDistributed eXecute Framework of tidbseverity/criticaltype/bugThe issue is confirmed as a bug.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions