Hybrid search on multi struct array field,is it possiple #45779

avayayu · 2025-11-23T08:34:18Z

avayayu
Nov 23, 2025

my schema like on the picture above.,and my milvus is v2.6.4.

then I do hybrid search like this:

search_param_1 = {
                "data": [query_vector_image],
                "anns_field": "images[image_vector]",
                "param": {"nprobe": 10},
                #"param": {"metric_type": "MAX_SIM_L2"},
                 "limit": 10
            }
search_request.append(AnnSearchRequest(**search_param_1))

search_param_2 = {
                "data": [query_vector_feature],
                "anns_field": "features[feature_vector]",
                "param": {"nprobe": 10},
               # "param": {"metric_type": "MAX_SIM_L2"},
                "limit": 10
            }
   
search_request.append(AnnSearchRequest(**search_param_2))
ranker = Function(
        name="rrf",
        input_field_names=[],  # Must be an empty list
        function_type=FunctionType.RERANK,
        params={
            "reranker": "rrf",
            "k": 100  # Optional
        }
    )
res = milvus_client.hybrid_search(
            collection_name='test3',
            reqs=search_request,
            ranker=ranker,
            limit=5,
            output_fields=["item_id"]
)

but i always get errors like this:

pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2314] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266

can I do hybrid search like this?

xiaofan-luan · 2025-11-24T00:41:08Z

xiaofan-luan
Nov 24, 2025
Maintainer

I think you might need to create index on image_vector field and future_vector field.

@alwayslove2013 we probably need to reject search on non indexed multi vector field. and also support flat index.
Just like traditional vector index

5 replies

avayayu Nov 24, 2025
Author

you mean autoindex index is not an index? image_vector and feature_vector field have an autoindex already!

xiaofan-luan Nov 24, 2025
Maintainer

can you offer full log of all your querynode?

it seems to be right but from the error stack it falls into SearchBruteForce.cpp which means it is running on bruteforce mode

avayayu Nov 24, 2025
Author

[2025/11/24 01:04:32.869 +00:00] [INFO] [observers/target_observer.go:526] ["Update readable segment version"] [collectionID=462141493833216217] [channelName=by-dev-rootcoord-dml_8_462141493833216217v0] [nodeID=3] [oldVersion=1763946252873048737] [newVersion=1763946262870787995]
[2025/11/24 01:04:32.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=d40281f6b3616d73027dd082ce85838f] [collectionID=462141493836587714] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946262870769658] [checkPoint=2025/11/24 01:03:20.000 +00:00] [deleteCP=2025/11/24 01:03:20.000 +00:00] [partitions="[462141493836587715]"]
[2025/11/24 01:04:32.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=ca963ebfbda99213c9c3b7ed8c797313] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946262870800785] [checkPoint=2025/11/24 00:58:14.400 +00:00] [deleteCP=2025/11/24 00:58:14.400 +00:00] [partitions="[462141493836954626]"]
[2025/11/24 01:04:32.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=fd644b9fb3737fa50fd0199b04a808fd] [collectionID=462141493833216217] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946262870787995] [checkPoint=2025/11/24 01:03:45.200 +00:00] [deleteCP=2025/11/24 01:03:45.200 +00:00] [partitions="[462141493833216218]"]
[2025/11/24 01:04:32.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=8a5f6da0ee69a58a221d52a76b16a9f6] [collectionID=462141493833572742] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946262870750326] [checkPoint=2025/11/24 01:03:22.600 +00:00] [deleteCP=2025/11/24 01:03:22.600 +00:00] [partitions="[462141493833572743]"]
[2025/11/24 01:04:32.870 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:04:32.870 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=15] [totalSealedRowCount=15] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:04:32.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=0ea36f1ede740bfb525bded9dddf07e9] [collectionID=462141493833627568] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946262870777617] [checkPoint=2025/11/24 01:04:08.600 +00:00] [deleteCP=2025/11/24 01:04:08.600 +00:00] [partitions="[462141493833627569]"]
[2025/11/24 01:04:32.871 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [partitions="[462141493836587715]"] [oldVersion=1763946252872666607] [newVersion=1763946262870769658] [serviceable=true] [loadedRatio=1] [growingSegmentNum=1] [sealedSegmentNum=0]
[2025/11/24 01:04:32.871 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=a267e1fdb900adb1a02eaf2edc635a40] [collectionID=462141493836581225] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946262870674518] [checkPoint=2025/11/24 01:03:33.600 +00:00] [deleteCP=2025/11/24 01:03:33.600 +00:00] [partitions="[462141493836581226]"]
[2025/11/24 01:04:32.871 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [partitions="[462141493833216218]"] [oldVersion=1763946252873048737] [newVersion=1763946262870787995] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=1]
[2025/11/24 01:04:32.871 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:04:32.871 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=4] [totalSealedRowCount=4] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:04:32.871 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [partitions="[462141493836954626]"] [oldVersion=1763946252872854709] [newVersion=1763946262870800785] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=0]
[2025/11/24 01:04:32.871 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=2] [totalSealedRowCount=2] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:04:32.871 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:04:32.871 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [partitions="[462141493833627569]"] [oldVersion=1763946252873101326] [newVersion=1763946262870777617] [serviceable=true] [loadedRatio=1] [growingSegmentNum=1] [sealedSegmentNum=1]
[2025/11/24 01:04:32.871 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [partitions="[462141493836581226]"] [oldVersion=1763946252873037232] [newVersion=1763946262870674518] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=0]
[2025/11/24 01:04:32.871 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [partitions="[462141493833572743]"] [oldVersion=1763946252872882346] [newVersion=1763946262870750326] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=1]
[2025/11/24 01:04:32.871 +00:00] [INFO] [observers/target_observer.go:562] ["observer trigger update current target"] [collectionID=462141493836587714]
[2025/11/24 01:04:32.871 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836587714] [partitionIDs="[]"]
[2025/11/24 01:04:32.871 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833216217] [partitionIDs="[]"]
[2025/11/24 01:04:32.871 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836954625] [partitionIDs="[]"]
[2025/11/24 01:04:32.871 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833627568] [partitionIDs="[]"]
[2025/11/24 01:04:32.871 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836581225] [partitionIDs="[]"]
[2025/11/24 01:04:32.871 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836587714] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=1] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:32.871 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833216217] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [numOfSegments=1] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:32.871 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [posTs=462407928381440001] [posTime=2025/11/24 01:04:20.000 +00:00]
[2025/11/24 01:04:32.871 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833572742] [partitionIDs="[]"]
[2025/11/24 01:04:32.871 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [posTs=462407919258828816] [posTime=2025/11/24 01:03:45.200 +00:00]
[2025/11/24 01:04:32.871 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:32.871 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836587714] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] ["# of unflushed segments"=1] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:20.000 +00:00] ["# of delete check point"=2025/11/24 01:04:20.000 +00:00]
[2025/11/24 01:04:32.871 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836581225] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [numOfSegments=0] ["result flushed"=0] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:32.872 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [posTs=462407832541593619] [posTime=2025/11/24 00:58:14.400 +00:00]
[2025/11/24 01:04:32.871 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833627568] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [numOfSegments=2] ["result flushed"=1] ["result growing"=1] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:32.872 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833216217] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:03:45.200 +00:00] ["# of delete check point"=2025/11/24 01:03:45.200 +00:00]
[2025/11/24 01:04:32.872 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [posTs=462407916217958407] [posTime=2025/11/24 01:03:33.600 +00:00]
[2025/11/24 01:04:32.872 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836954625] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] ["# of unflushed segments"=0] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 00:58:14.400 +00:00] ["# of delete check point"=2025/11/24 00:58:14.400 +00:00]
[2025/11/24 01:04:32.872 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [posTs=462407925392998411] [posTime=2025/11/24 01:04:08.600 +00:00]
[2025/11/24 01:04:32.872 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836581225] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] ["# of unflushed segments"=0] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:03:33.600 +00:00] ["# of delete check point"=2025/11/24 01:03:33.600 +00:00]
[2025/11/24 01:04:32.872 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833572742] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [numOfSegments=1] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:32.872 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833627568] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] ["# of unflushed segments"=1] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:08.600 +00:00] ["# of delete check point"=2025/11/24 01:04:08.600 +00:00]
[2025/11/24 01:04:32.872 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [posTs=462407929063014407] [posTime=2025/11/24 01:04:22.600 +00:00]
[2025/11/24 01:04:32.872 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833572742] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:22.600 +00:00] ["# of delete check point"=2025/11/24 01:04:22.600 +00:00]
[2025/11/24 01:04:34.074 +00:00] [INFO] [datacoord/meta.go:1978] ["UpdateChannelCheckpoint done"] [traceID=6b9b05f249ff8612c19b02f5b8e8c194] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [ts=462407931946598412] [time=2025/11/24 01:04:33.600 +00:00]
[2025/11/24 01:04:34.074 +00:00] [INFO] [datacoord/segment_manager.go:599] ["try remove empty sealed segment after channel cp updated failed to get channel"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0]
[2025/11/24 01:04:34.074 +00:00] [INFO] [recovery/recovery_storage_impl.go:124] ["update flush checkpoint"] [module=streamingnode] [nodeID=3] [component=recovery-storage] [channel=by-dev-rootcoord-dml_15:rw@3] [state=working] [vchannel=by-dev-rootcoord-dml_15_462141493836581225v0] [messageID=462141496768593923] [timeTick=462407931946598412]
[2025/11/24 01:04:35.169 +00:00] [INFO] [balance/score_based_balancer.go:644] ["node channel workload status"] [collectionID=462141493836587714] [replicaID=462141493833891857] [nodes="["{NodeID: 3, AssignedScore: 15.000000, CurrentScore: 15.000000, Priority: 0}"]"]
[2025/11/24 01:04:38.940 +00:00] [INFO] [proxy/impl.go:5897] ["connect received"] [traceID=d53234342f87a4d1d7a809585fb2c023] [sdk_type=Python] [sdk_version=2.6.3] [local_time="2025-11-24 09:04:38.937547"] [user=] [host=llm-server] [db=default]
[2025/11/24 01:04:38.941 +00:00] [INFO] [rootcoord/root_coord.go:914] ["received request to list databases"] [traceID=d53234342f87a4d1d7a809585fb2c023] [msgID=0]
[2025/11/24 01:04:38.941 +00:00] [INFO] [rootcoord/root_coord.go:940] ["done to list databases"] [traceID=d53234342f87a4d1d7a809585fb2c023] [msgID=0] ["num of databases"=1]
[2025/11/24 01:04:38.942 +00:00] [INFO] [connection/manager.go:106] ["client register"] [traceID=d53234342f87a4d1d7a809585fb2c023] [sdk_type=Python] [sdk_version=2.6.3] [local_time="2025-11-24 09:04:38.937547"] [user=] [host=llm-server] [identifier=462407933335961602] [last_active_time=2025/11/24 01:04:38.942 +00:00]
[2025/11/24 01:04:41.469 +00:00] [INFO] [balance/score_based_balancer.go:644] ["node channel workload status"] [collectionID=462141493833627568] [replicaID=462141493833891852] [nodes="["{NodeID: 3, AssignedScore: 15.000000, CurrentScore: 15.000000, Priority: 0}"]"]
W20251124 01:04:42.458482 6547 emb_list_utils.h:112] [KNOWHERE][find_min_in_range][knowhere_search] invalid range, start_idx: 0, end_idx: 0
W20251124 01:04:42.458577 6547 brute_force.cc:290] [KNOWHERE][operator()][knowhere_search] get_sum_max_sim failed, num_query_vectors: 1, num_base_vectors: 0
=> Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266

I20251124 01:04:42.458734 8658 thread_pool.h:67] [SERVER][operator()][knowhere_search][]Successfully set priority of knowhere thread.
[2025/11/24 01:04:42.460 +00:00] [WARN] [segments/segment.go:611] ["Search failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [mvcc=462407934253203468] [collectionID=462141493836954625] [segmentID=462141493829685674] [segmentType=Growing] [withIndex=false]
[2025/11/24 01:04:42.460 +00:00] [WARN] [querynodev2/services.go:803] ["failed to search segments"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [msgID=0] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [scope=Streaming] [error="N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2358] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:42.461 +00:00] [WARN] [delegator/delegator.go:849] ["failed to execute sub task"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [replicaID=462141493833891858] [taskType=Search] [nodeID=3] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2358] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:42.461 +00:00] [WARN] [delegator/delegator.go:875] ["some tasks failed to complete"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [replicaID=462141493833891858] [taskType=Search] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2358] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:42.461 +00:00] [WARN] [delegator/delegator.go:364] ["Delegator search failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [replicaID=462141493833891858] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2358] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:42.461 +00:00] [WARN] [querynodev2/handlers.go:423] ["failed to search on delegator"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [msgID=462141493828439172] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [scope=All] [nq=1] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2358] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:42.461 +00:00] [WARN] [proxy/task_search.go:940] ["QueryNode search result error"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collection=462141493836954625] [partitionIDs="[]"] [nodeID=3] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [reason="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2358] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:42.461 +00:00] [WARN] [proxy/lb_policy.go:266] ["search/query channel failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channelName=by-dev-rootcoord-dml_5_462141493836954625v0] [nodeID=3] [error="fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2358] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "] [errorVerbose="fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2358] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n | \t/workspace/source/internal/proxy/task_search.go:942\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:264\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:145\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:283\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:313\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:93\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) fail to search on QueryNode 3\nWraps: (3) worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2358] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n |\n | Stack trace: \nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2025/11/24 01:04:42.461 +00:00] [WARN] [retry/retry.go:147] ["retry func failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [retried=0] [caller=/workspace/source/internal/proxy/lb_policy.go:283] [error="failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0: fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2358] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\nStack trace: "] [errorVerbose="failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0: fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2358] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:270\n | [...repeated from below...]\nWraps: (2) failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n | \t/workspace/source/internal/proxy/task_search.go:942\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:264\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:145\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:283\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:313\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:93\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (4) fail to search on QueryNode 3\nWraps: (5) worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2358] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n |\n | Stack trace: \nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) merr.milvusError"]
[2025/11/24 01:04:42.662 +00:00] [WARN] [proxy/lb_policy.go:171] ["failed to select shard"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channelName=by-dev-rootcoord-dml_5_462141493836954625v0] [excluded="[3]"] [candidates="<NodeID: 3, serviceable: true, address: 192.168.100.12:21123>"] [serviceableNodes=] [error="no available shard leaders: channel not available[channel=by-dev-rootcoord-dml_5_462141493836954625v0]"] [errorVerbose="no available shard leaders: channel not available[channel=by-dev-rootcoord-dml_5_462141493836954625v0]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/v2/util/merr.warpChannelErr\n | \t/workspace/source/pkg/util/merr/utils.go:712\n | github.com/milvus-io/milvus/pkg/v2/util/merr.WrapErrChannelNotAvailable\n | \t/workspace/source/pkg/util/merr/utils.go:734\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).selectNode.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:189\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).selectNode\n | \t/workspace/source/internal/proxy/lb_policy.go:215\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:238\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:145\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:283\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:313\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:93\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) no available shard leaders\nWraps: (3) channel not available[channel=by-dev-rootcoord-dml_5_462141493836954625v0]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2025/11/24 01:04:42.663 +00:00] [WARN] [proxy/lb_policy.go:156] ["all replicas are excluded after refresh shard leader cache, clear it and try to select node"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channelName=by-dev-rootcoord-dml_5_462141493836954625v0]
W20251124 01:04:42.664639 8658 emb_list_utils.h:112] [KNOWHERE][find_min_in_range][knowhere_search] invalid range, start_idx: 0, end_idx: 0
W20251124 01:04:42.664734 8658 brute_force.cc:290] [KNOWHERE][operator()][knowhere_search] get_sum_max_sim failed, num_query_vectors: 1, num_base_vectors: 0
=> Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266

[2025/11/24 01:04:42.665 +00:00] [WARN] [segments/segment.go:611] ["Search failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [mvcc=462407934305894412] [collectionID=462141493836954625] [segmentID=462141493829685674] [segmentType=Growing] [withIndex=false]
[2025/11/24 01:04:42.665 +00:00] [WARN] [querynodev2/services.go:803] ["failed to search segments"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [msgID=0] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [scope=Streaming] [error="N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2362] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:42.665 +00:00] [WARN] [delegator/delegator.go:849] ["failed to execute sub task"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [replicaID=462141493833891858] [taskType=Search] [nodeID=3] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2362] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:42.665 +00:00] [WARN] [delegator/delegator.go:875] ["some tasks failed to complete"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [replicaID=462141493833891858] [taskType=Search] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2362] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:42.665 +00:00] [WARN] [delegator/delegator.go:364] ["Delegator search failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [replicaID=462141493833891858] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2362] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:42.665 +00:00] [WARN] [querynodev2/handlers.go:423] ["failed to search on delegator"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [msgID=462141493828439172] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [scope=All] [nq=1] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2362] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:42.665 +00:00] [WARN] [proxy/task_search.go:940] ["QueryNode search result error"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collection=462141493836954625] [partitionIDs="[]"] [nodeID=3] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [reason="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2362] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:42.665 +00:00] [WARN] [proxy/lb_policy.go:266] ["search/query channel failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channelName=by-dev-rootcoord-dml_5_462141493836954625v0] [nodeID=3] [error="fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2362] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "] [errorVerbose="fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2362] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n | \t/workspace/source/internal/proxy/task_search.go:942\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:264\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:145\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:283\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:313\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:93\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) fail to search on QueryNode 3\nWraps: (3) worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2362] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n |\n | Stack trace: \nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2025/11/24 01:04:42.869 +00:00] [INFO] [observers/target_observer.go:526] ["Update readable segment version"] [collectionID=462141493833216217] [channelName=by-dev-rootcoord-dml_8_462141493833216217v0] [nodeID=3] [oldVersion=1763946262870787995] [newVersion=1763946272872158049]
[2025/11/24 01:04:42.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=7a16a7a930a12b585d3e661ebecf2539] [collectionID=462141493833216217] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946272872158049] [checkPoint=2025/11/24 01:03:45.200 +00:00] [deleteCP=2025/11/24 01:03:45.200 +00:00] [partitions="[462141493833216218]"]
[2025/11/24 01:04:42.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=4f5e4a416715fac1bb45dde6f3e34449] [collectionID=462141493836581225] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946272872228397] [checkPoint=2025/11/24 01:03:33.600 +00:00] [deleteCP=2025/11/24 01:03:33.600 +00:00] [partitions="[462141493836581226]"]
[2025/11/24 01:04:42.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=91cb515ae8bebb58cfb142bf14e6cb25] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946272872180435] [checkPoint=2025/11/24 00:58:14.400 +00:00] [deleteCP=2025/11/24 00:58:14.400 +00:00] [partitions="[462141493836954626]"]
[2025/11/24 01:04:42.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=edd638b97c5bff97fb97c61d249f5166] [collectionID=462141493836587714] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946272872050184] [checkPoint=2025/11/24 01:04:20.000 +00:00] [deleteCP=2025/11/24 01:04:20.000 +00:00] [partitions="[462141493836587715]"]
[2025/11/24 01:04:42.870 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:04:42.870 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=15] [totalSealedRowCount=15] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:04:42.870 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [partitions="[462141493836581226]"] [oldVersion=1763946262870674518] [newVersion=1763946272872228397] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=0]
[2025/11/24 01:04:42.870 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [partitions="[462141493833216218]"] [oldVersion=1763946262870787995] [newVersion=1763946272872158049] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=1]
[2025/11/24 01:04:42.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=72c9ca13b5395d122573b8122d5d0b93] [collectionID=462141493833572742] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946272872345008] [checkPoint=2025/11/24 01:04:22.600 +00:00] [deleteCP=2025/11/24 01:04:22.600 +00:00] [partitions="[462141493833572743]"]
[2025/11/24 01:04:42.870 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:04:42.870 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:04:42.871 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [partitions="[462141493836954626]"] [oldVersion=1763946262870800785] [newVersion=1763946272872180435] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=0]
[2025/11/24 01:04:42.871 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [partitions="[462141493836587715]"] [oldVersion=1763946262870769658] [newVersion=1763946272872050184] [serviceable=true] [loadedRatio=1] [growingSegmentNum=1] [sealedSegmentNum=0]
[2025/11/24 01:04:42.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=a07e0cd56edb2453eb9f5c2f760e518f] [collectionID=462141493833627568] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946272872279727] [checkPoint=2025/11/24 01:04:08.600 +00:00] [deleteCP=2025/11/24 01:04:08.600 +00:00] [partitions="[462141493833627569]"]
[2025/11/24 01:04:42.871 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=2] [totalSealedRowCount=2] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:04:42.871 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=4] [totalSealedRowCount=4] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:04:42.871 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [partitions="[462141493833572743]"] [oldVersion=1763946262870750326] [newVersion=1763946272872345008] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=1]
[2025/11/24 01:04:42.871 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [partitions="[462141493833627569]"] [oldVersion=1763946262870777617] [newVersion=1763946272872279727] [serviceable=true] [loadedRatio=1] [growingSegmentNum=1] [sealedSegmentNum=1]
[2025/11/24 01:04:42.871 +00:00] [INFO] [observers/target_observer.go:562] ["observer trigger update current target"] [collectionID=462141493836581225]
[2025/11/24 01:04:42.871 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836581225] [partitionIDs="[]"]
[2025/11/24 01:04:42.871 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836954625] [partitionIDs="[]"]
[2025/11/24 01:04:42.871 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833216217] [partitionIDs="[]"]
[2025/11/24 01:04:42.871 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836587714] [partitionIDs="[]"]
[2025/11/24 01:04:42.871 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833572742] [partitionIDs="[]"]
[2025/11/24 01:04:42.871 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833627568] [partitionIDs="[]"]
[2025/11/24 01:04:42.871 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836581225] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [numOfSegments=0] ["result flushed"=0] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:42.871 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [posTs=462407931946598412] [posTime=2025/11/24 01:04:33.600 +00:00]
[2025/11/24 01:04:42.871 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836581225] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] ["# of unflushed segments"=0] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:33.600 +00:00] ["# of delete check point"=2025/11/24 01:04:33.600 +00:00]
[2025/11/24 01:04:42.871 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:42.871 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833572742] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [numOfSegments=1] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:42.871 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [posTs=462407832541593619] [posTime=2025/11/24 00:58:14.400 +00:00]
[2025/11/24 01:04:42.871 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836954625] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] ["# of unflushed segments"=0] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 00:58:14.400 +00:00] ["# of delete check point"=2025/11/24 00:58:14.400 +00:00]
[2025/11/24 01:04:42.871 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [posTs=462407929063014407] [posTime=2025/11/24 01:04:22.600 +00:00]
[2025/11/24 01:04:42.872 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836587714] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=1] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:42.872 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833572742] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:22.600 +00:00] ["# of delete check point"=2025/11/24 01:04:22.600 +00:00]
[2025/11/24 01:04:42.872 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833627568] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [numOfSegments=2] ["result flushed"=1] ["result growing"=1] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:42.872 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [posTs=462407928381440001] [posTime=2025/11/24 01:04:20.000 +00:00]
[2025/11/24 01:04:42.872 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836587714] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] ["# of unflushed segments"=1] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:20.000 +00:00] ["# of delete check point"=2025/11/24 01:04:20.000 +00:00]
[2025/11/24 01:04:42.872 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833216217] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [numOfSegments=1] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:42.872 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [posTs=462407925392998411] [posTime=2025/11/24 01:04:08.600 +00:00]
[2025/11/24 01:04:42.872 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [posTs=462407919258828816] [posTime=2025/11/24 01:03:45.200 +00:00]
[2025/11/24 01:04:42.872 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833627568] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] ["# of unflushed segments"=1] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:08.600 +00:00] ["# of delete check point"=2025/11/24 01:04:08.600 +00:00]
[2025/11/24 01:04:42.872 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833216217] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:03:45.200 +00:00] ["# of delete check point"=2025/11/24 01:03:45.200 +00:00]
[2025/11/24 01:04:43.067 +00:00] [WARN] [proxy/lb_policy.go:171] ["failed to select shard"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channelName=by-dev-rootcoord-dml_5_462141493836954625v0] [excluded="[3]"] [candidates="<NodeID: 3, serviceable: true, address: 192.168.100.12:21123>"] [serviceableNodes=] [error="no available shard leaders: channel not available[channel=by-dev-rootcoord-dml_5_462141493836954625v0]"] [errorVerbose="no available shard leaders: channel not available[channel=by-dev-rootcoord-dml_5_462141493836954625v0]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/v2/util/merr.warpChannelErr\n | \t/workspace/source/pkg/util/merr/utils.go:712\n | github.com/milvus-io/milvus/pkg/v2/util/merr.WrapErrChannelNotAvailable\n | \t/workspace/source/pkg/util/merr/utils.go:734\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).selectNode.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:189\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).selectNode\n | \t/workspace/source/internal/proxy/lb_policy.go:215\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:238\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:145\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:283\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:313\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:93\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) no available shard leaders\nWraps: (3) channel not available[channel=by-dev-rootcoord-dml_5_462141493836954625v0]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2025/11/24 01:04:43.068 +00:00] [WARN] [proxy/lb_policy.go:156] ["all replicas are excluded after refresh shard leader cache, clear it and try to select node"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channelName=by-dev-rootcoord-dml_5_462141493836954625v0]
W20251124 01:04:43.069484 6547 emb_list_utils.h:112] [KNOWHERE][find_min_in_range][knowhere_search] invalid range, start_idx: 0, end_idx: 0
W20251124 01:04:43.069561 6547 brute_force.cc:290] [KNOWHERE][operator()][knowhere_search] get_sum_max_sim failed, num_query_vectors: 1, num_base_vectors: 0
=> Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266

[2025/11/24 01:04:43.070 +00:00] [WARN] [segments/segment.go:611] ["Search failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [mvcc=462407934410752016] [collectionID=462141493836954625] [segmentID=462141493829685674] [segmentType=Growing] [withIndex=false]
[2025/11/24 01:04:43.070 +00:00] [WARN] [querynodev2/services.go:803] ["failed to search segments"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [msgID=0] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [scope=Streaming] [error="N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2366] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:43.070 +00:00] [WARN] [delegator/delegator.go:849] ["failed to execute sub task"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [replicaID=462141493833891858] [taskType=Search] [nodeID=3] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2366] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:43.070 +00:00] [WARN] [delegator/delegator.go:875] ["some tasks failed to complete"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [replicaID=462141493833891858] [taskType=Search] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2366] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:43.070 +00:00] [WARN] [delegator/delegator.go:364] ["Delegator search failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [replicaID=462141493833891858] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2366] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:43.070 +00:00] [WARN] [querynodev2/handlers.go:423] ["failed to search on delegator"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [msgID=462141493828439172] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [scope=All] [nq=1] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2366] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:43.070 +00:00] [WARN] [proxy/task_search.go:940] ["QueryNode search result error"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collection=462141493836954625] [partitionIDs="[]"] [nodeID=3] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [reason="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2366] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:43.070 +00:00] [WARN] [proxy/lb_policy.go:266] ["search/query channel failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channelName=by-dev-rootcoord-dml_5_462141493836954625v0] [nodeID=3] [error="fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2366] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "] [errorVerbose="fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2366] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n | \t/workspace/source/internal/proxy/task_search.go:942\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:264\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:145\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:283\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:313\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:93\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) fail to search on QueryNode 3\nWraps: (3) worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2366] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n |\n | Stack trace: \nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2025/11/24 01:04:43.872 +00:00] [WARN] [proxy/lb_policy.go:171] ["failed to select shard"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channelName=by-dev-rootcoord-dml_5_462141493836954625v0] [excluded="[3]"] [candidates="<NodeID: 3, serviceable: true, address: 192.168.100.12:21123>"] [serviceableNodes=] [error="no available shard leaders: channel not available[channel=by-dev-rootcoord-dml_5_462141493836954625v0]"] [errorVerbose="no available shard leaders: channel not available[channel=by-dev-rootcoord-dml_5_462141493836954625v0]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/v2/util/merr.warpChannelErr\n | \t/workspace/source/pkg/util/merr/utils.go:712\n | github.com/milvus-io/milvus/pkg/v2/util/merr.WrapErrChannelNotAvailable\n | \t/workspace/source/pkg/util/merr/utils.go:734\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).selectNode.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:189\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).selectNode\n | \t/workspace/source/internal/proxy/lb_policy.go:215\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:238\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:145\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:283\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:313\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:93\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) no available shard leaders\nWraps: (3) channel not available[channel=by-dev-rootcoord-dml_5_462141493836954625v0]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2025/11/24 01:04:43.873 +00:00] [WARN] [proxy/lb_policy.go:156] ["all replicas are excluded after refresh shard leader cache, clear it and try to select node"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channelName=by-dev-rootcoord-dml_5_462141493836954625v0]
W20251124 01:04:43.874661 8658 emb_list_utils.h:112] [KNOWHERE][find_min_in_range][knowhere_search] invalid range, start_idx: 0, end_idx: 0
W20251124 01:04:43.874729 8658 brute_force.cc:290] [KNOWHERE][operator()][knowhere_search] get_sum_max_sim failed, num_query_vectors: 1, num_base_vectors: 0
=> Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266

[2025/11/24 01:04:43.875 +00:00] [WARN] [segments/segment.go:611] ["Search failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [mvcc=462407934620467216] [collectionID=462141493836954625] [segmentID=462141493829685674] [segmentType=Growing] [withIndex=false]
[2025/11/24 01:04:43.875 +00:00] [WARN] [querynodev2/services.go:803] ["failed to search segments"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [msgID=0] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [scope=Streaming] [error="N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2370] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:43.875 +00:00] [WARN] [delegator/delegator.go:849] ["failed to execute sub task"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [replicaID=462141493833891858] [taskType=Search] [nodeID=3] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2370] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:43.875 +00:00] [WARN] [delegator/delegator.go:875] ["some tasks failed to complete"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [replicaID=462141493833891858] [taskType=Search] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2370] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:43.875 +00:00] [WARN] [delegator/delegator.go:364] ["Delegator search failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [replicaID=462141493833891858] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2370] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:43.875 +00:00] [WARN] [querynodev2/handlers.go:423] ["failed to search on delegator"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [msgID=462141493828439172] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [scope=All] [nq=1] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2370] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:43.875 +00:00] [WARN] [proxy/task_search.go:940] ["QueryNode search result error"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collection=462141493836954625] [partitionIDs="[]"] [nodeID=3] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [reason="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2370] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:43.875 +00:00] [WARN] [proxy/lb_policy.go:266] ["search/query channel failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channelName=by-dev-rootcoord-dml_5_462141493836954625v0] [nodeID=3] [error="fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2370] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "] [errorVerbose="fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2370] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n | \t/workspace/source/internal/proxy/task_search.go:942\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:264\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:145\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:283\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:313\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:93\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) fail to search on QueryNode 3\nWraps: (3) worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2370] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n |\n | Stack trace: \nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2025/11/24 01:04:44.169 +00:00] [INFO] [balance/score_based_balancer.go:644] ["node channel workload status"] [collectionID=462141493833572742] [replicaID=462141493833891851] [nodes="["{NodeID: 3, AssignedScore: 15.000000, CurrentScore: 15.000000, Priority: 0}"]"]
[2025/11/24 01:04:44.470 +00:00] [INFO] [balance/score_based_balancer.go:644] ["node channel workload status"] [collectionID=462141493836954625] [replicaID=462141493833891858] [nodes="["{NodeID: 3, AssignedScore: 15.000000, CurrentScore: 15.000000, Priority: 0}"]"]
[2025/11/24 01:04:45.476 +00:00] [WARN] [proxy/lb_policy.go:171] ["failed to select shard"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channelName=by-dev-rootcoord-dml_5_462141493836954625v0] [excluded="[3]"] [candidates="<NodeID: 3, serviceable: true, address: 192.168.100.12:21123>"] [serviceableNodes=] [error="no available shard leaders: channel not available[channel=by-dev-rootcoord-dml_5_462141493836954625v0]"] [errorVerbose="no available shard leaders: channel not available[channel=by-dev-rootcoord-dml_5_462141493836954625v0]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/v2/util/merr.warpChannelErr\n | \t/workspace/source/pkg/util/merr/utils.go:712\n | github.com/milvus-io/milvus/pkg/v2/util/merr.WrapErrChannelNotAvailable\n | \t/workspace/source/pkg/util/merr/utils.go:734\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).selectNode.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:189\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).selectNode\n | \t/workspace/source/internal/proxy/lb_policy.go:215\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:238\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:145\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:283\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:313\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:93\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) no available shard leaders\nWraps: (3) channel not available[channel=by-dev-rootcoord-dml_5_462141493836954625v0]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2025/11/24 01:04:45.477 +00:00] [WARN] [proxy/lb_policy.go:156] ["all replicas are excluded after refresh shard leader cache, clear it and try to select node"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channelName=by-dev-rootcoord-dml_5_462141493836954625v0]
W20251124 01:04:45.479177 8658 emb_list_utils.h:112] [KNOWHERE][find_min_in_range][knowhere_search] invalid range, start_idx: 0, end_idx: 0
W20251124 01:04:45.479281 8658 brute_force.cc:290] [KNOWHERE][operator()][knowhere_search] get_sum_max_sim failed, num_query_vectors: 1, num_base_vectors: 0
=> Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266

[2025/11/24 01:04:45.480 +00:00] [WARN] [segments/segment.go:611] ["Search failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [mvcc=462407935039897615] [collectionID=462141493836954625] [segmentID=462141493829685674] [segmentType=Growing] [withIndex=false]
[2025/11/24 01:04:45.480 +00:00] [WARN] [querynodev2/services.go:803] ["failed to search segments"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [msgID=0] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [scope=Streaming] [error="N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:45.480 +00:00] [WARN] [delegator/delegator.go:849] ["failed to execute sub task"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [replicaID=462141493833891858] [taskType=Search] [nodeID=3] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:45.480 +00:00] [WARN] [delegator/delegator.go:875] ["some tasks failed to complete"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [replicaID=462141493833891858] [taskType=Search] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:45.480 +00:00] [WARN] [delegator/delegator.go:364] ["Delegator search failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [replicaID=462141493833891858] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:45.480 +00:00] [WARN] [querynodev2/handlers.go:423] ["failed to search on delegator"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [msgID=462141493828439172] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [scope=All] [nq=1] [error="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:45.480 +00:00] [WARN] [proxy/task_search.go:940] ["QueryNode search result error"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collection=462141493836954625] [partitionIDs="[]"] [nodeID=3] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [reason="worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "]
[2025/11/24 01:04:45.480 +00:00] [WARN] [proxy/lb_policy.go:266] ["search/query channel failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channelName=by-dev-rootcoord-dml_5_462141493836954625v0] [nodeID=3] [error="fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n\nStack trace: "] [errorVerbose="fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n | \t/workspace/source/internal/proxy/task_search.go:942\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:264\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:145\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:283\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:313\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:93\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) fail to search on QueryNode 3\nWraps: (3) worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n |\n | Stack trace: \nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2025/11/24 01:04:45.480 +00:00] [WARN] [retry/retry.go:147] ["retry func failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [retried=4] [caller=/workspace/source/internal/proxy/lb_policy.go:283] [error="failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0: fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\nStack trace: "] [errorVerbose="failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0: fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:270\n | [...repeated from below...]\nWraps: (2) failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n | \t/workspace/source/internal/proxy/task_search.go:942\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:264\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:145\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:283\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:313\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:93\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (4) fail to search on QueryNode 3\nWraps: (5) worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n |\n | Stack trace: \nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) merr.milvusError"]
[2025/11/24 01:04:48.368 +00:00] [INFO] [balance/score_based_balancer.go:644] ["node channel workload status"] [collectionID=462141493836581225] [replicaID=462141493833891853] [nodes="["{NodeID: 3, AssignedScore: 15.000000, CurrentScore: 15.000000, Priority: 0}"]"]
[2025/11/24 01:04:48.481 +00:00] [WARN] [retry/retry.go:205] ["retry func failed, reach max retry"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [attempt=5] [caller=/workspace/source/internal/proxy/lb_policy.go:283]
[2025/11/24 01:04:48.481 +00:00] [WARN] [proxy/lb_policy.go:285] ["failed to execute"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [collectionID=462141493836954625] [channelName=by-dev-rootcoord-dml_5_462141493836954625v0] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [error="failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0: fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\nStack trace: "] [errorVerbose="failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0: fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:270\n | [...repeated from below...]\nWraps: (2) failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n | \t/workspace/source/internal/proxy/task_search.go:942\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:264\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:145\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:283\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:313\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:93\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (4) fail to search on QueryNode 3\nWraps: (5) worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n |\n | Stack trace: \nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) merr.milvusError"]
[2025/11/24 01:04:48.481 +00:00] [WARN] [proxy/task_search.go:729] ["search execute failed"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [nq=1] [error="failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0: fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\nStack trace: "] [errorVerbose="failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0: fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:270\n | [...repeated from below...]\nWraps: (2) failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n | \t/workspace/source/internal/proxy/task_search.go:942\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:264\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:145\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:283\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:313\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:93\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (4) fail to search on QueryNode 3\nWraps: (5) worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n |\n | Stack trace: \nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) merr.milvusError"]
[2025/11/24 01:04:48.481 +00:00] [WARN] [proxy/task_scheduler.go:514] ["Failed to execute task: "] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [error="failed to search: failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0: fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\nStack trace: "] [errorVerbose="failed to search: failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0: fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Execute\n | \t/workspace/source/internal/proxy/task_search.go:730\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n | \t/workspace/source/internal/proxy/task_scheduler.go:511\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n | \t/workspace/source/internal/proxy/task_scheduler.go:616\n | github.com/milvus-io/milvus/pkg/v2/util/conc.(*Pool[...]).Submit.func1\n | \t/workspace/source/pkg/util/conc/pool.go:82\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:73\nWraps: (2) failed to search\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:270\n | [...repeated from below...]\nWraps: (4) failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0\nWraps: (5) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n | \t/workspace/source/internal/proxy/task_search.go:942\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:264\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:145\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:283\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:313\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:93\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (6) fail to search on QueryNode 3\nWraps: (7) worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n |\n | Stack trace: \nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) merr.milvusError"]
[2025/11/24 01:04:48.481 +00:00] [WARN] [proxy/impl.go:3082] ["HybridSearch failed to WaitToFinish"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [role=proxy] [db=default] [collection=test5] [partitions="[]"] [OutputFields="[item_id]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [dsls="[No.0 req, expr: ][No.1 req, expr: ]"] [error="failed to search: failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0: fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\nStack trace: "] [errorVerbose="failed to search: failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0: fail to search on QueryNode 3: worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Execute\n | \t/workspace/source/internal/proxy/task_search.go:730\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n | \t/workspace/source/internal/proxy/task_scheduler.go:511\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n | \t/workspace/source/internal/proxy/task_scheduler.go:616\n | github.com/milvus-io/milvus/pkg/v2/util/conc.(*Pool[...]).Submit.func1\n | \t/workspace/source/pkg/util/conc/pool.go:82\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:73\nWraps: (2) failed to search\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:270\n | [...repeated from below...]\nWraps: (4) failed to search/query delegator 3 for channel by-dev-rootcoord-dml_5_462141493836954625v0\nWraps: (5) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n | \t/workspace/source/internal/proxy/task_search.go:942\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:264\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:145\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:283\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:313\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:93\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (6) fail to search on QueryNode 3\nWraps: (7) worker(3) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 2374] : => Brute force search fail: brute_force inner error at /workspace/source/internal/core/src/query/SearchBruteForce.cpp:266\n |\n | Stack trace: \nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) merr.milvusError"]
[2025/11/24 01:04:48.481 +00:00] [INFO] [proxy/impl.go:3043] ["HybridSearch slow"] [clientRequestUnixmsec=1763946282453] [traceID=a95d52cd40a296a4d98fb3ac4a0ad929] [role=proxy] [db=default] [collection=test5] [partitions="[]"] [OutputFields="[item_id]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [dsls="[No.0 req, expr: ][No.1 req, expr: ]"] [guarantee_timestamp=462407932957425664] [duration=6.025194724s]
[2025/11/24 01:04:51.320 +00:00] [INFO] [importv2/util.go:540] ["import task stats"] [type=PreImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
[2025/11/24 01:04:51.320 +00:00] [INFO] [importv2/util.go:540] ["import task stats"] [type=ImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
[2025/11/24 01:04:52.830 +00:00] [INFO] [broadcaster/tombstone_scheduler.go:108] [triggerGCTombstone] [module=streamingcoord] [component=broadcaster] ["tombstone count"=0] ["expired offset"=0] ["expired time"=2025/11/24 00:34:52.830 +00:00]
[2025/11/24 01:04:52.861 +00:00] [INFO] [datacoord/compaction_trigger.go:336] ["handleSignal receive"] [compactionID=462141493836957083] [signal.collectionID=0] [signal.partitionID=0] [signal.segmentIDs="[]"]
[2025/11/24 01:04:52.861 +00:00] [INFO] [datacoord/index_meta.go:250] ["update index metric"] [collectionNum=6]
[2025/11/24 01:04:52.861 +00:00] [INFO] [datacoord/import_checker.go:156] ["import job stats"] [stateNum="{"Pending":0,"PreImporting":0,"Importing":0,"Failed":0,"Completed":0,"IndexBuilding":0,"Sorting":0}"]
[2025/11/24 01:04:52.861 +00:00] [INFO] [datacoord/import_checker.go:168] ["import task stats"] [type=PreImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
[2025/11/24 01:04:52.861 +00:00] [INFO] [datacoord/import_checker.go:168] ["import task stats"] [type=ImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
[2025/11/24 01:04:52.868 +00:00] [INFO] [observers/target_observer.go:526] ["Update readable segment version"] [collectionID=462141493833216217] [channelName=by-dev-rootcoord-dml_8_462141493833216217v0] [nodeID=3] [oldVersion=1763946272872158049] [newVersion=1763946282872333123]
[2025/11/24 01:04:52.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=63f0f00524054153a21569e09bb17dc8] [collectionID=462141493833627568] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946282872297975] [checkPoint=2025/11/24 01:04:08.600 +00:00] [deleteCP=2025/11/24 01:04:08.600 +00:00] [partitions="[462141493833627569]"]
[2025/11/24 01:04:52.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=d855086d938980b611b09c52e095738c] [collectionID=462141493836587714] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946282872201184] [checkPoint=2025/11/24 01:04:20.000 +00:00] [deleteCP=2025/11/24 01:04:20.000 +00:00] [partitions="[462141493836587715]"]
[2025/11/24 01:04:52.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=ae55af9ad9c351a1814360995aeb7e33] [collectionID=462141493833216217] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946282872333123] [checkPoint=2025/11/24 01:03:45.200 +00:00] [deleteCP=2025/11/24 01:03:45.200 +00:00] [partitions="[462141493833216218]"]
[2025/11/24 01:04:52.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=4] [totalSealedRowCount=4] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:04:52.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=15] [totalSealedRowCount=15] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:04:52.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [partitions="[462141493833627569]"] [oldVersion=1763946272872279727] [newVersion=1763946282872297975] [serviceable=true] [loadedRatio=1] [growingSegmentNum=1] [sealedSegmentNum=1]
[2025/11/24 01:04:52.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [partitions="[462141493833216218]"] [oldVersion=1763946272872158049] [newVersion=1763946282872333123] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=1]
[2025/11/24 01:04:52.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=df0c45ef05755d15eb103cedef2ff952] [collectionID=462141493833572742] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946282872145376] [checkPoint=2025/11/24 01:04:22.600 +00:00] [deleteCP=2025/11/24 01:04:22.600 +00:00] [partitions="[462141493833572743]"]
[2025/11/24 01:04:52.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:04:52.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [partitions="[462141493836587715]"] [oldVersion=1763946272872050184] [newVersion=1763946282872201184] [serviceable=true] [loadedRatio=1] [growingSegmentNum=1] [sealedSegmentNum=0]
[2025/11/24 01:04:52.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=2] [totalSealedRowCount=2] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:04:52.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=9e5d0524d294f799c1ec113fc756dea9] [collectionID=462141493836581225] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946282871922276] [checkPoint=2025/11/24 01:04:33.600 +00:00] [deleteCP=2025/11/24 01:04:33.600 +00:00] [partitions="[462141493836581226]"]
[2025/11/24 01:04:52.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=661e1a9592e1c008e7b0be35329605a0] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946282872033480] [checkPoint=2025/11/24 00:58:14.400 +00:00] [deleteCP=2025/11/24 00:58:14.400 +00:00] [partitions="[462141493836954626]"]
[2025/11/24 01:04:52.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [partitions="[462141493833572743]"] [oldVersion=1763946272872345008] [newVersion=1763946282872145376] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=1]
[2025/11/24 01:04:52.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:04:52.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [partitions="[462141493836581226]"] [oldVersion=1763946272872228397] [newVersion=1763946282871922276] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=0]
[2025/11/24 01:04:52.869 +00:00] [INFO] [observers/target_observer.go:562] ["observer trigger update current target"] [collectionID=462141493836587714]
[2025/11/24 01:04:52.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:04:52.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [partitions="[462141493836954626]"] [oldVersion=1763946272872180435] [newVersion=1763946282872033480] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=0]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836587714] [partitionIDs="[]"]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833627568] [partitionIDs="[]"]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833216217] [partitionIDs="[]"]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833572742] [partitionIDs="[]"]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836587714] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=1] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836581225] [partitionIDs="[]"]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [posTs=462407928381440001] [posTime=2025/11/24 01:04:20.000 +00:00]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833216217] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [numOfSegments=1] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836587714] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] ["# of unflushed segments"=1] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:20.000 +00:00] ["# of delete check point"=2025/11/24 01:04:20.000 +00:00]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833572742] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [numOfSegments=1] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [posTs=462407919258828816] [posTime=2025/11/24 01:03:45.200 +00:00]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836954625] [partitionIDs="[]"]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836581225] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [numOfSegments=0] ["result flushed"=0] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833216217] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:03:45.200 +00:00] ["# of delete check point"=2025/11/24 01:03:45.200 +00:00]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [posTs=462407929063014407] [posTime=2025/11/24 01:04:22.600 +00:00]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [posTs=462407931946598412] [posTime=2025/11/24 01:04:33.600 +00:00]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833627568] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [numOfSegments=2] ["result flushed"=1] ["result growing"=1] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833572742] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:22.600 +00:00] ["# of delete check point"=2025/11/24 01:04:22.600 +00:00]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [posTs=462407925392998411] [posTime=2025/11/24 01:04:08.600 +00:00]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836581225] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] ["# of unflushed segments"=0] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:33.600 +00:00] ["# of delete check point"=2025/11/24 01:04:33.600 +00:00]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833627568] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] ["# of unflushed segments"=1] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:08.600 +00:00] ["# of delete check point"=2025/11/24 01:04:08.600 +00:00]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [posTs=462407832541593619] [posTime=2025/11/24 00:58:14.400 +00:00]
[2025/11/24 01:04:52.869 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836954625] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] ["# of unflushed segments"=0] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 00:58:14.400 +00:00] ["# of delete check point"=2025/11/24 00:58:14.400 +00:00]
[2025/11/24 01:04:53.169 +00:00] [INFO] [balance/score_based_balancer.go:644] ["node channel workload status"] [collectionID=462141493833216217] [replicaID=462141493833891850] [nodes="["{NodeID: 3, AssignedScore: 15.000000, CurrentScore: 15.000000, Priority: 0}"]"]
[2025/11/24 01:04:54.081 +00:00] [INFO] [datacoord/meta.go:1978] ["UpdateChannelCheckpoint done"] [traceID=60685ad3a9e92175d2077015d88a2371] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [ts=462407934987468815] [time=2025/11/24 01:04:45.200 +00:00]
[2025/11/24 01:04:54.081 +00:00] [INFO] [datacoord/segment_manager.go:599] ["try remove empty sealed segment after channel cp updated failed to get channel"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0]
[2025/11/24 01:04:54.082 +00:00] [INFO] [recovery/recovery_storage_impl.go:124] ["update flush checkpoint"] [module=streamingnode] [nodeID=3] [component=recovery-storage] [channel=by-dev-rootcoord-dml_8:rw@3] [state=working] [vchannel=by-dev-rootcoord-dml_8_462141493833216217v0] [messageID=461804254268106398] [timeTick=462407934987468815]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:352] ["Rocksmq stats"] [module=rocksmq] [cache=1870848] ["rockskv memtable "=1050624] ["rockskv table readers"=192] ["rockskv pinned"=96] ["store memtable "=1050624] ["store table readers"=655488] ["store pinned"=96] ["store l0 file num"=0] ["store l1 file num"=0] ["store l2 file num"=1] ["store l3 file num"=0] ["store l4 file num"=0]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:472] ["Rocksmq Info"] [module=rocksmq] [topic=by-dev-rootcoord-dml_7] ["consumer num"=0] ["min position group names"=] ["min positions"=-1] ["page sum"=0] ["last page size"=42318912]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:472] ["Rocksmq Info"] [module=rocksmq] [topic=by-dev-rootcoord-dml_12] ["consumer num"=0] ["min position group names"=] ["min positions"=-1] ["page sum"=0] ["last page size"=19169]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:472] ["Rocksmq Info"] [module=rocksmq] [topic=by-dev-rootcoord-dml_11] ["consumer num"=0] ["min position group names"=] ["min positions"=-1] ["page sum"=0] ["last page size"=1418]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:472] ["Rocksmq Info"] [module=rocksmq] [topic=by-dev-rootcoord-dml_4] ["consumer num"=0] ["min position group names"=] ["min positions"=-1] ["page sum"=0] ["last page size"=42289659]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:472] ["Rocksmq Info"] [module=rocksmq] [topic=by-dev-rootcoord-dml_1] ["consumer num"=0] ["min position group names"=] ["min positions"=-1] ["page sum"=0] ["last page size"=44182494]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:472] ["Rocksmq Info"] [module=rocksmq] [topic=by-dev-rootcoord-dml_14] ["consumer num"=0] ["min position group names"=] ["min positions"=-1] ["page sum"=0] ["last page size"=48746]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:472] ["Rocksmq Info"] [module=rocksmq] [topic=by-dev-rootcoord-dml_0] ["consumer num"=0] ["min position group names"=] ["min positions"=-1] ["page sum"=0] ["last page size"=44346283]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:472] ["Rocksmq Info"] [module=rocksmq] [topic=by-dev-rootcoord-dml_5] ["consumer num"=0] ["min position group names"=] ["min positions"=-1] ["page sum"=0] ["last page size"=42273584]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:472] ["Rocksmq Info"] [module=rocksmq] [topic=by-dev-rootcoord-dml_6] ["consumer num"=0] ["min position group names"=] ["min positions"=-1] ["page sum"=0] ["last page size"=42258623]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:472] ["Rocksmq Info"] [module=rocksmq] [topic=by-dev-rootcoord-dml_15] ["consumer num"=0] ["min position group names"=] ["min positions"=-1] ["page sum"=0] ["last page size"=1558]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:472] ["Rocksmq Info"] [module=rocksmq] [topic=by-dev-rootcoord-dml_2] ["consumer num"=0] ["min position group names"=] ["min positions"=-1] ["page sum"=0] ["last page size"=42356509]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:472] ["Rocksmq Info"] [module=rocksmq] [topic=by-dev-rootcoord-dml_13] ["consumer num"=0] ["min position group names"=] ["min positions"=-1] ["page sum"=0] ["last page size"=3512]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:472] ["Rocksmq Info"] [module=rocksmq] [topic=by-dev-rootcoord-dml_8] ["consumer num"=0] ["min position group names"=] ["min positions"=-1] ["page sum"=0] ["last page size"=42751404]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:472] ["Rocksmq Info"] [module=rocksmq] [topic=by-dev-rootcoord-dml_9] ["consumer num"=0] ["min position group names"=] ["min positions"=-1] ["page sum"=0] ["last page size"=1454]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:472] ["Rocksmq Info"] [module=rocksmq] [topic=by-dev-rootcoord-dml_3] ["consumer num"=0] ["min position group names"=] ["min positions"=-1] ["page sum"=0] ["last page size"=42314750]
[2025/11/24 01:04:56.083 +00:00] [INFO] [server/rocksmq_impl.go:472] ["Rocksmq Info"] [module=rocksmq] [topic=by-dev-rootcoord-dml_10] ["consumer num"=0] ["min position group names"=] ["min positions"=-1] ["page sum"=0] ["last page size"=8600]
[2025/11/24 01:05:02.869 +00:00] [INFO] [observers/target_observer.go:526] ["Update readable segment version"] [collectionID=462141493833216217] [channelName=by-dev-rootcoord-dml_8_462141493833216217v0] [nodeID=3] [oldVersion=1763946282872333123] [newVersion=1763946292869555906]
[2025/11/24 01:05:02.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=c944e2345000fe65ea8ef0ef7eb4ab51] [collectionID=462141493836581225] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946292869582782] [checkPoint=2025/11/24 01:04:33.600 +00:00] [deleteCP=2025/11/24 01:04:33.600 +00:00] [partitions="[462141493836581226]"]
[2025/11/24 01:05:02.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=f6b58f7738f94c7eb3854615e12432db] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946292869617012] [checkPoint=2025/11/24 00:58:14.400 +00:00] [deleteCP=2025/11/24 00:58:14.400 +00:00] [partitions="[462141493836954626]"]
[2025/11/24 01:05:02.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=c4832413d1a53f93406163aa1d796ad4] [collectionID=462141493836587714] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946292869524334] [checkPoint=2025/11/24 01:04:20.000 +00:00] [deleteCP=2025/11/24 01:04:20.000 +00:00] [partitions="[462141493836587715]"]
[2025/11/24 01:05:02.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=4ad4a52c65b9fff04d44d4965a543a25] [collectionID=462141493833216217] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946292869555906] [checkPoint=2025/11/24 01:03:45.200 +00:00] [deleteCP=2025/11/24 01:03:45.200 +00:00] [partitions="[462141493833216218]"]
[2025/11/24 01:05:02.870 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:05:02.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=e21350e98031124683a77ce6242bb419] [collectionID=462141493833627568] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946292869591229] [checkPoint=2025/11/24 01:04:08.600 +00:00] [deleteCP=2025/11/24 01:04:08.600 +00:00] [partitions="[462141493833627569]"]
[2025/11/24 01:05:02.870 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=912d8f71361a1a18c052d198af83b4b6] [collectionID=462141493833572742] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946292869572376] [checkPoint=2025/11/24 01:04:22.600 +00:00] [deleteCP=2025/11/24 01:04:22.600 +00:00] [partitions="[462141493833572743]"]
[2025/11/24 01:05:02.870 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:05:02.870 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:05:02.870 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=15] [totalSealedRowCount=15] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:05:02.870 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [partitions="[462141493836581226]"] [oldVersion=1763946282871922276] [newVersion=1763946292869582782] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=0]
[2025/11/24 01:05:02.870 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [partitions="[462141493833216218]"] [oldVersion=1763946282872333123] [newVersion=1763946292869555906] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=1]
[2025/11/24 01:05:02.870 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [partitions="[462141493836954626]"] [oldVersion=1763946282872033480] [newVersion=1763946292869617012] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=0]
[2025/11/24 01:05:02.870 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=2] [totalSealedRowCount=2] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:05:02.870 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [partitions="[462141493836587715]"] [oldVersion=1763946282872201184] [newVersion=1763946292869524334] [serviceable=true] [loadedRatio=1] [growingSegmentNum=1] [sealedSegmentNum=0]
[2025/11/24 01:05:02.870 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [partitions="[462141493833572743]"] [oldVersion=1763946282872145376] [newVersion=1763946292869572376] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=1]
[2025/11/24 01:05:02.870 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=4] [totalSealedRowCount=4] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:05:02.870 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [partitions="[462141493833627569]"] [oldVersion=1763946282872297975] [newVersion=1763946292869591229] [serviceable=true] [loadedRatio=1] [growingSegmentNum=1] [sealedSegmentNum=1]
[2025/11/24 01:05:02.870 +00:00] [INFO] [observers/target_observer.go:562] ["observer trigger update current target"] [collectionID=462141493836581225]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833216217] [partitionIDs="[]"]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836581225] [partitionIDs="[]"]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833572742] [partitionIDs="[]"]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836587714] [partitionIDs="[]"]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836954625] [partitionIDs="[]"]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833627568] [partitionIDs="[]"]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836581225] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [numOfSegments=0] ["result flushed"=0] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [posTs=462407931946598412] [posTime=2025/11/24 01:04:33.600 +00:00]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833216217] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [numOfSegments=1] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836581225] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] ["# of unflushed segments"=0] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:33.600 +00:00] ["# of delete check point"=2025/11/24 01:04:33.600 +00:00]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833627568] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [numOfSegments=2] ["result flushed"=1] ["result growing"=1] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [posTs=462407934987468815] [posTime=2025/11/24 01:04:45.200 +00:00]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836587714] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=1] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833216217] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:45.200 +00:00] ["# of delete check point"=2025/11/24 01:04:45.200 +00:00]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [posTs=462407832541593619] [posTime=2025/11/24 00:58:14.400 +00:00]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [posTs=462407928381440001] [posTime=2025/11/24 01:04:20.000 +00:00]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836587714] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] ["# of unflushed segments"=1] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:20.000 +00:00] ["# of delete check point"=2025/11/24 01:04:20.000 +00:00]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [posTs=462407925392998411] [posTime=2025/11/24 01:04:08.600 +00:00]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833572742] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [numOfSegments=1] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833627568] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] ["# of unflushed segments"=1] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:08.600 +00:00] ["# of delete check point"=2025/11/24 01:04:08.600 +00:00]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [posTs=462407929063014407] [posTime=2025/11/24 01:04:22.600 +00:00]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836954625] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] ["# of unflushed segments"=0] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 00:58:14.400 +00:00] ["# of delete check point"=2025/11/24 00:58:14.400 +00:00]
[2025/11/24 01:05:02.870 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833572742] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:22.600 +00:00] ["# of delete check point"=2025/11/24 01:04:22.600 +00:00]
[2025/11/24 01:05:05.169 +00:00] [INFO] [balance/score_based_balancer.go:644] ["node channel workload status"] [collectionID=462141493836587714] [replicaID=462141493833891857] [nodes="["{NodeID: 3, AssignedScore: 15.000000, CurrentScore: 15.000000, Priority: 0}"]"]
[2025/11/24 01:05:11.077 +00:00] [INFO] [balancer/balancer_impl.go:393] ["start to balance"] [module=streamingcoord] [component=balancer] [policy=vchannelFair]
[2025/11/24 01:05:11.077 +00:00] [INFO] [balancer/balancer_impl.go:396] ["collect all status..."] [module=streamingcoord] [component=balancer] [policy=vchannelFair]
[2025/11/24 01:05:11.082 +00:00] [INFO] [balancer/balancer_impl.go:413] ["balance policy generate result success, try to assign..."] [module=streamingcoord] [component=balancer] [policy=vchannelFair] [expectedLayout=by-dev-rootcoord-dml_4:rw@4>[email protected]:22222,by-dev-rootcoord-dml_2:rw@4>[email protected]:22222,by-dev-rootcoord-dml_14:rw@4>[email protected]:22222,by-dev-rootcoord-dml_8:rw@4>[email protected]:22222,by-dev-rootcoord-dml_7:rw@4>[email protected]:22222,by-dev-rootcoord-dml_15:rw@4>[email protected]:22222,by-dev-rootcoord-dml_11:rw@4>[email protected]:22222,by-dev-rootcoord-dml_3:rw@4>[email protected]:22222,by-dev-rootcoord-dml_13:rw@4>[email protected]:22222,by-dev-rootcoord-dml_1:rw@4>[email protected]:22222,by-dev-rootcoord-dml_0:rw@4>[email protected]:22222,by-dev-rootcoord-dml_5:rw@4>[email protected]:22222,by-dev-rootcoord-dml_10:rw@4>[email protected]:22222,by-dev-rootcoord-dml_6:rw@4>[email protected]:22222,by-dev-rootcoord-dml_12:rw@4>[email protected]:22222,by-dev-rootcoord-dml_9:rw@4>[email protected]:22222]
[2025/11/24 01:05:11.082 +00:00] [INFO] [balancer/balancer_impl.go:421] ["no change of balance result need to be applied"] [module=streamingcoord] [component=balancer] [policy=vchannelFair]
[2025/11/24 01:05:11.082 +00:00] [INFO] [balancer/balancer_impl.go:271] ["apply balance success"] [module=streamingcoord] [component=balancer] [policy=vchannelFair]
[2025/11/24 01:05:11.082 +00:00] [INFO] [balancer/balancer_impl.go:234] ["balance wait"] [module=streamingcoord] [component=balancer] [policy=vchannelFair] [nextBalanceInterval=1m0s]
[2025/11/24 01:05:11.470 +00:00] [INFO] [balance/score_based_balancer.go:644] ["node channel workload status"] [collectionID=462141493833627568] [replicaID=462141493833891852] [nodes="["{NodeID: 3, AssignedScore: 15.000000, CurrentScore: 15.000000, Priority: 0}"]"]
[2025/11/24 01:05:12.869 +00:00] [INFO] [observers/target_observer.go:526] ["Update readable segment version"] [collectionID=462141493833216217] [channelName=by-dev-rootcoord-dml_8_462141493833216217v0] [nodeID=3] [oldVersion=1763946292869555906] [newVersion=1763946302870423720]
[2025/11/24 01:05:12.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=ebd0c1733dc24e23911a1eb977419c59] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946302870468451] [checkPoint=2025/11/24 00:58:14.400 +00:00] [deleteCP=2025/11/24 00:58:14.400 +00:00] [partitions="[462141493836954626]"]
[2025/11/24 01:05:12.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=3aee19b821d77448a987405b02a2f9ce] [collectionID=462141493836587714] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946302870439348] [checkPoint=2025/11/24 01:04:20.000 +00:00] [deleteCP=2025/11/24 01:04:20.000 +00:00] [partitions="[462141493836587715]"]
[2025/11/24 01:05:12.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=547c2d577a89b9d457a25bfe3747bfbf] [collectionID=462141493833216217] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946302870423720] [checkPoint=2025/11/24 01:04:45.200 +00:00] [deleteCP=2025/11/24 01:04:45.200 +00:00] [partitions="[462141493833216218]"]
[2025/11/24 01:05:12.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=07475a0823fc533b38921db61148707b] [collectionID=462141493833627568] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946302870459488] [checkPoint=2025/11/24 01:04:08.600 +00:00] [deleteCP=2025/11/24 01:04:08.600 +00:00] [partitions="[462141493833627569]"]
[2025/11/24 01:05:12.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=15] [totalSealedRowCount=15] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:05:12.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:05:12.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=7d5e330757b20085938465dc3b5cf7eb] [collectionID=462141493833572742] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946302870494236] [checkPoint=2025/11/24 01:04:22.600 +00:00] [deleteCP=2025/11/24 01:04:22.600 +00:00] [partitions="[462141493833572743]"]
[2025/11/24 01:05:12.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=74ab1eebb4828fa6a9000c72344803e5] [collectionID=462141493836581225] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946302870394365] [checkPoint=2025/11/24 01:04:33.600 +00:00] [deleteCP=2025/11/24 01:04:33.600 +00:00] [partitions="[462141493836581226]"]
[2025/11/24 01:05:12.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=4] [totalSealedRowCount=4] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:05:12.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [partitions="[462141493833627569]"] [oldVersion=1763946292869591229] [newVersion=1763946302870459488] [serviceable=true] [loadedRatio=1] [growingSegmentNum=1] [sealedSegmentNum=1]
[2025/11/24 01:05:12.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:05:12.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=2] [totalSealedRowCount=2] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:05:12.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [partitions="[462141493836581226]"] [oldVersion=1763946292869582782] [newVersion=1763946302870394365] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=0]
[2025/11/24 01:05:12.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [partitions="[462141493833572743]"] [oldVersion=1763946292869572376] [newVersion=1763946302870494236] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=1]
[2025/11/24 01:05:12.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [partitions="[462141493833216218]"] [oldVersion=1763946292869555906] [newVersion=1763946302870423720] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=1]
[2025/11/24 01:05:12.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:05:12.870 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [partitions="[462141493836954626]"] [oldVersion=1763946292869617012] [newVersion=1763946302870468451] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=0]
[2025/11/24 01:05:12.870 +00:00] [INFO] [observers/target_observer.go:562] ["observer trigger update current target"] [collectionID=462141493833627568]
[2025/11/24 01:05:12.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [partitions="[462141493836587715]"] [oldVersion=1763946292869524334] [newVersion=1763946302870439348] [serviceable=true] [loadedRatio=1] [growingSegmentNum=1] [sealedSegmentNum=0]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833627568] [partitionIDs="[]"]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833216217] [partitionIDs="[]"]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836581225] [partitionIDs="[]"]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833572742] [partitionIDs="[]"]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836581225] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [numOfSegments=0] ["result flushed"=0] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833627568] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [numOfSegments=2] ["result flushed"=1] ["result growing"=1] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836954625] [partitionIDs="[]"]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836587714] [partitionIDs="[]"]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [posTs=462407925392998411] [posTime=2025/11/24 01:04:08.600 +00:00]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [posTs=462407931946598412] [posTime=2025/11/24 01:04:33.600 +00:00]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833627568] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] ["# of unflushed segments"=1] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:08.600 +00:00] ["# of delete check point"=2025/11/24 01:04:08.600 +00:00]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836581225] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] ["# of unflushed segments"=0] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:33.600 +00:00] ["# of delete check point"=2025/11/24 01:04:33.600 +00:00]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833216217] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [numOfSegments=1] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [posTs=462407934987468815] [posTime=2025/11/24 01:04:45.200 +00:00]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833572742] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [numOfSegments=1] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833216217] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:45.200 +00:00] ["# of delete check point"=2025/11/24 01:04:45.200 +00:00]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [posTs=462407929063014407] [posTime=2025/11/24 01:04:22.600 +00:00]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [posTs=462407832541593619] [posTime=2025/11/24 00:58:14.400 +00:00]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833572742] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:22.600 +00:00] ["# of delete check point"=2025/11/24 01:04:22.600 +00:00]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836587714] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=1] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836954625] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] ["# of unflushed segments"=0] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 00:58:14.400 +00:00] ["# of delete check point"=2025/11/24 00:58:14.400 +00:00]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [posTs=462407928381440001] [posTime=2025/11/24 01:04:20.000 +00:00]
[2025/11/24 01:05:12.870 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836587714] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] ["# of unflushed segments"=1] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:20.000 +00:00] ["# of delete check point"=2025/11/24 01:04:20.000 +00:00]
[2025/11/24 01:05:14.072 +00:00] [INFO] [datacoord/meta.go:1978] ["UpdateChannelCheckpoint done"] [traceID=dff1516165dc60e32d7a30182cec492c] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [ts=462407941121638407] [time=2025/11/24 01:05:08.600 +00:00]
[2025/11/24 01:05:14.072 +00:00] [INFO] [datacoord/segment_manager.go:599] ["try remove empty sealed segment after channel cp updated failed to get channel"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0]
[2025/11/24 01:05:14.072 +00:00] [INFO] [recovery/recovery_storage_impl.go:124] ["update flush checkpoint"] [module=streamingnode] [nodeID=3] [component=recovery-storage] [channel=by-dev-rootcoord-dml_14:rw@3] [state=working] [vchannel=by-dev-rootcoord-dml_14_462141493833627568v0] [messageID=462141496768331799] [timeTick=462407941121638407]
[2025/11/24 01:05:14.080 +00:00] [INFO] [datacoord/segment_manager.go:599] ["try remove empty sealed segment after channel cp updated failed to get channel"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0]
[2025/11/24 01:05:14.080 +00:00] [INFO] [recovery/recovery_storage_impl.go:124] ["update flush checkpoint"] [module=streamingnode] [nodeID=3] [component=recovery-storage] [channel=by-dev-rootcoord-dml_5:rw@3] [state=working] [vchannel=by-dev-rootcoord-dml_5_462141493836954625v0] [messageID=461804230457305017] [timeTick=462407832541593619]
[2025/11/24 01:05:14.170 +00:00] [INFO] [balance/score_based_balancer.go:644] ["node channel workload status"] [collectionID=462141493833572742] [replicaID=462141493833891851] [nodes="["{NodeID: 3, AssignedScore: 15.000000, CurrentScore: 15.000000, Priority: 0}"]"]
[2025/11/24 01:05:14.469 +00:00] [INFO] [balance/score_based_balancer.go:644] ["node channel workload status"] [collectionID=462141493836954625] [replicaID=462141493833891858] [nodes="["{NodeID: 3, AssignedScore: 15.000000, CurrentScore: 15.000000, Priority: 0}"]"]
[2025/11/24 01:05:18.369 +00:00] [INFO] [balance/score_based_balancer.go:644] ["node channel workload status"] [collectionID=462141493836581225] [replicaID=462141493833891853] [nodes="["{NodeID: 3, AssignedScore: 15.000000, CurrentScore: 15.000000, Priority: 0}"]"]
[2025/11/24 01:05:22.868 +00:00] [INFO] [observers/target_observer.go:526] ["Update readable segment version"] [collectionID=462141493836954625] [channelName=by-dev-rootcoord-dml_5_462141493836954625v0] [nodeID=3] [oldVersion=1763946302870468451] [newVersion=1763946312870345257]
[2025/11/24 01:05:22.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=0b6593bff32b032091db43f4f332f256] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946312870345257] [checkPoint=2025/11/24 00:58:14.400 +00:00] [deleteCP=2025/11/24 00:58:14.400 +00:00] [partitions="[462141493836954626]"]
[2025/11/24 01:05:22.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=8a13c67e840f85a2cf0ef052bf66b3cc] [collectionID=462141493836587714] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946312870361441] [checkPoint=2025/11/24 01:04:20.000 +00:00] [deleteCP=2025/11/24 01:04:20.000 +00:00] [partitions="[462141493836587715]"]
[2025/11/24 01:05:22.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:05:22.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [partitions="[462141493836954626]"] [oldVersion=1763946302870468451] [newVersion=1763946312870345257] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=0]
[2025/11/24 01:05:22.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:05:22.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=0614adcc23db5c3501744c5a9ca46a6b] [collectionID=462141493833216217] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946312870323772] [checkPoint=2025/11/24 01:04:45.200 +00:00] [deleteCP=2025/11/24 01:04:45.200 +00:00] [partitions="[462141493833216218]"]
[2025/11/24 01:05:22.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [partitions="[462141493836587715]"] [oldVersion=1763946302870439348] [newVersion=1763946312870361441] [serviceable=true] [loadedRatio=1] [growingSegmentNum=1] [sealedSegmentNum=0]
[2025/11/24 01:05:22.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=15] [totalSealedRowCount=15] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:05:22.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [partitions="[462141493833216218]"] [oldVersion=1763946302870423720] [newVersion=1763946312870323772] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=1]
[2025/11/24 01:05:22.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=1da8579205d85e6be7bf79aac18df4cd] [collectionID=462141493833572742] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946312870333912] [checkPoint=2025/11/24 01:04:22.600 +00:00] [deleteCP=2025/11/24 01:04:22.600 +00:00] [partitions="[462141493833572743]"]
[2025/11/24 01:05:22.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=2] [totalSealedRowCount=2] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:05:22.869 +00:00] [INFO] [observers/target_observer.go:562] ["observer trigger update current target"] [collectionID=462141493836587714]
[2025/11/24 01:05:22.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [partitions="[462141493833572743]"] [oldVersion=1763946302870494236] [newVersion=1763946312870333912] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=1]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836587714] [partitionIDs="[]"]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836954625] [partitionIDs="[]"]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833216217] [partitionIDs="[]"]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833216217] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [numOfSegments=1] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:22.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=4c1881f9e5b3584e74f4de66659fb8ec] [collectionID=462141493833627568] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946312870294137] [checkPoint=2025/11/24 01:04:08.600 +00:00] [deleteCP=2025/11/24 01:04:08.600 +00:00] [partitions="[462141493833627569]"]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] [posTs=462407934987468815] [posTime=2025/11/24 01:04:45.200 +00:00]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833216217] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_8_462141493833216217v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:45.200 +00:00] ["# of delete check point"=2025/11/24 01:04:45.200 +00:00]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836587714] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=1] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836954625] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [posTs=462407928381440001] [posTime=2025/11/24 01:04:20.000 +00:00]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] [posTs=462407832541593619] [posTime=2025/11/24 00:58:14.400 +00:00]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836587714] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] ["# of unflushed segments"=1] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:20.000 +00:00] ["# of delete check point"=2025/11/24 01:04:20.000 +00:00]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836954625] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_5_462141493836954625v0] ["# of unflushed segments"=0] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 00:58:14.400 +00:00] ["# of delete check point"=2025/11/24 00:58:14.400 +00:00]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833572742] [partitionIDs="[]"]
[2025/11/24 01:05:22.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=4] [totalSealedRowCount=4] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=1] [action=SyncTargetVersion]
[2025/11/24 01:05:22.869 +00:00] [INFO] [querynodev2/services.go:1393] ["sync action"] [traceID=eb3e73d48c6a68b849914c9fe0307c82] [collectionID=462141493836581225] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [currentNodeID=3] [Action=UpdateVersion] [TargetVersion=1763946312870296348] [checkPoint=2025/11/24 01:04:33.600 +00:00] [deleteCP=2025/11/24 01:04:33.600 +00:00] [partitions="[462141493836581226]"]
[2025/11/24 01:05:22.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [partitions="[462141493833627569]"] [oldVersion=1763946302870459488] [newVersion=1763946312870294137] [serviceable=true] [loadedRatio=1] [growingSegmentNum=1] [sealedSegmentNum=1]
[2025/11/24 01:05:22.869 +00:00] [INFO] [delegator/distribution.go:316] ["channel distribution serviceable changed"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [serviceable=true] [loadedRatio=1] [loadedSealedRowCount=0] [totalSealedRowCount=0] [unloadedSealedSegmentNum=0] [totalSealedSegmentNum=0] [action=SyncTargetVersion]
[2025/11/24 01:05:22.869 +00:00] [INFO] [delegator/distribution.go:463] ["Update channel query view"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [partitions="[462141493836581226]"] [oldVersion=1763946302870394365] [newVersion=1763946312870296348] [serviceable=true] [loadedRatio=1] [growingSegmentNum=0] [sealedSegmentNum=0]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833572742] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [numOfSegments=1] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [posTs=462407929063014407] [posTime=2025/11/24 01:04:22.600 +00:00]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833572742] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:22.600 +00:00] ["# of delete check point"=2025/11/24 01:04:22.600 +00:00]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493836581225] [partitionIDs="[]"]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/services.go:993] ["get recovery info request received"] [collectionID=462141493833627568] [partitionIDs="[]"]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493836581225] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [numOfSegments=0] ["result flushed"=0] ["result growing"=0] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] [posTs=462407931946598412] [posTime=2025/11/24 01:04:33.600 +00:00]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/handler.go:209] [GetQueryVChanPositions] [collectionID=462141493833627568] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [numOfSegments=2] ["result flushed"=1] ["result growing"=1] ["result L0"=0] ["partition stats"={}]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493836581225] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_15_462141493836581225v0] ["# of unflushed segments"=0] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:04:33.600 +00:00] ["# of delete check point"=2025/11/24 01:04:33.600 +00:00]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/handler.go:467] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] [posTs=462407941121638407] [posTime=2025/11/24 01:05:08.600 +00:00]
[2025/11/24 01:05:22.869 +00:00] [INFO] [datacoord/services.go:1013] ["datacoord append channelInfo in GetRecoveryInfo"] [collectionID=462141493833627568] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_14_462141493833627568v0] ["# of unflushed segments"=1] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] ["# of check point"=2025/11/24 01:05:08.600 +00:00] ["# of delete check point"=2025/11/24 01:05:08.600 +00:00]
[2025/11/24 01:05:23.169 +00:00] [INFO] [balance/score_based_balancer.go:644] ["node channel workload status"] [collectionID=462141493833216217] [replicaID=462141493833891850] [nodes="["{NodeID: 3, AssignedScore: 15.000000, CurrentScore: 15.000000, Priority: 0}"]"]
[2025/11/24 01:05:24.081 +00:00] [INFO] [datacoord/meta.go:1978] ["UpdateChannelCheckpoint done"] [traceID=e5cd882d34ae469dc97790758c40ca7a] [channel=by-dev-rootcoord-dml_13_462141493833572742v0] [ts=462407944791654405] [time=2025/11/24 01:05:22.600 +00:00]
[2025/11/24 01:05:24.081 +00:00] [INFO] [datacoord/segment_manager.go:599] ["try remove empty sealed segment after channel cp updated failed to get channel"] [channel=by-dev-rootcoord-dml_13_462141493833572742v0]
[2025/11/24 01:05:24.081 +00:00] [INFO] [recovery/recovery_storage_impl.go:124] ["update flush checkpoint"] [module=streamingnode] [nodeID=3] [component=recovery-storage] [channel=by-dev-rootcoord-dml_13:rw@3] [state=working] [vchannel=by-dev-rootcoord-dml_13_462141493833572742v0] [messageID=462141496769904651] [timeTick=462407944791654405]
[2025/11/24 01:05:24.082 +00:00] [INFO] [datacoord/meta.go:1978] ["UpdateChannelCheckpoint done"] [traceID=6ed18309a3dc6a9af0c45901b578ce95] [channel=by-dev-rootcoord-dml_4_462141493836587714v0] [ts=462407944110080006] [time=2025/11/24 01:05:20.000 +00:00]
[2025/11/24 01:05:24.082 +00:00] [INFO] [datacoord/segment_manager.go:599] ["try remove empty sealed segment after channel cp updated failed to get channel"] [channel=by-dev-rootcoord-dml_4_462141493836587714v0]
[2025/11/24 01:05:24.082 +00:00] [INFO] [recovery/recovery_storage_impl.go:124] ["update flush checkpoint"] [module=streamingnode] [nodeID=3] [component=recovery-storage] [channel=by-dev-rootcoord-dml_4:rw@3] [state=working] [vchannel=by-dev-rootcoord-dml_4_462141493836587714v0] [messageID=461804172747615242] [timeTick=462407944110080006]

avayayu Nov 24, 2025
Author

I run milvus with docker,image is milvus:v2.6.4. The log is docker logs XX --since 1m after i run the code above.

avayayu Nov 24, 2025
Author

see attachfile.
error.log

xiaofan-luan · 2025-11-24T01:22:16Z

xiaofan-luan
Nov 24, 2025
Maintainer

seems to be you don't have any data in the field?

but agree this is a issue, we should return emply result instead of throw error.

5 replies

avayayu Nov 24, 2025
Author

yes. in my test datas,features array may have zero element.

xiaofan-luan Nov 24, 2025
Maintainer

it is not as expected.
I think we will add a check here.

avayayu Nov 24, 2025
Author

Is this reasonable? In practical scenarios, it is quite common for some fields to lack data

xiaofan-luan Nov 24, 2025
Maintainer

milvus don't support vector field to be null.

xiaofan-luan Nov 24, 2025
Maintainer

this might be change in the future

yhmo · 2025-11-24T03:39:31Z

yhmo
Nov 24, 2025
Collaborator

Try this example, hybrid search on two struct fields:

import numpy as np
from pymilvus import (
    MilvusClient,
    DataType,
    AnnSearchRequest, RRFRanker
)
from pymilvus.client.embedding_list import EmbeddingList

COLLECTION_NAME = "AAA"
EMBEDDING_DIM = 32

milvus_client = MilvusClient("http://localhost:19530")
has_collection = milvus_client.has_collection(COLLECTION_NAME, timeout=5)
if has_collection:
    milvus_client.drop_collection(COLLECTION_NAME)

schema = milvus_client.create_schema(enable_dynamic_field=True)
schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
schema.add_field("text", DataType.VARCHAR, max_length=100)

struct_schema = milvus_client.create_struct_field_schema()
struct_schema.add_field("struct_str1", DataType.VARCHAR, max_length=65535)
struct_schema.add_field("struct_float_vec1", DataType.FLOAT_VECTOR, dim=EMBEDDING_DIM)
schema.add_field("struct_field1",datatype=DataType.ARRAY, element_type=DataType.STRUCT, struct_schema=struct_schema, max_capacity=1000)

struct_schema = milvus_client.create_struct_field_schema()
struct_schema.add_field("struct_str2", DataType.VARCHAR, max_length=65535)
struct_schema.add_field("struct_float_vec2", DataType.FLOAT_VECTOR, dim=EMBEDDING_DIM)
schema.add_field("struct_field2",datatype=DataType.ARRAY, element_type=DataType.STRUCT, struct_schema=struct_schema, max_capacity=1000)


milvus_client.create_collection(COLLECTION_NAME, schema=schema)

index_params = milvus_client.prepare_index_params()
index_params.add_index(field_name="struct_field1[struct_float_vec1]", index_type="HNSW", index_name="struct_float_vec_index1", metric_type="MAX_SIM_COSINE", index_params={"M": 16, "efConstruction": 200})
index_params.add_index(field_name="struct_field2[struct_float_vec2]", index_type="HNSW", index_name="struct_float_vec_index2", metric_type="MAX_SIM_IP", index_params={"M": 16, "efConstruction": 200})
milvus_client.create_index(COLLECTION_NAME, schema=schema, index_params=index_params)

coll = milvus_client.describe_collection(COLLECTION_NAME)
print("Describe collection", coll)

rng = np.random.default_rng(seed=19530)

N = 100
rows = [{
     "text": f"text_{i}",
     "struct_field1": [
         {
          "struct_str1": f"struct1_{i}",
          "struct_float_vec1": rng.random((1, EMBEDDING_DIM))[0],
          }
          for _ in range(3)],
     "struct_field2": [
         {
             "struct_str2": f"struct2_{i}",
             "struct_float_vec2": rng.random((1, EMBEDDING_DIM))[0],
         }
         for _ in range(2)],
     }
    for i in range(N)]
result = milvus_client.insert(COLLECTION_NAME, rows)

milvus_client.load_collection(COLLECTION_NAME)


# hybrid search on two struct fields
req1 = AnnSearchRequest(
    anns_field="struct_field1[struct_float_vec1]",
    data=[EmbeddingList._from_random_test(5, EMBEDDING_DIM, seed=19530)],
    param={"metric_type": "MAX_SIM_COSINE"},
    limit=3,
)
req2 = AnnSearchRequest(
    anns_field="struct_field2[struct_float_vec2]",
    data=[EmbeddingList._from_random_test(4, EMBEDDING_DIM, seed=19531)],
    param={"metric_type": "MAX_SIM_IP"},
    limit=3,
)
results = milvus_client.hybrid_search(
    collection_name=COLLECTION_NAME,
    reqs=[req1, req2],
    ranker=RRFRanker(100),
    limit=3,
    consistency_level="Strong",
    output_fields=["text"],
)
print("Hybrid search on two struct fields:")
for res in results:
    for r in res:
        print(r)

It works fine:

Hybrid search on two struct fields:
{'id': 462410046341949098, 'distance': 0.009900989942252636, 'entity': {'text': 'text_0'}}
{'id': 462410046341949109, 'distance': 0.009900989942252636, 'entity': {'text': 'text_11'}}
{'id': 462410046341949151, 'distance': 0.009803921915590763, 'entity': {'text': 'text_53'}}

0 replies

alwayslove2013 · 2025-11-25T08:13:13Z

alwayslove2013
Nov 25, 2025

@avayayu After thorough inspection, we have identified an issue with Knowhere's brute force algorithm when handling empty embedding lists. This problem will be fixed in this PR and will be included in the next release.

0 replies

Hybrid search on multi struct array field,is it possiple #45779

Uh oh!

avayayu Nov 23, 2025

Replies: 4 comments · 10 replies

Uh oh!

Uh oh!

xiaofan-luan Nov 24, 2025 Maintainer

Uh oh!

avayayu Nov 24, 2025 Author

Uh oh!

xiaofan-luan Nov 24, 2025 Maintainer

Uh oh!

avayayu Nov 24, 2025 Author

Uh oh!

avayayu Nov 24, 2025 Author

Uh oh!

avayayu Nov 24, 2025 Author

Uh oh!

xiaofan-luan Nov 24, 2025 Maintainer

Uh oh!

avayayu Nov 24, 2025 Author

Uh oh!

xiaofan-luan Nov 24, 2025 Maintainer

Uh oh!

avayayu Nov 24, 2025 Author

Uh oh!

xiaofan-luan Nov 24, 2025 Maintainer

Uh oh!

xiaofan-luan Nov 24, 2025 Maintainer

Uh oh!

yhmo Nov 24, 2025 Collaborator

Uh oh!

alwayslove2013 Nov 25, 2025

avayayu
Nov 23, 2025

Replies: 4 comments 10 replies

xiaofan-luan
Nov 24, 2025
Maintainer

avayayu Nov 24, 2025
Author

xiaofan-luan Nov 24, 2025
Maintainer

avayayu Nov 24, 2025
Author

avayayu Nov 24, 2025
Author

avayayu Nov 24, 2025
Author

xiaofan-luan
Nov 24, 2025
Maintainer

avayayu Nov 24, 2025
Author

xiaofan-luan Nov 24, 2025
Maintainer

avayayu Nov 24, 2025
Author

xiaofan-luan Nov 24, 2025
Maintainer

xiaofan-luan Nov 24, 2025
Maintainer

yhmo
Nov 24, 2025
Collaborator

alwayslove2013
Nov 25, 2025