Currently, the data size of the computing cluster is obtained via the S3 client; however, this process becomes extremely slow when the number of files reaches hundreds of thousands or even millions. Can this metadata be stored in Milvus to enable fast retrieval through the Milvus client?

[Sallery-X]

[yhmo]

You can deploy a monitoring service to observe the milvus: https://milvus.io/docs/visualize.md
There is a metrics named "Stored Binlog Size" in data coord to display the total size of stored binlog: https://milvus.io/docs/metrics_dashboard.md

[xiaofan-luan]

We can probably add size information into manifest @tedxu similar to the footer size. Caching manifest should be good enough.

thoughts? @tedxu @liliu-z @jiaqizho @shaoting-huang

[jiaqizho]

Already support kv meta(milvus-io/milvus-storage@302143c) in manifest.