milvus backup fail

[rere950303]

• milvus version: v2.4.23 (cluster mode)
• milvus backup version: v0.5.2

---

Hello, I run a scheduled backup once a day, but the backup fails with the following error log.

[core/backup_impl_create_backup.go:814] ["Binlog file not exist"] [collection_id=459720225025021756] [partition_id=459720225025021771] [segment_id=459720225348323073] [group_id=459720225348323073] [] [file=files/delta_log/459720225025021756/459720225025021771/459720225348323073/459720225348325346] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).copySegments.func1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:814\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:71\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78"]

Would increasing the backup.gcPause.seconds field in the backup.yaml file possibly resolve this issue?

[rere950303]

@yhmo @huanghaoyuanhhy
thanks for your works! could check this issue plz? 🙏

[yhmo]

milvus-backup firstly fetches metadata(the metadata contains the storage paths of the binlog files and collection schema) of a collection from the etcd server, and then copies the binlog files into another path on minio/s3. There is a time gap between the first step and the second step. If the collection has data continually inserted, there will be segments created and deleted, the metadata is frequently changed, and GC process might delete some binlog files occasionally. milvus-backup might report this error if a binlog is deleted by GC while the previous metadata still contains the path of this binog.

[rere950303]

@yhmo
Yes, that’s right. So I thought increasing backup.gcPause.seconds could serve as a temporary workaround.
However, looking at this PR, it seems that the GC-related functionality has already been improved.
Unfortunately, due to the state_code bug in backup rest api response, I haven’t been able to upgrade to the newer version yet (issue #742).

Do you happen to know when the state_code bug in the backup REST API is expected to be fixed?

[yhmo]

I just read the source code of milvus-backup.
If gcPause.enable=true, milvus-backup will send a http request to milvus server with the url "http://xxxxxxx/management/datacoord/garbage_collection/pause"

In the source code of milvus v2.4.23, the "management/datacoord/garbage_collection/pause" is implemented, so I suppose the backup.gcPause.seconds can work with v2.4.23.

For the state_code issue, wait @huanghaoyuanhhy to comfirm.

[deepvita123]

Hi,

can you please share the process to take backup and restore of Milvus standalone version 2.4.X

Thank you !

Regards

[yhmo]

There is a special doc about how to use milvus-backup in different cases:
https://milvus.io/docs/single-instance-backup-and-restore.md

You need to build the tool first, download this repo to your local: https://github.com/zilliztech/milvus-backup
And switch to a version that is compatible with 2.4.x, I think v0.5.2 is ok

git checkout v0.5.2

Build the tool in your local(it requires Go environment)

go get
go build

You will see an executable tool named "milvus-backup" built under the repo path. Then follow the doc to configure the tool and do backup/restore.

Backup/restore in one milvus:
https://milvus.io/docs/single-instance-backup-and-restore.md

Backup/restore between two milvus that use the same minio/s3 bucket:
https://milvus.io/docs/shared-bucket-backup-and-restore.md

Backup/restore between two milvus that use different minio/s3 buckets:
https://milvus.io/docs/cross-bucket-backup-and-restore.md

Backup/restore between two milvus that use different minio/s3 service:
https://milvus.io/docs/multi-storage-backup-and-restore.md