Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lightning: retry when write tikv is slow and rate limiter context timeout #51383

Closed
D3Hunter opened this issue Feb 28, 2024 · 1 comment · Fixed by #51425
Closed

lightning: retry when write tikv is slow and rate limiter context timeout #51383

D3Hunter opened this issue Feb 28, 2024 · 1 comment · Fixed by #51425
Assignees
Labels
affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects 5.4.x versions. affects-6.1 affects-6.5 affects-7.1 affects-7.5 component/ddl This issue is related to DDL of TiDB. component/lightning This issue is related to Lightning of TiDB. severity/moderate type/bug This issue is a bug.

Comments

@D3Hunter
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

with store-write-bwlimit set, if write tikv is slow(might due to tikv io overload), and we cannot wait enough token to write to tikv within 15 minutes, it will report this error, we should retry on this

[errorVerbose="rate: Wait(n=28406) would exceed context deadline
github.com/pingcap/errors.AddStack
    /go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174
github.com/pingcap/errors.Trace
    /go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doWrite.func5
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/region_job.go:354
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doWrite
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/region_job.go:395
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).writeToTiKV
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/region_job.go:190
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).executeJob
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1436
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).startWorker
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1345
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.func5
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1678
golang.org/x/sync/errgroup.(*Group).Go.func1
    /go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1650"

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

@D3Hunter D3Hunter added type/bug This issue is a bug. severity/moderate component/ddl This issue is related to DDL of TiDB. labels Feb 28, 2024
@D3Hunter D3Hunter changed the title retry when write tikv is slow and rate limiter context timeout lightning: retry when write tikv is slow and rate limiter context timeout Feb 28, 2024
@D3Hunter D3Hunter added the component/lightning This issue is related to Lightning of TiDB. label Feb 28, 2024
@lance6716
Copy link
Contributor

Apart from retry, I can change the hard-coded 15 minutes timeout to a computed value from store-write-bwlimit

@lance6716 lance6716 self-assigned this Feb 29, 2024
@lance6716 lance6716 added affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects 5.4.x versions. affects-6.1 affects-6.5 affects-7.1 affects-7.5 labels Feb 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects 5.4.x versions. affects-6.1 affects-6.5 affects-7.1 affects-7.5 component/ddl This issue is related to DDL of TiDB. component/lightning This issue is related to Lightning of TiDB. severity/moderate type/bug This issue is a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants