Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lightning: split backoff is not long enough, and fail when EBS BR disabled region split #49517

Closed
mittalrishabh opened this issue Dec 16, 2023 · 1 comment · Fixed by #49518
Closed

Lightning: split backoff is not long enough, and fail when EBS BR disabled region split #49517

mittalrishabh opened this issue Dec 16, 2023 · 1 comment · Fixed by #49518
Labels
affects-6.5 affects-7.1 affects-7.5 component/lightning This issue is related to Lightning of TiDB. severity/major type/bug This issue is a bug.

Comments

@mittalrishabh
Copy link
Contributor

mittalrishabh commented Dec 16, 2023 •
edited by D3Hunter

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. Run EBS BR while lightning is running. It might fail lightning if BR suspends region split for more than 1 - 2 min.

2. What did you expect to see? (Required)

Lightning to not fail if region split are suspended because of EBS BR

3. What did you see instead (Required)

Lightning failed

4. What is your TiDB version? (Required)

6.5
@mittalrishabh mittalrishabh added the type/bug This issue is a bug. label Dec 16, 2023
@mittalrishabh mittalrishabh mentioned this issue Dec 16, 2023
Merged
13 tasks
@aytrack aytrack added severity/major component/lightning This issue is related to Lightning of TiDB. labels Dec 18, 2023
@D3Hunter
Copy link
Contributor 

failed to split: Err(Error(Kv(Error(Other(\\\"[src/server/raftkv/raft_extension.rs:128]: oneshot canceled\\\")))))\" "]

the error happens when doing EBS based backup, in which lightning don't support it

@D3Hunter D3Hunter changed the title Lightning:backoff if split and scatter fails Lightning: split backoff is not long enough, and fail when EBS BR disabled region split Dec 19, 2023
ti-chi-bot bot pushed a commit that referenced this issue Jan 8, 2024
@mittalrishabh
Lightning: increase backoff if split fails (#49518)
2a564d4 
close #49517
@ti-chi-bot ti-chi-bot mentioned this issue Jan 8, 2024
Merged
13 tasks
AilinKid pushed a commit to AilinKid/tidb that referenced this issue Jan 17, 2024
@mittalrishabh @AilinKid
Lightning: increase backoff if split fails (pingcap#49518)
846fb38 
close pingcap#49517
ti-chi-bot bot pushed a commit that referenced this issue Jan 25, 2024
@ti-chi-bot
Lightning: increase backoff if split fails (#49518) (#50164)
6f6386a 
close #49517
@ti-chi-bot ti-chi-bot mentioned this issue Mar 20, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue May 10, 2024
@ti-chi-bot
Lightning: increase backoff if split fails (#49518) (#51929)
5678c38 
close #49517
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.5 affects-7.1 affects-7.5 component/lightning This issue is related to Lightning of TiDB. severity/major type/bug This issue is a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Lightning: increase backoff if split fails mittalrishabh/tidb
4 participants
@lance6716 @D3Hunter @aytrack @mittalrishabh