Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tiup upgrade from v6.5.0 to nightly fail for telemetry related panic #50426

Closed
seiya-annie opened this issue Jan 15, 2024 · 1 comment · Fixed by #50625
Closed

tiup upgrade from v6.5.0 to nightly fail for telemetry related panic #50426

seiya-annie opened this issue Jan 15, 2024 · 1 comment · Fixed by #50625
Assignees
Labels
affects-7.6 severity/major sig/diagnosis SIG: Diagnosis sig/sql-infra SIG: SQL Infra type/bug This issue is a bug.

Comments

@seiya-annie
Copy link

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. deploy v6.5.0 cluster with config:
server_configs:
 tidb:
   enable-telemetry: true
   new_collations_enabled_on_first_bootstrap: false
   performance.gogc: 200
  1. upgrade to nightly

2. What did you expect to see? (Required)

upgrade successfully

3. What did you see instead (Required)

panic: runtime error: index out of range [0] with length 0
goroutine 7710 [running]:
github.com/pingcap/tidb/pkg/util/chunk.(*Column).GetString(...)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/chunk/column.go:588
github.com/pingcap/tidb/pkg/util/chunk.Row.GetString({0xc004237130?, 0xf?}, 0x0?)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/chunk/row.go:76 +0xcd
github.com/pingcap/tidb/pkg/telemetry.getClusterInfo({0x6416a28, 0xc003ea1a10}, {0x648dd40?, 0xc004e6cf00})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/telemetry/data_cluster_info.go:65 +0x285
github.com/pingcap/tidb/pkg/telemetry.generateTelemetryData({0x648dd40, 0xc004e6cf00}, {0xc003401d10, 0x24})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/telemetry/data.go:45 +0x186
github.com/pingcap/tidb/pkg/telemetry.reportUsageData({0x648dd40, 0xc004e6cf00}, 0xc004faafc0)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/telemetry/telemetry.go:116 +0xd6
github.com/pingcap/tidb/pkg/telemetry.ReportUsageData({0x648dd40, 0xc004e6cf00}, 0xc004faafc0)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/telemetry/telemetry.go:159 +0x8c
github.com/pingcap/tidb/pkg/telemetry.InitialRun({0x648dd40, 0xc004e6cf00}, 0xc002afef70?)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/telemetry/telemetry.go:177 +0x2e8
github.com/pingcap/tidb/pkg/domain.(*Domain).TelemetryReportLoop(0xc0027d2600, {0x648dd40?, 0xc004e6cf00?})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/domain/domain.go:1919 +0x70
github.com/pingcap/tidb/pkg/session.bootstrapSessionImpl.func1()
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/session.go:3384 +0x3a
created by github.com/pingcap/tidb/pkg/session.bootstrapSessionImpl in goroutine 1
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/session.go:3383 +0xa27
panic: runtime error: index out of range [0] with length 0

4. What is your TiDB version? (Required)

master

@mornyx
Copy link
Contributor

mornyx commented Jan 16, 2024

Minimal reproducible test case:

func TestXxx(t *testing.T) {
    ctx := util.WithInternalSourceType(context.TODO(), kv.InternalTxnAdmin)
    store := testkit.CreateMockStore(t)
    se, err := session.CreateSession4Test(store)
    require.NoError(t, err)
    defer se.Close()
    exec := se.(sqlexec.RestrictedSQLExecutor)
    rows, _, err := exec.ExecRestrictedSQL(ctx, nil, `SELECT TYPE, INSTANCE, STATUS_ADDRESS, VERSION, GIT_HASH, START_TIME, UPTIME FROM information_schema.cluster_info`)
    require.NoError(t, err)
    for _, row := range rows {
        if row.Len() < 7 {
            continue
        }
        _ = row.GetString(0)
        _ = row.GetString(1)
        _ = row.GetString(2)
        _ = row.GetString(3)
        _ = row.GetString(4)
        _ = row.GetString(5)
        _ = row.GetString(6)
    }
}

Test passed on release-6.5, but on master (im on this commit 9fb89b002cd0ea410404756eab0ce710dbf68656), we got panic:

--- FAIL: TestXxx (1.07s)
panic: runtime error: index out of range [0] with length 0 [recovered]
	panic: runtime error: index out of range [0] with length 0
goroutine 213 [running]:
testing.tRunner.func1.2({0x1088e5e60, 0x14006e43218})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1545 +0x1c4
testing.tRunner.func1()
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1548 +0x360
panic({0x1088e5e60?, 0x14006e43218?})
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/panic.go:914 +0x218
github.com/pingcap/tidb/pkg/util/chunk.(*Column).GetString(...)
	/Users/mornyx/Desktop/repos/github.com/pingcap/tidb/pkg/util/chunk/column.go:588
github.com/pingcap/tidb/pkg/util/chunk.Row.GetString(...)
	/Users/mornyx/Desktop/repos/github.com/pingcap/tidb/pkg/util/chunk/row.go:76
github.com/pingcap/tidb/pkg/telemetry_test.TestXxx(0x14000115798?)
	/Users/mornyx/Desktop/repos/github.com/pingcap/tidb/pkg/telemetry/telemetry_test.go:166 +0x514
testing.tRunner(0x14000f22340, 0x108be4398)
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0xe8
created by testing.(*T).Run in goroutine 1
	/opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x33c
FAIL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-7.6 severity/major sig/diagnosis SIG: Diagnosis sig/sql-infra SIG: SQL Infra type/bug This issue is a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants