Skip to content

Goroutine leak #1793

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Conversation

bufferflies
Copy link

@bufferflies bufferflies commented Jun 11, 2025

ref: tikv/pd#9402

goroutine profile:

goroutine profile: total 2916
681 @ 0xf61910 0xf60825 0xf922e5 0xf922c2 0xf91f33 0x11ee43f 0x11f00c8 0x11eeb10 0xf83007 0x231e0da 0x231e097 0xf46921
#	0xf6190f	syscall.Syscall6+0x2f							/usr/local/go/src/syscall/syscall_linux.go:91
#	0xf60824	syscall.Splice+0x44							/usr/local/go/src/syscall/zsyscall_linux_amd64.go:1356
#	0xf922e4	internal/poll.splice+0x184						/usr/local/go/src/internal/poll/splice_linux.go:155
#	0xf922c1	internal/poll.spliceDrain+0x161						/usr/local/go/src/internal/poll/splice_linux.go:92
#	0xf91f32	internal/poll.Splice+0x172						/usr/local/go/src/internal/poll/splice_linux.go:42
#	0x11ee43e	net.splice+0xde								/usr/local/go/src/net/splice_linux.go:39
#	0x11f00c7	net.(*TCPConn).readFrom+0x27						/usr/local/go/src/net/tcpsock_posix.go:48
#	0x11eeb0f	net.(*TCPConn).ReadFrom+0x2f						/usr/local/go/src/net/tcpsock.go:130
#	0xf83006	io.copyBuffer+0x146							/usr/local/go/src/io/io.go:416
#	0x231e0d9	io.Copy+0x79								/usr/local/go/src/io/io.go:389
#	0x231e096	github.com/pingcap/tidb-dashboard/pkg/tidb.(*proxy).serve.func1+0x36	/root/go/pkg/mod/github.com/pingcap/[email protected]/pkg/tidb/proxy.go:117

664 @ 0xf11dee 0xf23a38 0xf23a0f 0xf42865 0xf8a7a5 0x11d3472 0x11e7236 0x231dfdf 0xf46921
#	0xf42864	internal/poll.runtime_Semacquire+0x24				/usr/local/go/src/runtime/sema.go:67
#	0xf8a7a4	internal/poll.(*FD).Close+0x64					/usr/local/go/src/internal/poll/fd_unix.go:113
#	0x11d3471	net.(*netFD).Close+0x31						/usr/local/go/src/net/fd_posix.go:37
#	0x11e7235	net.(*conn).Close+0x35						/usr/local/go/src/net/net.go:203
#	0x231dfde	github.com/pingcap/tidb-dashboard/pkg/tidb.(*proxy).serve+0xfe	/root/go/pkg/mod/github.com/pingcap/[email protected]/pkg/tidb/proxy.go:123

17 @ 0xf11dee 0xf09837 0xf40225 0xf89827 0xf9238f 0xf9237d 0xf91f33 0x11ee43f 0x11f00c8 0x11eeb10 0xf83007 0x231dfcf 0x231df8a 0xf46921
#	0xf40224	internal/poll.runtime_pollWait+0x84				/usr/local/go/src/runtime/netpoll.go:343
#	0xf89826	internal/poll.(*pollDesc).wait+0x26				/usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#	0xf9238e	internal/poll.(*pollDesc).waitRead+0x22e			/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#	0xf9237c	internal/poll.spliceDrain+0x21c					/usr/local/go/src/internal/poll/splice_linux.go:100
#	0xf91f32	internal/poll.Splice+0x172					/usr/local/go/src/internal/poll/splice_linux.go:42
#	0x11ee43e	net.splice+0xde							/usr/local/go/src/net/splice_linux.go:39
#	0x11f00c7	net.(*TCPConn).readFrom+0x27					/usr/local/go/src/net/tcpsock_posix.go:48
#	0x11eeb0f	net.(*TCPConn).ReadFrom+0x2f					/usr/local/go/src/net/tcpsock.go:130
#	0xf83006	io.copyBuffer+0x146						/usr/local/go/src/io/io.go:416
#	0x231dfce	io.Copy+0xee							/usr/local/go/src/io/io.go:389
#	0x231df89	github.com/pingcap/tidb-dashboard/pkg/tidb.(*proxy).serve+0xa9	/root/go/pkg/mod/github.com/pingcap/[email protected]/pkg/tidb/proxy.go:122


11 @ 0xf11dee 0xf09837 0xf40225 0xf89827 0xf8ac5a 0xf8ac48 0x11d3605 0x11e6f05 0x22c1fdb 0x22c2107 0x22cba45 0x22cc727 0x22c6b1f 0x21c6257 0x21c89b7 0x21c884f 0x21c7042 0x21c8765 0x21c9431 0x230d90c 0x22199ff 0x231b6cd 0x23321ee 0x257fd09 0x233164a 0x233151f 0x22514a2 0x23337ca 0x1e9628b 0x25b522a 0x1e9628b 0x25b4925
#	0xf40224	internal/poll.runtime_pollWait+0x84								/usr/local/go/src/runtime/netpoll.go:343
#	0xf89826	internal/poll.(*pollDesc).wait+0x26								/usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#	0xf8ac59	internal/poll.(*pollDesc).waitRead+0x279							/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#	0xf8ac47	internal/poll.(*FD).Read+0x267									/usr/local/go/src/internal/poll/fd_unix.go:164
#	0x11d3604	net.(*netFD).Read+0x24										/usr/local/go/src/net/fd_posix.go:55
#	0x11e6f04	net.(*conn).Read+0x44										/usr/local/go/src/net/net.go:179
#	0x22c1fda	github.com/go-sql-driver/mysql.(*buffer).fill+0x23a						/root/go/pkg/mod/github.com/go-sql-driver/[email protected]/buffer.go:90
#	0x22c2106	github.com/go-sql-driver/mysql.(*buffer).readNext+0x26						/root/go/pkg/mod/github.com/go-sql-driver/[email protected]/buffer.go:119
#	0x22cba44	github.com/go-sql-driver/mysql.(*mysqlConn).readPacket+0x84					/root/go/pkg/mod/github.com/go-sql-driver/[email protected]/packets.go:32
#	0x22cc726	github.com/go-sql-driver/mysql.(*mysqlConn).readHandshakePacket+0x26				/root/go/pkg/mod/github.com/go-sql-driver/[email protected]/packets.go:187
#	0x22c6b1e	github.com/go-sql-driver/mysql.(*connector).Connect+0x57e					/root/go/pkg/mod/github.com/go-sql-driver/[email protected]/connector.go:81
#	0x21c6256	database/sql.(*DB).conn+0x716									/usr/local/go/src/database/sql/sql.go:1387
#	0x21c89b6	database/sql.(*DB).query+0x56									/usr/local/go/src/database/sql/sql.go:1721
#	0x21c884e	database/sql.(*DB).QueryContext.func1+0x4e							/usr/local/go/src/database/sql/sql.go:1704
#	0x21c7041	database/sql.(*DB).retry+0x41									/usr/local/go/src/database/sql/sql.go:1538
#	0x21c8764	database/sql.(*DB).QueryContext+0xc4								/usr/local/go/src/database/sql/sql.go:1703
#	0x21c9430	database/sql.(*DB).QueryRowContext+0x30								/usr/local/go/src/database/sql/sql.go:1804
#	0x230d90b	gorm.io/driver/mysql.Dialector.Initialize+0x16b							/root/go/pkg/mod/gorm.io/driver/[email protected]/mysql.go:113
#	0x22199fe	gorm.io/gorm.Open+0x97e										/root/go/pkg/mod/gorm.io/[email protected]/gorm.go:179
#	0x231b6cc	github.com/pingcap/tidb-dashboard/pkg/tidb.(*Client).OpenSQLConn+0x50c				/root/go/pkg/mod/github.com/pingcap/[email protected]/pkg/tidb/client.go:145
#	0x23321ed	github.com/pingcap/tidb-dashboard/pkg/apiserver/user.VerifySQLUser+0x4d				/root/go/pkg/mod/github.com/pingcap/[email protected]/pkg/apiserver/user/verify_sql_user.go:33
#	0x257fd08	github.com/pingcap/tidb-dashboard/pkg/apiserver/user/sqlauth.(*Authenticator).Authenticate+0x88	/root/go/pkg/mod/github.com/pingcap/[email protected]/pkg/apiserver/user/sqlauth/sqlauth.go:44
#	0x2331649	github.com/pingcap/tidb-dashboard/pkg/apiserver/user.(*AuthService).authForm+0x89		/root/go/pkg/mod/github.com/pingcap/[email protected]/pkg/apiserver/user/auth.go:239
#	0x233151e	github.com/pingcap/tidb-dashboard/pkg/apiserver/user.NewAuthService.func1+0x7e			/root/go/pkg/mod/github.com/pingcap/[email protected]/pkg/apiserver/user/auth.go:121
#	0x22514a1	github.com/breeswish/gin-jwt/v2.(*GinJWTMiddleware).LoginHandler+0x41				/root/go/pkg/mod/github.com/breeswish/gin-jwt/[email protected]/auth_jwt.go:437
#	0x23337c9	github.com/pingcap/tidb-dashboard/pkg/apiserver/user.(*AuthService).LoginHandler+0x29		/root/go/pkg/mod/github.com/pingcap/[email protected]/pkg/apiserver/user/auth.go:344
#	0x1e9628a	github.com/gin-gonic/gin.(*Context).Next+0x2a							/root/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174
#	0x25b5229	github.com/pingcap/tidb-dashboard/pkg/apiserver.newAPIHandlerEngine.ErrorHandlerFn.func3+0x29	/root/go/pkg/mod/github.com/pingcap/[email protected]/util/rest/error.go:70
#	0x1e9628a	github.com/gin-gonic/gin.(*Context).Next+0x2a							/root/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174
#	0x25b4924	github.com/pingcap/tidb-dashboard/pkg/apiserver.newAPIHandlerEngine.Gzip.func2+0x1e4		/root/go/pkg/mod/github.com/gin-contrib/[email protected]/gzip.go:47

3 @ 0xf11dee 0xf09837 0xf40225 0xf89827 0xf8ac5a 0xf8ac48 0x11d3605 0x11e6f05 0x22c1fdb 0x22c2107 0x22cbaca 0x22d145d 0x22d2f5b 0x21ced27 0x21cebe9 0x21d10a2 0x21ceb65 0x221f95f 0x2302a25 0x220d119 0x2214272 0x2593394 0x2595e51 0x1e9628b 0x2595808 0x22511cb 0x2251068 0x259593d 0x1e9628b 0x25b522a 0x1e9628b 0x25b4925
#	0xf40224	internal/poll.runtime_pollWait+0x84															/usr/local/go/src/runtime/netpoll.go:343
#	0xf89826	internal/poll.(*pollDesc).wait+0x26															/usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#	0xf8ac59	internal/poll.(*pollDesc).waitRead+0x279														/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#	0xf8ac47	internal/poll.(*FD).Read+0x267																/usr/local/go/src/internal/poll/fd_unix.go:164
#	0x11d3604	net.(*netFD).Read+0x24																	/usr/local/go/src/net/fd_posix.go:55
#	0x11e6f04	net.(*conn).Read+0x44																	/usr/local/go/src/net/net.go:179
#	0x22c1fda	github.com/go-sql-driver/mysql.(*buffer).fill+0x23a													/root/go/pkg/mod/github.com/go-sql-driver/[email protected]/buffer.go:90
#	0x22c2106	github.com/go-sql-driver/mysql.(*buffer).readNext+0x26													/root/go/pkg/mod/github.com/go-sql-driver/[email protected]/buffer.go:119
#	0x22cbac9	github.com/go-sql-driver/mysql.(*mysqlConn).readPacket+0x109												/root/go/pkg/mod/github.com/go-sql-driver/[email protected]/packets.go:68
#	0x22d145c	github.com/go-sql-driver/mysql.(*binaryRows).readRow+0x3c												/root/go/pkg/mod/github.com/go-sql-driver/[email protected]/packets.go:1174
#	0x22d2f5a	github.com/go-sql-driver/mysql.(*binaryRows).Next+0x5a													/root/go/pkg/mod/github.com/go-sql-driver/[email protected]/rows.go:198
#	0x21ced26	database/sql.(*Rows).nextLocked+0x106															/usr/local/go/src/database/sql/sql.go:3019
#	0x21cebe8	database/sql.(*Rows).Next.func1+0x28															/usr/local/go/src/database/sql/sql.go:2994
#	0x21d10a1	database/sql.withLock+0x81																/usr/local/go/src/database/sql/sql.go:3502
#	0x21ceb64	database/sql.(*Rows).Next+0x84																/usr/local/go/src/database/sql/sql.go:2993
#	0x221f95e	gorm.io/gorm.Scan+0x101e																/root/go/pkg/mod/gorm.io/[email protected]/scan.go:258
#	0x2302a24	gorm.io/gorm/callbacks.Query+0x104															/root/go/pkg/mod/gorm.io/[email protected]/callbacks/query.go:26
#	0x220d118	gorm.io/gorm.(*processor).Execute+0x398															/root/go/pkg/mod/gorm.io/[email protected]/callbacks.go:130
#	0x2214271	gorm.io/gorm.(*DB).Find+0x131																/root/go/pkg/mod/gorm.io/[email protected]/finisher_api.go:170
#	0x2593393	github.com/pingcap/tidb-dashboard/pkg/apiserver/statement.(*Service).queryStatements+0x613								/root/go/pkg/mod/github.com/pingcap/[email protected]/pkg/apiserver/statement/queries.go:94
#	0x2595e50	github.com/pingcap/tidb-dashboard/pkg/apiserver/statement.(*Service).listHandler+0x190									/root/go/pkg/mod/github.com/pingcap/[email protected]/pkg/apiserver/statement/service.go:171
#	0x1e9628a	github.com/gin-gonic/gin.(*Context).Next+0x2a														/root/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174
#	0x2595807	github.com/pingcap/tidb-dashboard/pkg/apiserver/statement.registerRouter.MWConnectTiDB.func1+0x127							/root/go/pkg/mod/github.com/pingcap/[email protected]/pkg/apiserver/utils/tidb_conn.go:66
#	0x22511ca	github.com/gin-gonic/gin.(*Context).Next+0x32a														/root/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174
#	0x2251067	github.com/breeswish/gin-jwt/v2.(*GinJWTMiddleware).middlewareImpl+0x1c7										/root/go/pkg/mod/github.com/breeswish/gin-jwt/[email protected]/auth_jwt.go:403
#	0x259593c	github.com/pingcap/tidb-dashboard/pkg/apiserver/statement.registerRouter.(*AuthService).MWAuthRequired.(*GinJWTMiddleware).MiddlewareFunc.func4+0x1c	/root/go/pkg/mod/github.com/breeswish/gin-jwt/[email protected]/auth_jwt.go:365
#	0x1e9628a	github.com/gin-gonic/gin.(*Context).Next+0x2a														/root/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174
#	0x25b5229	github.com/pingcap/tidb-dashboard/pkg/apiserver.newAPIHandlerEngine.ErrorHandlerFn.func3+0x29								/root/go/pkg/mod/github.com/pingcap/[email protected]/util/rest/error.go:70
#	0x1e9628a	github.com/gin-gonic/gin.(*Context).Next+0x2a														/root/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174
#	0x25b4924	github.com/pingcap/tidb-dashboard/pkg/apiserver.newAPIHandlerEngine.Gzip.func2+0x1e4									/root/go/pkg/mod/github.com/gin-contrib/[email protected]/gzip.go:47

Signed-off-by: 童剑 <[email protected]>
Signed-off-by: 童剑 <[email protected]>
Copy link
Contributor

ti-chi-bot bot commented Jun 11, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign deardrops for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot requested a review from Renkai June 11, 2025 11:40
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@ti-chi-bot ti-chi-bot bot added the size/M label Jun 11, 2025
Signed-off-by: 童剑 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant