Skip to content

GDB server gets killed too early on debug stop, causing the target to halt #1122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
robin96c opened this issue Apr 22, 2025 · 3 comments
Open

Comments

@robin96c
Copy link

Describe the bug
When I stop the debug session, the target usually (not always) remains halted. It doesn't matter when I exit from a halted or running target.

To Reproduce

  • STM32H7 microcontroller
  • cortex-debug v1.12.1
  • using PyOCD - see launch.json below

Expected behavior
The target should keep running. This is the expected behavior I get when manually starting a gdb + PyOCD terminal session. If I quit gdb, PyOCD will exit automatically as well and leave the target running.

Fix
I noticed that the PyOCD gdb-server gets forcefully killed too quickly. Normally it will exit on its own gracefully, and in such a case the target continues to run as expected. But I noticed that this graceful exit can take anywhere from a couple of milliseconds to 3 seconds. By that time, cortex-debug will have called gdb-exit and/or forcefully kill the gdb process. By increasing the timeout, the issue is completely fixed for my case.

My forked commit that fixes the issue for me: 3541297

Notice that I increased the timeout by a lot (10 seconds). This is because the PyOCD server will exit on its own anyway. So it doesn't just hang for 10 seconds, the debug session stops whenever PyOCD exits. I don't know if this is the same behavior for all gdb servers though.

Related issues: same symptoms, not sure if the cause is the same

launch.json
        {
            "name": "TESTING",
            "cwd": "${workspaceFolder}",
            "executable": "./build/app.elf",
            "request": "attach",
            "breakAfterReset": false,
            "type": "cortex-debug",
            "showDevDebugOutput": "both",
            "servertype": "pyocd",
        },
Additional logs
From /tmp/cortex-debug-server-exiting.log 

[2025-04-18T12:07:58.169Z] ppid=490172 pid=490172 TESTING: ******* Starting new session request type="attach"
[2025-04-18T12:07:58.187Z] ppid=490172 pid=490182 GDB started ppid=490172 pid=490182
[2025-04-18T12:08:02.328Z] ppid=490172 pid=490182 TESTING: Begin disconnectRequest
[2025-04-18T12:08:02.685Z] ppid=490172 pid=490182 GDB: exited
[2025-04-18T12:08:02.686Z] ppid=490172 pid=490182 TESTING: quitEvent: Killing server
[2025-04-18T12:08:02.686Z] ppid=490172 pid=490182 GDBServer(490184): forcing an exit with kill()
[2025-04-18T12:08:02.695Z] ppid=490172 pid=490182 TESTING: disconnectRequest sendResponse 3
[2025-04-18T12:08:02.697Z] ppid=490172 pid=490182 TESTING: quitEvent: sending VSCode TerminatedEvent
[2025-04-18T12:08:02.701Z] ppid=490172 pid=490182 GDBServer(490184): exited code=null signal=SIGTERM

[2025-04-18T12:03:27.861Z] ppid=479881 pid=479881 TESTING: ******* Starting new session request type="attach"
[2025-04-18T12:03:27.892Z] ppid=479881 pid=479891 GDB started ppid=479881 pid=479891
[2025-04-18T12:03:33.545Z] ppid=479881 pid=479891 TESTING: Begin disconnectRequest
[2025-04-18T12:03:34.263Z] ppid=479881 pid=479891 GDB Kill timer expired for a disconnect+exit, so forcing a kill
[2025-04-18T12:03:34.264Z] ppid=479881 pid=479891 GDB kill()
[2025-04-18T12:03:34.264Z] ppid=479881 pid=479891 GDB: exited
[2025-04-18T12:03:34.264Z] ppid=479881 pid=479891 TESTING: quitEvent: Killing server
[2025-04-18T12:03:34.264Z] ppid=479881 pid=479891 GDBServer(479893): forcing an exit with kill()
[2025-04-18T12:03:34.270Z] ppid=479881 pid=479891 TESTING: disconnectRequest sendResponse 3
[2025-04-18T12:03:34.275Z] ppid=479881 pid=479891 TESTING: quitEvent: sending VSCode TerminatedEvent
[2025-04-18T12:03:34.278Z] ppid=479881 pid=479891 GDBServer(479893): exited code=null signal=SIGTERM
@haneefdm
Copy link
Collaborator

pyocd should behave itself. Putting a 10second delay is not acceptable. It will affect restarts and in general a super bad idea. Even the one second is not normal but there are servers that do not respect a request for a graceful shutdown.

We asked the server to disconnect nicely, so it should do that. This used to happen with OpenOCD a long time ago. Now, it doesn't. It also prevents the next session to start because the USB is locked up.

Have to you looked into

  • What are we doing wrong? What spec did we violate? Please point that out. Increasing timeout is not a solution
  • What is pyocd doing that could be wrong

The target should keep running.

Why does it take pyocd 10 seconds to shutdown. What is it waiting for? Maybe another gdb connection. Maybe it should have an option to shutdown when the last gdb connection is disconnected. There is an option for some servers to run once or to be persistent. Well, pyocd seems to be neither.

This is also the gdb servers responsibility. Not ours. JLink had this problem and we had that fixed but pointing them to gdb's documentation. OpenOCD has that problem and we could NOT get that fixed. Please read gdb's documentation on what a proper shutdown procedure is.

You should take these issues to the creator of the gdb-server. Fixing them here is not the solution. GDB defines the rules and servers that mis-implemented should fix their stuff.

@haneefdm
Copy link
Collaborator

I may have a solution that is better than any timeout -- which we should never need. There is now a monitor command called. I am ashamed that it is even there because none of the gdb-servers did it properly or similarly.

monitor exit

Note that this monitor command seems to be new and may not be implemented by many gdb-servers. I can experiment with it.

https://sourceware.org/gdb/current/onlinedocs/gdb.html/Server.html#Other-Command_002dLine-Arguments-for-gdbserver-1

Please note that we follow rules that GDB has established on how gdb-servers should behave and GDB clients should behave. GDB is the master here. Both us and the gdb-servers are clients of its protocol.

@haneefdm
Copy link
Collaborator

https://pyocd.io/docs/gdbserver.html

See the Gdbserver exit section. Based on their own docs, why does pyocd hang around for up to 10 seconds? What if they change that to 20 seconds. What is the purpose? Do you know?

Btw, I have submitted PRs (I added the initial multi-core support) to pyocd. They were easy to work with and responsive.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants