Difficulties with breakpoints

[danrose501]

In summary, I've been using Ida for debugging a particular Windows application but am trying to switch over to Ghidra 10. I've tried using dbgeng locally in IN-VM and locally via GADP. I am able to get he application to run but I'm having difficulty with breakpoints. I have only been able to set BPs in the Interpreter window with the bp <address> command after the app has been started and paused. However, when I resume (F5) the process it continually stops in ntdll.dll (I think). After many F5s after I delete the BP and the process then runs.

Per suggestions from d-millar I installed the latest SDK/Windbg. I then started notepad.exe with the debugger using locally IN-VM and, after the initial stopping in ntdll.dll I set a BP a couple of instructions later, F5 and voila, the BP hit so it appears I can debug that app.

My problems are no doubt due to "cockpit error" so I would appreciate any thoughts/suggestions of things to try.

[d-millar]

@danrose501 Are you getting any other messages, esp. in the Interpreter, indicating why the application is stopping? And is the behavior the same if you run the application in Windbg? The debugger typically stops for events and exceptions that it either doesn't know how to handle or that you've explicitly marked "Break". If you open the Debug node in the Objects view (under Process), you'll see subnodes for Events and Exceptions, and within each is an Execute node indicating how that event/exception is handled. "Break" means the obvious thing, and you can toggle through the other options using "T". So, if, for example, the process breaks that you're F5ing through corresponded to DLL loads, I would check the Module Load event and make sure it didn't accidentally get set to "Break". Alternatively, you might be hitting some exception that normally wouldn't cause the debugger to stop or the process to fail (C Runtime Exceptions are a typical candidate), but, if the debugger hasn't been explicitly told not to stop, it will until you define that exception and the behavior you'd like to see in the debugger.

[d-millar]

ok, just to be clear, I’m not talking about errors in Ghidra (which would correspond to the dialog boxes popping up, although we’re obviously interested those too) - am talking about messages the Interpreter from the underlying debugger saying things like, “Error 0x8000blah encountered”.

[danrose501]

Roger that - I'll be sure to gather that info tomorrow.

[danrose501]

I've checked the entries in the Objects window, all are disabled (open red circles) and none are set to "Break". Here's what I've done so far. I load the app in the debug tool, F5, wait for everything to settle, verify that the app is running normally, Pause, insert a BP and F5 to get it going.
dbgeng locally IN-VM, dbgeng locally via GADP, dbgmodel locally via GADP and dbgmodel locally IN-VM:
dbgeng locally IN-VM is very slow because the debugger stops for each "ModLoad:" dll
Unable to insert BP by right clicking in the code browser window so I used "bp" in the Interpreter window
Following BP insertion, the app won't run as I get the following in the Interpreter window for each F5:

Unable to insert breakpoint 0 at xxxxxxxxx, Win32 error 0n299 "Only part of a ReadProcessMemory of WriteProcessMemory request was completed." The breakpoint was set with BP. If you want breakpoints to track module load/unload state you must us BU. Bp 0 at xxxxxxxxx

Somehow yesterday I was able to set BPs by right clicking in the code window but I can't seem to do that now.

[danrose501]

As a follow up, I just figured out how to set BPs in WIndbg and was successful in setting them in my application with WIndbg.

[d-millar]

Soooo, same address, same time, bp in windbg works, bp in the Interpreter in Ghidra does not - is that correct? If so, what happens if you copy the dbgXXX.dll's from the Windbg folder to the JDK bin directory?

[d-millar]

@danrose501 So am guessing you haven't gotten a copy of windbg yet. If and when you get that, running your target in windbg would really really help us figure out what's going on. Leaving that aside, a few notes....

First, open and closed red circles in Objects do NOT mean enabled and disabled - solid means it has elements, i.e. is a container, and open means it does not. So, one question I still have is, particularly for dbgeng IN-VM, what is the setting for Debugger->Sessions->[0]->Processes->[xxx]->Debug->Events->Load Module->Execute & Continue? If it's set to "Output-only" but still stopping at every load, something odd is going on. I would like to know why. Assuming there are no relevant messages in the Interpreter window (the breakpoint messages are not relevant), try running Ghidra with support/ghidraDebug, rather than ghidraRun. Some chance we can find a clue in the console.

On the BP front, can you dop the equivalent of the "notepad test" for your target, i.e. set a breakpoint at an instruction you absolutely know has to be executed?

[danrose501]

I have installed Windbg and have been able to set BPs in my target application with it.

Today I verified that my object settings were as described above and there was no stopping when modules loaded so I must have had things messed up yesterday.

I am using Ghidra 10.1.2 with the IN-VM debugger was able to duplicate the notepad experiment with my app as follows:

1. Load the app and execute with F5
2. After everything is up and stable, the code window points to ntdll.dll (180070d40 - RtlUserThereadStart) which has code very similar to that of notepad.exe
3. Add BP just below the TEST instruction that executes when the app is gracefully exited but the BP window only shows a BP in the upper pane with solid gray circle
4. Pause - BP now shows in top/bottom panes with solid red circles
5. F5 - app continues as normal
6. Exit - BP fires and I have to F5 again for it to complete

However, I still cannot set a BP in the app itself

[d-millar]

OK, sorry - wasn't sufficiently clear.... By "the equivalent of the notepad test", I meant can you:
(1) Load your app (NOT notepad) into the debugger
(2) Run it using the IN-VM dbgeng
(3) Wait until it stops at the initial load point.
(4) Set a breakpoint two or three instructions after the point it stopped but before any conditional jumps or calls.
(5) Hit resume and see if the breakpoint fires.

[danrose501]

This morning I copied the WIndbg/dbgeng.dll into the java/bin directory but there was no change in the behavior. FWIW, I did learn that I needed the x86 version and not the 64bit bit version.

I did understand what your original notepad.exe experiment but my app does not pause at the initial load point as does notepad so that's why I improvised last week. BUT it did show I could set BPs in ntdll. My app starts 40+ threads when it comes up so maybe that has something to do with that?

[d-millar]

@danrose501 I don't think the thread count should affect anything other than perhaps a slow start. Needing the x86 version does not make sense to me. Do you have a sense as to why? Are you running on a 32-bit OS? Did you try running using ghidraDebug? Basically, am looking for anything approaching a clue here. :)

[danrose501]

OK, some minor success - apparently I've lied to you. This afternoon my app stopped on startup just like notepad and I was able to set a BP at the TEST instruction, F5 and the BP fired. HOORAY - but it wasn't 100%, more like 4 out of times. I tried letting it stop at that point, setting a BP in the app code and that fired, but I could only do it once. Don't know if it's necessary but things seem to work best if I close the debug tool and Ghidra between tries. I'll continue playing with it tomorrow.

[danrose501]

I decided to see what messages are generated in the debug window when I set a BP in my app that should fire when I apply a specific stimulus. So I followed this procedure:

1. start my app with the debugger
2. wait for it to do all of its initial housekeeping and settle
3. PAUSE
4. set the BP with the interpreter using WIndbg parlance (bp <app_name>+offset)
5. BP shows up in BP window w/ solid red circles in top and bottom panes
6. In the debug window I see the following:

INFO ***BreakpointChanged: [BREAKPOINTS], 0 on ``ThreadDebugClient,6,Ghidra] (DbgDebugEventCallbacksAdpater.java:119)
WARN Breakpoint 0 is not known (DbgManagerImpl.java:297

7. F5
8. In the debug window I see:
   ERROR Could not auto-read memory: java.util.concurrent.CompletionException: ghidra.dbg.error.DebuggerModelAccessException: Cannot process command readMemory while engine is waiting for events (DebuggerReadMemoryTrait:java:230)
INFO`` ***BreakpointChanged: [BREAKPOINTS], 0 on ThreadDebugClient,6,Ghidra] (DbgDebugEventCallbackAdapter,java:119)
9. apply the stimulus - BP doesn't hit but code runs normally.

Messages in step 6 appear to be benign but problem appears to be in step 8. Can you explain the ghidra.dbg.error message?

[danrose501]

Should add one note: in steps 4 and 5, I don't see any indication of a BP in the static code browser window - should I?

[d-millar]

OK, so re (6), that message is fine. Windbg does a 3-part install for breakpoints, where is adds it, sets the address, and enables it (if I remember correctly), and that message is a casualty of that process. Re (8), also OK - that's telling you that the process is running and some (presumably delayed) request to read memory (likely from the GUI) showed up and could not be serviced because the process was running.

Some bits I don't quite understand or maybe you could elaborate on.... First, <app_name>+offset - is this breakpoint one that is guaranteed to be in the execution path, i.e. are you absolutely sure it should fire? Does it always fire in Windbg? Second, re added note, when you say it doesn't show up in the static window, can I assume you are viewing the file where you set the breakpoint? (Sorry if that seems like a dumb question, but just checking....) If yes and no breakpoint, does the address in Breakpoints match the one you're on in the file. Finally, another dumb question, what do you mean be "apply the stimulus"?

[danrose501]

I have finally had success in setting a BP in my application and here's what I learned about using the Ghidra debugger (v10.1.2) in my Windows application:

1. The debugged app needs to be started and allowed to settle into a steady state - Objects window finished updating
2. The app must be paused to enter BPs- preferably with the PAUSE button in the Interpreter
3. All BP commands must be entered via the Interpreter
4. All BPs must be entered in Windbg parlance e.g., bp app_name+offset in virtual space

Observations:

1. BPs usually can't be set by right-clicking a line in the code browser window and this should be fixed. For some reason ntdll.dll showed up as a separate code browser tab and in one instance (on startup) I could right-click and set a BP in ntdll.dll that would fire, but not so with my app.
2. BPs should be settable while app is running (I know a fix is being worked on)
3. When a BP is set there is no visual indication in the code browser window - this would be helpful.
4. The PAUSE in the Objects window is not reliable - this should be fixed
5. A User Manual section of the Help documents should be written that gets a new user started, regardless of his/her experience with RE.

[d-millar]

Interesting observations - am interested in following up if and when you have time. Some quick comments:

1. Definitely true - think we need a more obvious indicator of this.
2. Also true - in general for windows debuggers, you are not allowed to do anything in the running state.
3. Not sure why this is true for your target - not generally true.
4. Also not generally true, so interested in pursuing.

Re Observations:
1a. Right-click->Toggle does work in the general case, so worth following-up.
1b. Programs only appear in the Static Listing if you import them. Did you import ntdll.dll? Did you import your app?
2. Even if we allow this, the BP will not be active until you break. Windbg does not permit breaking on-the-fly.
3. Also, not generically true, so interested in why in your case.
4. Agreed.
5. Possibly, can you id things explicitly missing in Help?

[d-millar]

Also, if the name in the Modules view and the name of your program in the Static view are not exact matches, you're probably going to have issues. You'll need to right-click on the module and assign it by hand to the program.

[d-millar]

Another question for you: what is Objects' Process->Debug->Events->Initial Breakpoint->Execute set to?

[danrose501]

GLORYOSKI! I think that spoke to the bulk of my issues. I wasn't paying any attention to the Modules window (more like "Huh, wonder what that does") and when I checked this morning sure enough my app was named sam.exe in Modules and the listing was named zorch.exe. So when I right-clicked on sam.exe in Modules I selected Map Module to zorch.exe and voila, I was able to set BPs from the listing and there is a BP indicator in the listing at that line.

I think my app is stopping at the very beginning because as soon as I start it I am able to see its address in Modules, add the offset for my BP and set it in the Interpreter with bp 0xcomputed_address.

Objects' Process->Debug->Events->Initial Breakpoint->Execute is set to Ignore.

Thanks again for you help, this really helped.

[d-millar]

Wow, ok - I really should have put 2 and 2 together a lot earlier, but super happy it's working for you. And, with luck, will be useful to other folks (and probably as a reminder to me). Still interested in any thoughts you might have a specifics for new folks, especially things missing on the Debugger/Troubleshooting.html.

[danrose501]

I have written a Confluence page, complete with screenshots documenting how to get started with the debugger. I'll be happy to share that with you when you get back in to the office.