Debugging Chromium on Windows

Profiles

.sympath+ SRV*c:\Symbols*https://chromium-browser-symsrv.commondatastorage.googleapis.com

 

_NT_SYMBOL_PATH=SRV*C:\symbols*https://msdl.microsoft.com/download/symbols;SRV*C:\symbols*https://chromium-browser-symsrv.commondatastorage.googleapis.com

 

Multi-process issues

Chromium can be challenging to debug because of its multi-process architecture. When you select Run in the debugger, only the main browser process will be debugged. The code that actually renders web pages (the Renderer) and the plugins will be in separate processes that‘s not (yet!) being debugged. The ProcessExplorer tool has a process tree view where you can see how these processes are related. You can also get the process IDs associated with each tab from the Chrome Task Manager (right-click on an empty area of the window title bar to open).

Automatically attach to child processes

There are two Visual Studio extensions that enable the debugger to automatically attach to all Chrome processes, so you can debug all of Chrome at once. Microsoft‘s Child Process Debugging Power Tool is a standalone extension for this, and VsChromium is another option that bundles many other additional features. In addition to installing one of these extensions, you must run Visual Studio as Administrator, or it will silently fail to attach to some of Chrome‘s child processes.

Single-process mode

One way to debug issues is to run Chromium in single-process mode. This will allow you to see the entire state of the program without extra work (although it will still have many threads). To use single-process mode, add the command-line flag

This approach isn‘t perfect because some problems won‘t manifest themselves in this mode and some features don‘t work and worker threads are still spawned into new processes.

Manually attaching to a child process

--renderer-startup-dialog --no-sandbox

 

Semi-automatically attaching the debugger to child processes

V8 and Chromium

V8 supports many command-line flags that are useful for debugging. V8 command-line flags can be set via the Chromium command-line flag --js-flags; for instance:

chrome.exe --js-flags="--trace_exception --heap_stats"

Note that some V8 command-line flags exist only in the debug build of V8. For a list of all V8 flags try:

chrome.exe --js-flags="--help"

Graphics debugging

GPU Acceleration of rendering can be more easily debugged with tools. See:

• Graphics Debugging in Visual Studio 2013
• Graphical debugging with NVIDIA NSight

Debugging on another machine

Sometimes it‘s useful to debug installation and execution on a machine other than your primary build box. To run the installer on said other machine, first build the mini_installer target on your main build machine (e.g., ninja -C out\Debug mini_installer). Next, on the debug machine:

• Make the build machine‘s build volume available on the debug machine either by mounting it locally (e.g., Z:\) or by crafting a UNC path to it (e.g., \\builder\src)
• Open up a command prompt and change to a local disk
• Run src\tools\win\copy-installer.bat in the remote checkout by way of the mount (e.g., Z:\PATHTOCHECKOUT\src\...) or UNC path (e.g., \\builder\src\...). This will copy the installer, DLLs, and PDBs into your debug machine‘s C:\out or C:\build (depending on if you‘re rocking the component=shared_library build or not)
• Run C:\out\Debug\mini_installer.exe with the flags of your choice to install Chrome. This can take some time, especially on a slow machine. Watch the Task Manager and wait until mini_installer.exe exits before trying to launch Chrome (by way of the shortcut(s) created by the installer)
• For extra pleasure, add C:\out\Debug to your _NT_SYMBOL_PATH environment variable

Consider reading the documentation at the top of copy-installer.bat to see how you can run it. It tries to be smart and copy the right things, but you may need to be explicit (e.g., "copy-installer.bat out Debug"). It is safe to re-run the script to copy only modified files (after a rebuild, for example).

Find memory leaks

The Windows heap manager has a really useful debug flag, where it can be asked to capture and store a stack trace with every allocation. The tool to scrape these stack traces out of processes is UMDH, which comes with WinDbg.

 

UMDH is great. It will capture a snapshot of the heap state as many times as you like, and it‘ll do it fairly quickly. You then run it again against either a single snapshot, or a pair of snapshots, at which time it‘ll symbolize the stack traces and aggregate usage up to unique stack traces.

 

Turning on the user stack trace database for chrome.exe with gflags.exe makes it run unbearably slowly; however, turning on the user stack trace database on for the browser alone is just fine.

 

While it‘s possible to turn on the user stack database with the "!gflag" debugging extension, it‘s too late to do this by the time the initial debugger breakpoint hits. The only reasonable way to do this is to 

1. Launch GFlags.exe,
2. Enable the user stack trace database (per image below),
3. Launch Chrome under the debugger.
4. Set a breakpont when chrome.dll loads with "sxe ld chrome.dll".
5. Step up, to allow Chrome.dll to initialize.
6. Disable the stack trace database in GFlags.exe.
7. Continue chrome, optionally detaching the debugger.

 

GFlags.exe settings for user mode stack trace database.

If you then ever suffer a browser memory leak, you can snarf a dump of the process with 

• Application Verifier is a free tool from Microsoft (available as part of the Windows SDK) that can be used to flush out programming errors. Starting with M68 Application Verifier can be enabled for chrome.exe without needing to disable the sandbox. After adding chrome.exe to the list of applications to be stressed you need to expand the list of Basics checks and disable the Leak checks. You may also need to disable Handles and Locks checks depending on your graphics driver and specific Chrome version, but the eventual goal is to have Chrome run with Handles and Locks checks enabled. When bugs are found Chrome will trigger a breakpoint so running all Chrome processes under a debugger is recommended. Chrome will run much more slowly because Application Verifier puts every allocation on a separate page.
• You can check the undocumented ‘Cuzz‘ checkbox in Application Verifier to get the Windows thread scheduler to add some extra randomness in order to help expose race conditions in your code.
• Putting every allocation on a separate page will dramatically affect performance so you may want to only do this for some applications. If you right-click on the Heaps checkbox and select Properties you can edit things like the size range for what allocations go into PageHeap (the page-per-allocation system) and you can set a RandRate percentage to randomly put allocations in PageHeap.

• To put a breakpoint on CreateFile(), add this break point:

{,,kernel32.dll}_CreateFileW@28

• {,,kernel32.dll}specifies the DLL (context operator).
• _ prefix means extern "C".
• @28 postfix means _stdcall with the stack pop at the end of the function. i.e. the number of arguments in BYTES.
• You can use DebugView from SysInternals or sawbuck to view LOG() messages that normally goes to stderr on POSIX.