Finding the solution for this problem took a lot of setting breakpoints and tracing through the code to find the right bottlenecks. The key area, as I had discovered in my surfing last time, was the launch configuration, which was initialized with default VM arguments when it was created.

Since there are several similar operations users can go through that cause Eclipse to create new launch configurations, I had to check each one to see how the launch configurations were defaulted. New launch configurations can be created manually, when the user selects the "Run.." or "Debug.." menu items under "Run as" or "Debug as" in the right-click menu when launching. New launch configurations are also automatically created by Eclipse when the user selects "JUnit test" or "JUnit Plug-in test" from the "Run as" or "Debug as" menu if there is no previously created launch configuration for the selected item.

My first inclination was to add a constructor to the JUnitLaunchConfiguration class to set the defaults there, but the class didn't have an explicitly declared default constructor, and I didn't want to complicate the semantics of new object creation with searching properties and dealing with exceptions when something goes wrong. It seemed like I should look around for a better place.

Ultimately, the six use cases outlined above resolve to 3 methods in 3 different classes.
JUnitWorkbenchShortcut.createConfiguration() handles the cases where the user invokes the test directly and there is no default launch configuration.
JUnitTabGroup.setDefaults() handles manually created launch configurations for base JUnit tests.
AbstractPDELaunchConfigurationTabGroup.setDefaults() handles manually created launch configurations for JUnit plug-in tests.

Finding these spots was where I had to do my breakpoint setting and tracing. Launching a JUnit test involves sending an event to an event handler, so you have two potential threads to trace through, and once you find the right one, the setDefaults() protocols have fairly deep call stacks. At first I thought I could set the configuration in the common base of these two classes (AbstractLaunchConfigurationTabGroup) but unfortunately, they set defaults differently, and the PDE version monkeys with the VM args that somehow may be there already. So I had to create separate but similar implementations for the two classes.
The JUnitWorkbenchShortcut.createConfiguration() case was clearer to nail down, and didn't take so much headscratching. It turns out that case was the same as far as what I had to do to default the VM argument as the JUnitTabGroup.setDefaults().

So, the issue was I had 3 similar cases that shared some code - 2 of the 3 were exactly the same, and I needed to invoke them from 3 different spots in Eclipse code. So I made up a utility class that contained all the code to set and get the preference, and to initialize the launch config value, and exposed the functionality as static methods. Then I added invocations to these methods as one-line changes in the 3 different spots. I tested them with constants instead of real preferences and the launch configurations were as I expected and the JUnit tests ignored or failed assertions as I expected.

The next thing to do was add a checkbox to the Window Preferences JUnit page. I had never done this kind of thing before, so I looked at a lot of examples. I had to make changes to the JUnitPreferencePage class to create the checkbox and add a new composite so this checkbox could share the window with the stack trace filter pattern editing stuff that was already there. Then adding the preference turned out to be quite easy. Manual tests were working, so I was pretty close.

All in all, the surfing and tracing took the most time. I did the back-end parts with the new AssertionVMArg class and inserting its changes in two evenings home from work, and I did the preferences page in a weekend afternoon.

Next - writing JUnit tests.