I unsuccessfully try to expand the scope again...

When I was running my tests, I realized something about my new solution that I didn't like. It is this: a user will go to the preferences window and check a checkbox that says "enable assertions for JUnit tests." Then the user will rerun a JUnit test that already has a launch configuration setup, and assertions will not be enabled. That's because the code only enables assertions on new launch configurations as they are created, not on existing launch configurations.

So, I started looking into what it would take to enable or disable unit tests for all existing launch configurations when the checkbox is OK'ed. This turns out to be a fair pain-in-the-butt. You have to read all the launch configurations, filter the ones that are JUnit or JUnit plug-in tests (OK, that part's not so hard) but then you have to dig through all their JVM arguments and match the ones that enable assertions to see if they are already enabled or not. Since I don't know very well the constraints and variations among VMs on entering these args, I'm not confident I can detect them and frame their beginnings and endings in all cases. So I start implementing it, hoping I can work through the issue. But alsoI sense I am expanding the scope of my solution in a possibly controversial way, so I send off an e-mail to Markus explaining my dilemma.

- - - - - - - - - -

Markus - I have a preliminary solution done that adds a checkbox to the
JUnitPreferencesPage, uses that to set a preference, and then uses the
preference to default new JUnitLaunchConfiguration objects to have their VM
args defaulted to blank or to the enable assertions argument. It all works, and
I have an automated test for the page and the preference, but not yet the
launch configurations.

I ran into something, however, that seems like a bug in this solution. When you
set the preference, it only signals the system to add the VM arg to
subsequently created launch configurations. All the old launch configurations
are still sitting without the VM arg. So it seems to the user they just enabled
assertions, but when they rerun a unit test, assertions still aren't enabled
(and vice-versa with disabling).

So, I figured I should go through all the JUnitLaunchConfiguration objects, and
re-sync their arguments to however the user set this preference. I'm working my
way through the code of this, and almost have something to try.

Thought I should bring this up to you for comment before I spring the whole
thing on you at once. Will have a complete patch soon.
- John

- - - - - - - - - -

As I might have guessed Markus wants none of this idea, and writes back with a simplifying change instead.

- - - - - - - - - -

We should not fiddle around with the user's exisiting launch configurations.
The user may have set up some launch configurations with -ea and some without.
If the checkbox would change all existing configs, that would destroy the
user's configuration and would not allow him to enable -ea for new tests
without destroying his work.

You should rather call the checkbox 'Enable assertions for new JUnit launch
configurations' to avoid any confusion.

- - - - - - - - - -

As usual, I cave in immediately and get on with the fix. I changed the text, and commented out my half-implemented configuration-surfing-parsing-and-changing code. (You never know when you may need it later...)

- - - - - - - - - -

OK - I can see that - it's definitely the safer way to go.

I was almost done with the code to do it too - the only part I was having a
hard time with was how to reliably take an argument out of the VM args - I
imagine that's even worse from your perspective.

So, the only part I have left to do to finish is to figure out how to call the
right things programatically to create new launch configurations through
JUnitLaunchShortcut, JUnitTabGroup, and AbstractPDELaunchConfigurationTabGroup,
and then I can finish the automated tests.

- - - - - - - - - -

Next, the patch and the wait....

Making JUnit tests

The first rule if you're making or running JUnit tests for Eclipse code is you have to download the source for the org.eclipse.test.performance package. This magically enables JUnit plug-in tests to work, and without it, you get a bunch of inexplicable errors that you will waste your time trying to understand and fix.

Since this was the first time I ever made up a new JUnit plug-in test, I looked at other examples in the org.eclipse.jdt.ui.tests package. I had added tests to the NewTypeWizardTest before, I poked around in there for ideas, and it helped to review familiar territory.

The main issues with testing Eclipse code are having the right interfaces to test what you need, and getting the classes you want to test into the right state. Since these classes normally execute inside complex Eclipse runtime contexts, you have to do some experiments with setting up scaffolding and calling initialization methods. When you need test projects to work with, you can create them inside the setUp() method of the JUnit test. Since Markus told me he didn't want me actually running a JUnit test and checking to see if it actually asserted, but only checking if the launch configuration was initialized correctly, I didn't have to bother with setting up a test project to run. I could test each unit on a lower level.

The units of functionality I had to test were:
That the preference for enabling assertions was initially false
That enabling and disabling the checkbox on the preferences page properly set and cleared the enable assertions preference
That the preference caused new launch configurations to be initialized properly (JUnitTabGroup) with the assertion VM arg or not
The automated tests for default configuration creation (JUnitWorkbenchShortcut) and plug-in test initialization (AbstractPDELaunchConfigurationTabGroup) I left out because they looked like more trouble, so I hope I don't burn in hell too long for that.

The first test was a no-brainer, just put an assertion on the utility function for reading the preference to make sure it's off.

The second test for the preferences page had two issues. First, since you don't test by means of a UI robot, but test by means of calling utility methods on the page, I had to create a utility method to set and clear the check box. Second, you have to create a page and initialize it to test it. In the case of a JUnitPreferencesPage, I had to call the createControl() method to initialize the SWT widgets so I could set and clear the checkbox. The createControl() method needs an SWT UI context as the parent of the new page. Luckily, the JUnit plug-in scaffolding provides a DialogCheck class that has a getShell() method you can use as a parent context. After creating a new page and initializing it this way, all I had to do was call my checkbox enable/disable method, call the page's performOk() method, and assert that the preference had been set correctly.

The third test for the launch configuration initialization took more experimenting. First, you can’t change a launch configuration, so you have to use a launch configuration working copy. And you can’t instantiate one; you have to go through a 4-line song-and-dance that I put into a utility method. Then, you have to create a tab group and initialize it using createTabs(). Then you can call the setDefaults() method on the tab group, and then test to see if the launch configuration has the assertion enabling argument on or not as expected.

Finding the solution for this problem took a lot of setting breakpoints and tracing through the code to find the right bottlenecks. The key area, as I had discovered in my surfing last time, was the launch configuration, which was initialized with default VM arguments when it was created.

Since there are several similar operations users can go through that cause Eclipse to create new launch configurations, I had to check each one to see how the launch configurations were defaulted. New launch configurations can be created manually, when the user selects the "Run.." or "Debug.." menu items under "Run as" or "Debug as" in the right-click menu when launching. New launch configurations are also automatically created by Eclipse when the user selects "JUnit test" or "JUnit Plug-in test" from the "Run as" or "Debug as" menu if there is no previously created launch configuration for the selected item.

My first inclination was to add a constructor to the JUnitLaunchConfiguration class to set the defaults there, but the class didn't have an explicitly declared default constructor, and I didn't want to complicate the semantics of new object creation with searching properties and dealing with exceptions when something goes wrong. It seemed like I should look around for a better place.

Ultimately, the six use cases outlined above resolve to 3 methods in 3 different classes.
JUnitWorkbenchShortcut.createConfiguration() handles the cases where the user invokes the test directly and there is no default launch configuration.
JUnitTabGroup.setDefaults() handles manually created launch configurations for base JUnit tests.
AbstractPDELaunchConfigurationTabGroup.setDefaults() handles manually created launch configurations for JUnit plug-in tests.

Finding these spots was where I had to do my breakpoint setting and tracing. Launching a JUnit test involves sending an event to an event handler, so you have two potential threads to trace through, and once you find the right one, the setDefaults() protocols have fairly deep call stacks. At first I thought I could set the configuration in the common base of these two classes (AbstractLaunchConfigurationTabGroup) but unfortunately, they set defaults differently, and the PDE version monkeys with the VM args that somehow may be there already. So I had to create separate but similar implementations for the two classes.
The JUnitWorkbenchShortcut.createConfiguration() case was clearer to nail down, and didn't take so much headscratching. It turns out that case was the same as far as what I had to do to default the VM argument as the JUnitTabGroup.setDefaults().

So, the issue was I had 3 similar cases that shared some code - 2 of the 3 were exactly the same, and I needed to invoke them from 3 different spots in Eclipse code. So I made up a utility class that contained all the code to set and get the preference, and to initialize the launch config value, and exposed the functionality as static methods. Then I added invocations to these methods as one-line changes in the 3 different spots. I tested them with constants instead of real preferences and the launch configurations were as I expected and the JUnit tests ignored or failed assertions as I expected.

The next thing to do was add a checkbox to the Window Preferences JUnit page. I had never done this kind of thing before, so I looked at a lot of examples. I had to make changes to the JUnitPreferencePage class to create the checkbox and add a new composite so this checkbox could share the window with the stack trace filter pattern editing stuff that was already there. Then adding the preference turned out to be quite easy. Manual tests were working, so I was pretty close.

All in all, the surfing and tracing took the most time. I did the back-end parts with the new AssertionVMArg class and inserting its changes in two evenings home from work, and I did the preferences page in a weekend afternoon.

Next - writing JUnit tests.