Data Consumers and Discovered Tests

[thomhurst]

Hope you don't mind this discussion being reopened but I feel there was never really a solution or any proper ideas to solve this particular challenge.

As we mentioned in another PR, it would be great for library authors to be able to create things like their own filters, that you can just install, and they just work. Similar to how you install the code coverage extension and it just works. This of course is dependent on test frameworks sending the correct test node messages, but let's assume they are.

We've established that in the context of filters, there's some scenarios where a filter needs to know about all other tests within the test suite in order to know whether a test should be filtered or not.

Two examples of the top of my head:

• An "explicit" filter - running tests only if they have been run explicitly and not alongside other tests
• A batch filter - a way of splitting up a test suite and running them across different CI agents to speed up tests.

The problem with this is there's no generic way for a filter to do this at the moment. (Well filters aren't even extensible at the moment but let's pretend they were).

Filters aren't currently even allowed to be a data consumer unless they were also an extension afaik?

But then for it to know the context of other tests, it would have to collect the discovered test nodes.

This has the following flaw in my opinion:
If you're using --list-tests, you only send discovered nodes after you've done any filtering, to ensure you only show the correct tests in the CLI that would run for that filter.

That means as the nodes are sent AFTER filtering, if the filter was a data consumer, it wouldn't receive anything until after it's already done it's job. And essentially would only receive what it's already decided should be let through.

You guys have architected a great platform and you could probably solve this better than me.

My thoughts would be having a new node type state: FilteredTestState

So basically you perform your discovery and ALL tests, regardless of filtering (because you haven't done it yet) are sent to the message bus with the discovered state.

This means if a filter was a data consumer, it could receive these, and aggregate what it receives and use it to perform its logic.

Then, it can perform it's necessary filtering.

After which, the filtered tests are pushed with the updated state of filtered.

The --list-tests command would then show nodes with the filtered state. Maybe for backwards compatibility, if only discovered states and no filtered states are sent, just show the discovered ones like currently.

I hope you see the benefit that this could provide to some framework authors.

Keen to hear ideas from you guys!

[nohwnd]

I think extensibility is a good idea.

To me filtering is part of discovery and should be very fast and streamable, and not necessarilly connected to what tests will run. The filter should be atomic and be able to act on each test separately, or work on the accumulated state, but not require all the tests to be discovered before it can filter the test in or filter it out.

As an example, a test name test would be able to filter on the test. But a "explicit" filter needs all tests be discovered first to tell if any test can or cannot run.

So if we think about also having to stream the discovery results to IDE, what you describe is probably a separate step that is better located in in Run, and that we could call "run planner". (But at the same time if discovery should discover, and planner should then offload the tests to different runner across CI, then that does not fit, and imho that is not even a job the MTP should do, but rather should be done via some orchestrator - but maybe my view is too guided by how we did those things in other places like vstest).

One slight problem is that extensions now work mostly on single items rather than collection of things, I think it would be useful to have extensions that would get a collection of everything and could act on it, in a fashion that is similar to asp.net middleware, where you can process data, and send it to the next extension or say that data should not be processed anymore and you should move to next step.

This would allow us to encode the explicit tests quite easily, the "pre-run bulk extension", would receive all discovered tests (post filtering), and would select just the explicit tests, and forward that into the pipeline for the runner to run.

[thomhurst]

Maybe this should be something to flesh out before MTP V2? As I imagine it'd be a breaking change.

But working on a collection of things I think is sensible. Tests aren't always isolated and sometimes require the context of other tests.

To me filtering is part of discovery and should be very fast and streamable, and not necessarilly connected to what tests will run.

It's all subjective I guess but I think the opposite. You have a test suite, you discover all tests in that test suite.

Imo filtering is always just to show (or actually execute) which tests would be run.

Even with list tests, it shows which tests would be executed. Kinda like a dry run.

I consider filtered tests to relate to the execution, and not discovery. Tbh I find discovering a subset of tests a 0.1% scenario. Most people are only using filters in the context of actually executing their test suite.