Thoughts on MTP providing source generation of tests

[thompson-tomo]

Has there been any thought/discussion surrounding if MTP should provide a source generator for test cases & the corresponding attributes?

What I am thinking is a user decorates their test class with attributes describing the arguments or parameters and based on that it generates the test class which is run by the corresponding test framework. Obviously that is a simplistic view but it can be extended.

I like this idea as it is equivalent to what has occurred with logging where a user can seamlessly switch between framework NLOG, Serilog as long as it leverage MEL.

This approach would likely be a benefit to BenchmarkDotnet and others

[Youssef1313]

Thanks for opening this discussion @thompson-tomo

It's up to the test frameworks to use source generators (there is nothing to do on MTP side). For example, TUnit already uses source gen approach. Also, MSTest.Engine is using source gen, though it's not in complete feature parity with the reflection-based MSTest yet.

One problem with using source gen is that it supports C#/VB only. So, frameworks that want to support F# will still have to implement a reflection-based fallback for F#.

[thompson-tomo]

At a quick sketch.

A test is written and decorated with attributes to form a collection of test cases based on the arguments & parameters specified. So far nothing changes except maybe the attribute names but fundamentally same experience.

When the source generator runs for each class which contains tests it generates an entry point method for that class which can be used to discover the list of test cases including all the conditions/parameter. TBD if This info is passed to the test framework as a batch or in a single approach with my thought being singles.

Extensions can be written to handle repeated tests etc.

[Evangelink]

There is actually a lot of custom logic within each test framework for the order of tests, how to schedule, how to run on various threads, what to prepare and cleanup after each given test so there is not really any common enough context to be able to provide a common source gen for all.

[thompson-tomo]

Agree that there is alot of functionality in the test frameworks hence I would constrain the common generator to just extrapolating the tests that way the frameworks can do what they do best. At the same time we are opening up interoperability of switching frameworks etc.

[nohwnd]

It is really not that simple to do. In the simplest cases this could work, we scan all classes in the current assembly and pick up all classes that match a given predicate, and all methods from the that match given predicate.

But once you start going more detailed, each framework works slightly differently, one framework will want to get internal methods, other won't, and there is bunch of other differences, like how test methods are inherited, how attributes are derived, how assembly / class setups and teardowns are inherited etc.

All of this should work in the most efficient way possible, which is a big undertaking for us as platform, and is better done by each framework following their own rules.

[Evangelink]

The source generation mechanism is also not designed to be extensible so basically we would provide something that works only for simple scenario but is not extensible so needs to be redone completly when outside of the simple scenario. Note that even the simple scenario is actually not so simple (e.g. handling of inheritance of attribute etc)