Will MTP work with multiple test projects in a single solution?

[PureKrome]

👋🏻 G'Day!

<Narrator's voice>Previously, in dotnet Test</narrator's voice> we can use the command dotnet test and it would find the test projects and run the tests against them.

With MTP it looks like we now need to dotnet run the test project. Ok, fine .. but this means we need to do this explicitly for each project in the solution. Sure, it's a few more lines in a CI file .. but it's a bit more annoying when working in the CLI and we need to run each command separately to test all the tests in a solution.

Is there a way to have dotnet run be smarter and find and run all tests, even if we give it a filter?
Yes, i could create a PS script or shell script, but i'm wondering if this is a PITA and regression considering the easier barrier to entry via dotnet test.

Will dotnet test be the future way which is a wrapper to calling 'dotnet run' for each test if is told/finds/etc?

[Youssef1313]

With MTP it looks like we now need to dotnet run the test project

I'm curious from where you get this impression? If there is a misleading or unclear documentation somewhere we would definitely want to improve it.

You can use dotnet run, but it's not the only way to run the tests.

dotnet test works, as well as IDEs Test Explorer (at least Visual Studio, VS Code DevKit, and Rider).

The story for dotnet test is quite long, which we documented in Testing with dotnet test article. We would appreciate a feedback on the documentation if it's not clear or if you have further questions that are unanswered by that article.

[PureKrome]

👋🏻 Hi @Youssef1313 - thank you very much for you prompt reply, really appreciated!

I've read a few docs about this and it's all very confusing.

REF 1: https://learn.microsoft.com/en-us/dotnet/core/testing/microsoft-testing-platform-intro?tabs=dotnetcli#use-dotnet-test

So this doc starts out by saying:

• dotnet run
• dotnet exec
• dotnet test

If i'm trying to explain this to team members or newbies, I'm sure I would get asked: "ok.. so we have 3 options - which one, and why?". I know that's exactly what I'm thinking.

As to why I thought dotnet test is going away is because of the order of operations, listed above. I incorrectly assumed that dotnet test is here right now is because of this (emphasis, mine):

Microsoft.Testing.Platform offers a compatibility layer with vstest.console.exe and dotnet test ensuring you can run your tests as before while enabling new execution scenario.

That's how I read that paragraph. It felt like (to me) that it was/is a temp stop-gap solution until we all move over to some "new execution scenario". Execution scenario (to me) is 'dotnet run'

Also, trying to make sure i'm using MTP instead of VSTest is a struggle. I'm on .NET 9. So I've got 2x properties in my csproj files:

<UseMicrosoftTestingPlatformRunner>true</UseMicrosoftTestingPlatformRunner>
<TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>

and then i'm not 💯 sure what dotnet test is using behind the scenes. I still feel like dotnet test is setup to use vstest by default with a temp hack added into to switch over to MTP. But the real way should be dotnet run.

That's how i'm feeling, from reading docs and stuff.

How could the docs be updated? Glad you asked! I feel like they are ok right now but are just a dump of info: "Here's all the things. K-gl-have-fun-etc"

versus

Ok - here's the recommended way .. the Happy Path™️ . (aka. dotnet test)
and if you're still reading ...
here is all the advanced and hardcore ways to do stuff (aka. dotnet run, dotnet exec) etc. with reasons why i would be using this over that.

If dotnet test is just a wrapper for dotnet run with some other fancy pants things like test project discovery, etc... great! But .. the average jane/joe prolly doesn't need to know that in the Happy Path™️ .

Anyways, I hope this helps.

Oh. and other confusing things:

• settings files? can we say 'parallel all the things'?
• code coverage? (i saw coverlet is undersupported so we're stuck using ms coverage, I believe). Also, settings for this (exclude test projects, etc).
• merging of coverage files

I think I stumbled on the Coverage Samples. This is seriously, seriously awesome btw! 👏🏻

[Youssef1313]

Thanks for the feedback. We truly appreciate it. I opened dotnet/docs#45643 to track improving the documentation.

[PureKrome]

Pleasure! We also appreciate all what you guys are doing to making the environment and tooling, top notch, efficient and productive 💪🏻 🚀

[Evangelink]

@PureKrome also adding my thanks for the nice feedback! It's not always easy to write docs especially when we start to have deep knowledge on testing!

The short answer is always use dotnet test :).

Now because MTP is producing a simple exe app, you can also now use dotnet run or dotnet or dotnet exec as you would with any normal app.

When working on a single test project there should be no difference between dotnet test and dotnet run (while dotnet and dotnet exec are more equivalent to calling the exe directly - so no restore + build).

When working with multiple test projects, using dotnet test is the only way (the other commands don't work on a set of projects/solution) and with the new experience, dotnet test will flow down the parameters to each project and also provide some extra capabilities like pre/post processing.