I'd like to make dart test --coverage behave more like test_with_coverage and flutter test --coverage. Context is dart-lang/sdk#60958

test_with_coverage behavior:

• starts dart test with --pause-isolates-on-exit (and optionally --branch-coverage)
• calls collect
  • resume and waitPaused enabled
  • scopedOutput set to the package's name (by default)
  • Optionally with functionCoverage and branchCoverage enabled
• calls formatLcov to create an lcov file
  • also checks for ignored lines here
• outputs to <package-dir>/coverage/ by default. The lcov.info file is placed in this directory

flutter test --coverage behavior:

• broadly the same, except that resume and waitPaused are disabled in the collect call, and isolate lifecycles are managed by the test runner. Does multiple collect calls in a single run, so uses coverableLineCache to speed up collection
• No function coverage option, due to performance issues
• output is also in <package-dir>/coverage/lcov.info by default, but the flag (--coverage-path) specifies the output file, not directory

dart test --coverage behavior:

• flow is most similar to flutter test --coverage
  • resume and waitPaused are disabled in the collect call, so I assume that the test runner is managing the isolate lifecycles manually
  • passing isolateIds, which could miss coverage if the test is spawning isolates that aren't in the same group (eg using spawnUri)
    • Is this used to avoid double counting, because tests are run in separate isolate groups within the same process?
  • not setting scopedOutput, leaving filtering to a later step
  • not using coverableLineCache which could be slightly slower if it's doing multiple collect calls
• No function coverage or branch coverage options
• outputs location is controlled by the string passed to the --coverage flag, which is a directory
  • dart test --coverage by itself gives an error. Must pass a directory: dart test --coverage=<dir>
  • The final output is a bunch of json files placed in nested directories inside this directory, matching the directory structure of the tests
  • Documentation tells the user to convert this json to an lcov file using a separate coverage:format_coverage call

Questions for package:test maintainers

• Are tests started in subprocesses? Same process, different isolate groups? Different isolates in the same isolate group?
• What's the coverage collection flow? Are the isolate lifecycles (start/pause at exit/resume) being managed manually by package:test? What's the typical flow?
• Is isolateIds being used to avoid double counting coverage? Should we worry about missing coverage in other isolate groups that the user creates?
• Can there be multiple calls to collect in a single run of dart test?
• I want to add a --branch-coverage flag. This flag needs to be passed to the VM running the tests, and to the collect call. So I need to pass it to the VM, but also to package:test (to be forwarded to collect). The flag groups are separated like dart <vm flags> test <package:test flags>, so naively this would look like dart --branch-coverage test --branch-coverage .... Is there a way of doing this without duplicating the flags?

Proposal

I wanted to make it so that dart test --coverage by itself was a valid command, but it seems that's not really possible using package:args. It says Missing argument for "--coverage" instead of setting the value of the option to "". So:

• I'll add a --coverage-lcov option that takes a path to an lcov file to output (alternative names welcome).
  • This option implies --coverage, with a value that defaults to the directory containing the lcov file
  • Though I'll probably not output the json files if --coverage-lcov is set, so the value of --coverage probably doesn't matter
• I'll add a --branch-coverage flag
• I'll set scopedOutput to the package name
• If there can be multiple calls to collect, then I'll add a coverableLineCache
• Not totally sure what to do about isolateIds potentially causing missed coverage, so I'll just leave it for now.