Skip to content

Conversation

funnyboy-roks
Copy link
Member

@funnyboy-roks funnyboy-roks commented Aug 19, 2025

This is basically a full rewrite of the crate, now that we've used it and noticed issues.

There are a few important points that I'm trying to accomplish with this rewrite that were missing from the original:

  • A test context can be created once at the start of a long-running application. This context contains all of the generic information that applies to any test suite (i.e., tests, required files, compile/run commands)
    • From this context, individual test runners can be created for actually running the tests. These test runners should be able to specify what subset of tests will be run
  • Running tests should be fully asynchronous. This was the case before, but I want to take it to another level by allowing test status to be updated in any order.
    • For example, if test1 takes 50ms and test2 takes 25ms, then we should get the result of test2 before test1.
    • This should work in the same way that JoinSet works, with a join_next-equivalent function that will return the next finished test.

This will also close #2, #6, and #7.

Major TODO items:

  • Create type-state builder for TestContext
  • Actually run tests
  • Add useful logging
  • Add serde derives where logical
    • Decided against using serde for this as it doesn't really work well for a lot of the types. Mapping from other types is simple enough, so that's left to the consumer.
  • Copy directories
  • Add support for various output validation
    • Regex
    • Multiple Strings (? - Not sure if we need this on top of regex)
      Opting against this for now, we may want to re-evaluate at a later point in time.
  • Write Tests
    • At least 95% coverage since we have codecov now
  • Cleanup API (i.e., make every function useful, reorganise modules)
  • Add documentation

- Add additional data for the test cases (for things like visibility)
- `Unset<X>` -> `Missing<X>` for slightly nicer error messages
- Add `build_and_reset` and `build_and_consume` functions
- Start implementation of test runner
    - Create necessary files
    - Compile code
    - Setup framework for running tests
- Replace `anyhow` with `thiserror`
- Add tracing for logs
@funnyboy-roks funnyboy-roks self-assigned this Aug 19, 2025
@funnyboy-roks funnyboy-roks added enhancement New feature or request dependencies Pull requests that update a dependency file labels Aug 19, 2025
@funnyboy-roks funnyboy-roks force-pushed the rewrite branch 5 times, most recently from f81c207 to 21307a1 Compare August 21, 2025 08:56
Copy link

codecov bot commented Aug 21, 2025

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment

Thanks for integrating Codecov - We've got you covered ☂️

@jackjohn7
Copy link
Member

Love this so far!

@funnyboy-roks funnyboy-roks marked this pull request as ready for review August 24, 2025 08:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow expected output to be a Regex Add logging with tracing Allow multiple correct solutions

2 participants