Remove nbdev for core developments and use pytest for tests

[JQGoh]

Rationale

• Remove the dependency of nbdev as part of the development workflow
• Introduce test execution using pytest and provide the test coverage of code base
• Note that we do not remove nbdev completely at this stage as it could still be useful to support the generation of notebook-based documentations.
• Note that a number of Python packages need to be added because
  jupyter nbconvert --to notebook --execute --inplace "$nb" does indeed execute all the notebook cells, this behavior differs from the nbdev_test execution whereby it is suspected that it only executes selected cells (marked by nbdev directive)
  • See this job, it failed because lightgbm package was not installed during the notebook run.

Suggestions

• run-notebooks-test can be flaky and currently there is no convenient option like pytest-rerunfailures to rerun the test when encountered failure like ConnectError: [Errno 104] Connection reset by peer
• Nixtla client has retry_strategy and we could consider adding the following as part of the allowable retry scenarios (if rate limiting policy allows delayed retry)
  • urllib.error.HTTPError
  • httpx.ConnectError

Changes

• Migrate tests from notebook cells to pytest-styled test cases. The migrated test cases are those mainly registered in.

  • nbs/src/date_features.ipynb
  • nbs/src/nixtla_client.ipynb
  • nbs/src/utils.ipynb

• Migrated test cases are saved into nixtla_tests folder.

  • Move commonly used test fixtures into nixtla_tests/conftest.py file
  • Revise the ci workflow and contributing guides to omit the dependency on nbdev.
  • As nbdev_test also includes execution for notebook in nbs/docs folder such as nbs/docs/tutorials/15_missing_values.ipynb file, we also need to execute and run this. For notebooks saved in the nbs/docs, we shall replace with another test job=run-notebooks-test
  • New set of ci jobs
    • run-notebooks-test: It executes all the .ipynb files found within nbs/docs folder on ubuntu platform.
    • run-cross-platform-tests: It executes all the pytest cases, both local and distributed runs, on both mac and ubuntu platforms.
      • Caveat: For windows platform run, we currently disable the test runs for Spark because of error as shown below, which requires a revised system setup (per-my understanding, this requires Nixtla team to setup internally).

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/spark/launcher/Main has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 52.0      

Upgraded Packages

• ray upgraded to 2.20.0
  • Required because windows platform could not resolve dependencies during the installation, see job

Added Packages

• pytest
• pytest-cov
• pytest-rerunfailures [This is to support pytest re-running when failure like httpx.ConnectError, urllib.error.HTTPError happen, avoid rerun whole set of failed job when it failed occasionally]
  • httpx.ConnectError: [Errno 104] Connection reset by peer, see failed job
  • urllib.error.HTTPError: HTTP Error 429: Too Many Requests, see failed job
• shap [required as part of the execution of notebook doc]
• pandas_market_calendars [required as part of execution of notebook doc]
• mlforecast [required as part of the execution of notebook doc]
• lightgbm [required as part of the execution of notebook doc]

[JQGoh]

@goodwanghan
This PR is ready for review. I have included my suggestion and things to take note in the PR. Specifically,

• windows-platform tests disable running on spark related tests (JNI error)
• run-notebooks-test could be flaky due to ConnectError: [Errno 104] Connection reset by peer

cc: @mergenthaler

[goodwanghan]

@goodwanghan This PR is ready for review. I have included my suggestion and things to take note in the PR. Specifically,

• windows-platform tests disable running on spark related tests (JNI error)
• run-notebooks-test could be flaky due to ConnectError: [Errno 104] Connection reset by peer

cc: @mergenthaler

Amazing work! @JQGoh I will review it tomorrow.

[goodwanghan]

@JQGoh I want to say, this is a masterpiece, and it is worth every engineer in Nixtla to learn!

So I can see there are a few other things that we must change in order to release. For example doc gen is going to break with this change, and it may not be trivial to make that change as an external collaborator.

I think you have accomplished this task, I or some other Nixtla member can continue working on this PR to finish the docs piece.

Great job! I am deeply impressed.

[JQGoh]

@goodwanghan Thank you.

• Regarding doc gen, I believe specifically you are referring to scripts in docs-scripts/ such as update-quarto.py. I made a dummy change to documentation and found that the build-docs job was still successful. Since I did not remove nbdev dependency, I suppose the doc gen workflow will still work (assuming the success of build-docs job implies that doc gen is working?)
• Before merging this PR, I think the stability still require improvement. What do you think about the suggestion of adding httpx.ConnectError, urllib.error.HTTPError into the retry strategy of Nixtla client?
• It has been a few days since the last successful job run. Today I realize that windows-platform job consistently failed with Windows fatal exception: access violation and I suspect the failure happened when we execute ray package related run. This needs further investigation.

  **File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\ray\_private\worker.py", line 898 in print_logs**
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 917 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap

• This failure of job also worths the attention as there was an internal error

E           nixtla.nixtla_client.ApiError: status_code: 500, body: {'detail': 'Internal server error, please contact us at ops@nixtla.io', 'request_id': 'F823W9RYWQ'}