-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Add asynchronous load method #10327
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Add asynchronous load method #10327
Changes from 1 commit
Commits
Show all changes
144 commits
Select commit
Hold shift + click to select a range
01e7518
new blank whatsnew
TomNicholas 83e553b
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas e44326d
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas 4e4eeb0
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas d858059
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas d377780
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas 3132f6a
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas 900eef5
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas 4c4462f
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas 5b9b749
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas fadb953
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas 57d9d23
Merge branch 'main' of https://github.com/TomNicholas/xarray
TomNicholas 11170fc
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas 0b8fa41
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas f769f85
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas 4eef318
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas 29242a4
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas e6b3b3b
test async load using special zarr LatencyStore
TomNicholas 3ceab60
don't use dask
TomNicholas 071c35a
async all the way down
TomNicholas 29374f9
remove assert False
TomNicholas ab12bb8
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 62aa39d
add pytest-asyncio to CI envs
TomNicholas dfe8bf7
Merge branch 'async.load' of https://github.com/TomNicholas/xarray in…
TomNicholas a906dec
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 629ab31
assert results are identical
TomNicholas 7e9ae0f
implement async load for dataarray and dataset
TomNicholas d288351
factor out common logic
TomNicholas e0731a0
consolidate tests via a parametrized fixture
TomNicholas 9b41e78
async_load -> load_async
TomNicholas 67ba26a
make BackendArray an ABC
TomNicholas 9344e2e
explain how to add async support for any backend in the docs
TomNicholas f8f8563
add new methods to api docs
TomNicholas 30ce9be
whatsnew
TomNicholas 5d15bbd
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas 1f02de1
Merge branch 'main' into async.load
TomNicholas 2342b50
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] b6d4a82
Fix ci/minimum_versions.py
TomNicholas 2079d7e
fix formatting
TomNicholas 48e4534
concurrently load different variables in ds.load_async using asyncio.…
TomNicholas cca7589
test concurrent loading of multiple variables in one dataset
TomNicholas dfe9b87
fix non-awaited load_async
TomNicholas 84099f3
rearrange test order
TomNicholas ab000c8
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] a8b7b46
add test for orthogonal indexing
TomNicholas 82c7654
explicitly forbid orthogonal indexing
TomNicholas 5eacdb0
support async orthogonal indexing via https://github.com/zarr-develop…
TomNicholas 9f33c09
Merge branch 'async.load' of https://github.com/TomNicholas/xarray in…
TomNicholas 093bf50
add test for vectorized indexing (even if it doesn't work)
TomNicholas 4073a24
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 842a06c
add test for basic indexing
TomNicholas e19ab55
correct test to actually use vectorized indexing
TomNicholas b9e8e06
refactor to parametrize indexing test
TomNicholas 8bc7bea
implement async vectorized indexing
TomNicholas 6c47e3f
revert breaking change to BackendArray
TomNicholas a86f646
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 884ce13
remove indirection in _ensure_cached method
TomNicholas a43af86
IndexingAdapters don't need async get
dcherian 17d7a0e
Add tests
dcherian d824a2d
Add decoding test
dcherian 6a13611
Add IndexingAdapter mixin
dcherian d79ed54
[cherry] Making decoding arrays lazy too
dcherian 1da3359
parametrized over isel and sel
TomNicholas dded9e0
mock zarr.AsyncArray.getitem in test
TomNicholas 4c347ad
tidy up the mocking
TomNicholas b4ed8ee
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas 0b55247
Merge branch 'main' into async.load
TomNicholas 4018e28
ensure the correct zarr class's method is patched for each test
TomNicholas 6da81ce
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 7972164
add degenerate test case of no indexing
TomNicholas 618424a
factor out the Latency part of LatencyStore
TomNicholas cd97481
use mocks in multiple objects test
TomNicholas 75abdec
use mocks in multiple variables test
TomNicholas 7409372
trim latencystore down to just what's needed to dodge https://github.…
TomNicholas 1f79034
parametrizing indexing test over xarray classes
TomNicholas 9881e8d
ensure we actually test vectorized indexing for Variable
TomNicholas 1197798
use create_test_data
TomNicholas 642fd48
add @pytest.mark.asyncio
dcherian a5a44f5
Merge branch 'main' into async.load
TomNicholas f22b56b
remove outdated readonly_store
TomNicholas ebfede5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] e882914
Merge branch 'main' into async.load
TomNicholas 87c7fcb
enable tests to run when recent version of zarr-python is not available
TomNicholas 67c77cc
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 0d4bb0f
separate tests to only run on correct versions of zarr
TomNicholas ed2c808
clear error message if async oindexing not available
TomNicholas df32020
parametrize over zarr_format
TomNicholas aba2917
Merge branch 'async.load' of https://github.com/TomNicholas/xarray in…
TomNicholas d3e6a64
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 4570aed
add pytest-asyncio to other test CI env
TomNicholas ddde08e
Merge branch 'main' into async.load
TomNicholas 0ec670e
fix some mypy errors
TomNicholas a3a3b62
use method directly when possible
TomNicholas b747f1c
Merge branch 'main' into async.load
TomNicholas a28a6a9
remove repeated API docs from bad merge
TomNicholas 577cc72
fix bad merge in release note
TomNicholas a68579f
fix other bad merge in whatsnew
TomNicholas acc5c94
remove prints
TomNicholas 7776d41
remove last print statement
TomNicholas 0b1ebb5
test async basic indexing raises informative error before zarr-python…
TomNicholas df09780
test correct error message is raised for each indexing case
TomNicholas 84f8e30
ensure each test runs on the earliest version of xaarr it can
TomNicholas 19090b0
remove pointless repeated getitem
TomNicholas 49416db
set N_LAZY_VARS correctly in test
TomNicholas 2ed8455
remove unused import
TomNicholas a8a2860
rename flag to make it more clear its only for orthogonal and vectori…
TomNicholas ef6afdf
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] de98308
remove IndexingAdapter special case
TomNicholas e32ea13
type fixes
dcherian da2d43c
return a deepcopy
TomNicholas ac3127f
Merge branch 'async.load' of https://github.com/TomNicholas/xarray in…
TomNicholas d46fc3f
try again
dcherian cc253c7
one more
dcherian 78c9116
Try again
dcherian a727ecb
try fixing _in_memory error by not returning the adapter class
TomNicholas 959edc2
Merge branch 'main' into async.load
TomNicholas 9b7afc2
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] b4ef26f
remove scope=module from fixture for robustness
TomNicholas a7918e4
modify test to be happy with either error message
TomNicholas 199d50a
use Variable instead of Dataset to avoid race condition of indexing b…
TomNicholas 1cbe913
fix bad merge in API docs
TomNicholas 46d9414
add test to test_backends.py
TomNicholas b4a5a90
fix bug found by new test, causing pandas indexes to be converted to …
TomNicholas c5bed0d
Merge branch 'main' into async.load
TomNicholas bc1fe4e
add test to test_variable.py for lazy async indexing
TomNicholas 10f7e61
move async load tests from test_async.py to test_backends.py
TomNicholas 8416f00
parametrize all tests over zarr_format
TomNicholas 4edd503
remove test_async.py file entirely
TomNicholas fcb2c11
lint
TomNicholas 4760574
Stricter assertions
TomNicholas 8bce3bb
Update doc/whats-new.rst
TomNicholas 37be891
add docstring for Variable.load_async
TomNicholas 3bd7b8a
make all load-related docstrings consistent
TomNicholas dfaac7e
note about users being responsible for limiting concurrency
TomNicholas d0a129a
remove parametrization over zarr_format
TomNicholas f30a3a0
account for Dataset having multiple lazy vars
TomNicholas 432bbd5
refactor test parametrization to use pytest.param(..., id=...) syntax
TomNicholas 4f40792
refactor TestBackendIndexing to combine sync and async checks in one …
TomNicholas cf1d127
move test_load_async onto test base class
TomNicholas 54ab88c
Merge branch 'main' into async.load
TomNicholas dda58bf
should fix mypy error
TomNicholas 02d661d
add back in the parametrize_zarr_format to avoid trying to write v3 d…
TomNicholas a074a25
parametrize test over async
TomNicholas f8f5e82
Merge branch 'main' into async.load
TomNicholas File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.