Release/2.0.0

.github/workflows/linting.yml

@@ -1,6 +1,9 @@
 name: linting-and-code-formatting
 
-on: push
+on: 
+  push:
+  pull_request:
+    types: [opened, synchronize, reopened]
 
 jobs:
   build:

.github/workflows/publish.yml

@@ -1,6 +1,9 @@
 name: build-wheel-and-publish-test-pypi
 
-on: push
+on:
+  push:
+  pull_request:
+    types: [opened, synchronize, reopened]
 
 jobs:
   build:

.github/workflows/test_with_pytest.yml

@@ -1,6 +1,9 @@
 name: Test using pytest
 
-on: [push, pull_request]
+on: 
+    push:
+    pull_request:
+      types: [opened, synchronize, reopened]
 
 jobs:
   test:

.github/workflows/verify_sphinx_doc.yml

@@ -0,0 +1,53 @@
+name: verify-sphinx-doc-generation
+
+on:
+  push:
+  pull_request:
+    types: [opened, synchronize, reopened]
+
+jobs:
+  build-and-test:
+    runs-on: ubuntu-22.04
+
+    steps:
+      - name: Checkout IMAS-Python sources
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          with:
+          # until saxonche is available in 3.13
+          # https://saxonica.plan.io/issues/6561
+          python-version: "<3.13"
+
+      - name: Display Python version
+        run: python -c "import sys; print(sys.version)"
+
+
+      - name: Set up Python virtual environment
+        run: |
+          python -m venv venv
+          source venv/bin/activate
+
+      - name: Install build dependencies
+        run: |
+          pip install --upgrade pip setuptools wheel build
+
+      - name: Build package
+        run: |
+          rm -rf dist
+          python -m build .
+
+      - name: Install package and dependencies
+        run: |
+          pip install "$(readlink -f dist/*.whl)[docs,netcdf]"
+
+      - name: Debug dependencies
+        run: |
+          pip freeze
+
+      - name: Build Sphinx documentation
+        run: |
+          export SPHINXOPTS='-W -n --keep-going'
+          make -C docs clean html

README.md

@@ -32,11 +32,11 @@ equilibrium.ids_properties.homogeneous_time = imas.ids_defs.IDS_TIME_MODE_HOMOGE
 equilibrium.ids_properties.comment = "testing"
 equilibrium.time = [0.01]
 
-with imas.DBEntry("imas:hdf5?path=./testdb","w") as dbentry:
+with imas.DBEntry("imas:hdf5?path=./testdb", "w") as dbentry:
     dbentry.put(equilibrium)
 
 # or without imas_core dependency
-with imas.DBEntry("./test.nc","w") as dbentry:
+with imas.DBEntry("./test.nc", "w") as dbentry:
     dbentry.put(equilibrium)
 
 ```

docs/source/changelog.rst

@@ -3,13 +3,33 @@
 Changelog
 =========
 
+What's new in IMAS-Python 2.0.0
+-------------------------------
+
+Breaking change
+'''''''''''''''
+
+The package name was changed from ``imaspy`` to ``imas`` while porting the code to GitHub. This shall only affect the import statements in your code. 
+
+New features and improvements
+'''''''''''''''''''''''''''''
+
+- Add :py:func:`imas.util.to_xarray` to convert a full IDS or only specific paths herein to a Xarray ``Dataset``. See :ref:`Convert IMAS-Python IDSs directly to Xarray Datasets` for more details.
+- Implements automatic DD version conversion on :py:meth:`imas.db_entry.DBEntry.get` (conversion during :py:meth:`imas.db_entry.DBEntry.put` is not supported as this is rarely needed and easily worked around).
+- Enable lazy loading when reading data from a netCDF file.
+- Minor performance improvement loading data from a netCDF file.
+- Replace versioneer by setuptools-scm to determine the version of the code.
+- Use `saxonche <https://pypi.org/project/saxonche/>`__ instead of the JAR for XSL transforms (when building versions of the DD).
+- Updating the README, CONTRIBUTING guidelines and documentation after making the code open access.
+
+
 What's new in IMAS-Python 1.2.0
---------------------------
+-------------------------------
 
 New features and improvements
 '''''''''''''''''''''''''''''
 
-- Add :py:func:`imaspy.DBEntry.get_sample` (requires imas_core >= 5.4.0)
+- Add :py:func:`imaspy.DBEntry.get_sample <imas.db_entry.DBEntry.get_sample>` (requires imas_core >= 5.4.0)
 - Improved validation of netCDF files
 - Improve compatibility with the UDA backend in imas_core
 - Extend the support of netCDF to >= 1.4.1 (without complex numbers)

docs/source/intro.rst

@@ -20,11 +20,11 @@ be outdated.
 
     >>> import imas
     >>> print(imas.__version__)
-    1.0.0
+    2.0.0
 
 .. note::
 
-    If you have an IMAS-Python install without the IMAS Access Layer, importing
+    If you have an IMAS-Python install without the IMAS-Core, importing
     IMAS-Python will display an error message. You can still use IMAS-Python, but not all
     functionalities are available.
 
@@ -44,7 +44,7 @@ on different Data Dictionary versions.
     >>> import imas
     >>> import numpy as np
     >>> ids_factory = imas.IDSFactory()
-    13:26:47 [INFO] Parsing data dictionary version 3.38.1 @dd_zip.py:127
+    18:23:12 INFO Parsing data dictionary version 4.0.0 @dd_zip.py:166
     >>> # Create an empty core_profiles IDS
     >>> core_profiles = ids_factory.core_profiles()
 
@@ -56,6 +56,7 @@ We can now use this ``core_profiles`` IDS and assign some data to it:
     >>> core_profiles.ids_properties.homogeneous_time = imas.ids_defs.IDS_TIME_MODE_HOMOGENEOUS
     >>> # array quantities are automatically converted to the appropriate numpy arrays
     >>> core_profiles.time = [1, 2, 3]
+    18:24:58 INFO     Assigning incorrect type 'int64' to <IDSNumericArray (IDS:core_profiles, time, empty FLT_1D)>, attempting automatic conversion. @ids_primitive.py:483
     >>> # the python list of ints is converted to a 1D array of floats
     >>> core_profiles.time
     <IDSNumericArray (IDS:core_profiles, time, FLT_1D)>
@@ -66,7 +67,7 @@ We can now use this ``core_profiles`` IDS and assign some data to it:
     3
     >>> # assign some data for the first time slice
     >>> core_profiles.profiles_1d[0].grid.rho_tor_norm = [0, 0.5, 1.0]
-    >>> core_profiles.profiles_1d[0].j_tor = [0, 0, 0]
+    >>> core_profiles.profiles_1d[0].j_phi = [0., 0., 0.]
 
 As you can see in the example above, IMAS-Python automatically checks the data you try to
 assign to an IDS with the data type specified in the Data Dictionary. When
@@ -77,63 +78,77 @@ get an error message if this is not possible:
 
     >>> core_profiles.time = "Cannot be converted"
     ValueError: could not convert string to float: 'Cannot be converted'
-    >>> core_profiles.time = 1-1j
-    TypeError: can't convert complex to float
-    >>> core_profiles.ids_properties.source = 1-1j  # automatically converted to str
-    >>> core_profiles.ids_properties.source.value
-    '(1-1j)'
+    >>> core_profiles.ids_properties.comment = 1-1j  # automatically converted to str
+    >>> core_profiles.ids_properties.comment
+    str('(1-1j)')
 
 
-Store an IDS to disk
-''''''''''''''''''''
+Load and store an IDS to disk with IMAS-Core
+''''''''''''''''''''''''''''''''''''''''''''
 
 .. note::
 
-    - This functionality requires the IMAS Access Layer.
-    - This API will change when IMAS-Python is moving to Access Layer 5 (expected Q2
-      2023).
+    - This functionality requires the IMAS-Core, until this library is openly available
+      on GitHub you may need to fetch it from `git.iter.org <https://git.iter.org/>`_
+      (requires to have an ITER account). Using IMAS-Core also enable slicing methods
+      :py:meth:`~imas.db_entry.DBEntry.get_slice`, 
+      :py:meth:`~imas.db_entry.DBEntry.put_slice` and
+      :py:meth:`~imas.db_entry.DBEntry.get_sample` (with IMAS-Core>=5.4).
+    - If you can't have access to it, you can save IDS to disk with the built-in
+      netCDF backend :ref:`Store and load an IDS to disk with netCDF`
 
-To store an IDS to disk, we need to indicate the following information to the
-IMAS Access Layer. Please check the `IMAS Access Layer documentation
-<https://imas.iter.org/>`_ for more information on this.
+To store an IDS to disk, we need to indicate the following URI to the
+IMAS-Core: ``imas:<backend>?path=<path_to_folder>`` or using the legacy query keys
+``imas:<backend>?user=<user>;database=<database>;version=<version>;pulse=<pulse>;run=<run>``
+which are then converted as a path ``~user/public/imasdb/database/version/pulse/run``.
 
-- Which backend to use (e.g. MDSPLUS or HDF5)
-- ``tokamak`` (also known as database)
-- ``pulse``
-- ``run``
+Available ``<backend>`` may depend on your IMAS-Core install: ``hdf5``, ``mdsplus``,
+``ascii``, ``memory``, ``uda``.
 
 In IMAS-Python you do this as follows:
 
 .. code-block:: python
 
     >>> # Create a new IMAS data entry for storing the core_profiles IDS we created earlier
     >>> # Here we specify the backend, database, pulse and run
-    >>> dbentry = imas.DBEntry(imas.ids_defs.HDF5_BACKEND, "TEST", 10, 2)
-    >>> dbentry.create()
+    >>> dbentry = imas.DBEntry("imas:hdf5?path=./testdb","w")
     >>> # now store the core_profiles IDS we just populated
     >>> dbentry.put(core_profiles)
 
 .. image:: imas_structure.png
 
+To load an IDS from disk, you need to specify the same information as
+when storing the IDS (see above). Once the data entry is opened, you
+can use ``<IDS>.get()`` to load IDS data from disk: 
 
-Load an IDS from disk
-'''''''''''''''''''''
+.. code-block:: python
 
-.. note::
+    >>> # Now load the core_profiles IDS back from disk
+    >>> dbentry2 = imas.DBEntry("imas:hdf5?path=./testdb","r")
+    >>> core_profiles2 = dbentry2.get("core_profiles")
+    >>> print(core_profiles2.ids_properties.comment.value)
 
-    - This functionality requires the IMAS Access Layer.
-    - This API will change when IMAS-Python is moving to Access Layer 5 (expected Q2
-      2023).
 
-To load an IDS from disk, you need to specify the same information as
-when storing the IDS (see previous section). Once a data entry is opened, you
+Load and store an IDS to disk with IMAS-Core
+''''''''''''''''''''''''''''''''''''''''''''
+
+In IMAS-Python you do this as follows:
+
+.. code-block:: python
+
+    >>> # Create a new IMAS data entry for storing the core_profiles IDS we created earlier
+    >>> # here we directly point to a .nc filename in your system
+    >>> dbentry = imas.DBEntry("mypulsefile.nc","w")
+    >>> # now store the core_profiles IDS we just populated
+    >>> dbentry.put(core_profiles)
+
+To load an IDS from disk, you need to specify the same file information as
+when storing the IDS. Once the data entry is opened, you
 can use ``<IDS>.get()`` to load IDS data from disk: 
 
 .. code-block:: python
 
     >>> # Now load the core_profiles IDS back from disk
-    >>> dbentry2 = imas.DBEntry(imas.ids_defs.HDF5_BACKEND, "TEST", 10, 2)
-    >>> dbentry2.open()
+    >>> dbentry2 = imas.DBEntry("mypulsefile.nc","r")
     >>> core_profiles2 = dbentry2.get("core_profiles")
     >>> print(core_profiles2.ids_properties.comment.value)
-    Testing IMAS-Python

imas/backends/imas_core/al_context.py

@@ -282,6 +282,15 @@ def __init__(
         """Potential weak reference to opened context."""
 
     def get_child(self, child):
+        """
+        Retrieve a child entry from the field.
+
+        Args:
+            child (str): The name or identifier of the child entry to retrieve.
+
+        Returns:
+            The child entry retrieved from the database.
+        """
         imas.backends.imas_core.db_entry_helpers._get_child(child, self)
 
     def get_context(self) -> ALContext:

imas/backends/imas_core/db_entry_helpers.py

@@ -77,15 +77,11 @@ def get_children(
                 getattr(structure, name)._IDSPrimitive__value = data
 
 
-def _get_child(child: IDSBase, ctx: Optional[LazyALContext]):
+def _get_child(child: IDSBase, ctx: LazyALContext):
     """Get a single child when required (lazy loading)."""
     # NOTE: changes in this method must be propagated to _get_children and vice versa
     #   Performance: this method is specialized for the lazy get
 
-    # ctx can be None when the parent structure does not exist in the on-disk DD version
-    if ctx is None:
-        return  # There is no data to be loaded
-
     time_mode = ctx.time_mode
     if time_mode == IDS_TIME_MODE_INDEPENDENT and child.metadata.type.is_dynamic:
         return  # skip dynamic (time-dependent) nodes

imas/backends/netcdf/ids_tensorizer.py

@@ -169,6 +169,17 @@ def filter_coordinates(self, path: str) -> str:
         )
 
     def tensorize(self, path, fillvalue):
+        """
+        Tensorizes the data at the given path with the specified fill value.
+
+        Args:
+            path: The path to the data in the IDS.
+            fillvalue: The value to fill the tensor with. Can be of any type,
+                             including strings.
+
+        Returns:
+            A tensor filled with the data from the specified path.
+        """
         dimensions = self.ncmeta.get_dimensions(path, self.homogeneous_time)
         shape = tuple(self.dimension_size[dim] for dim in dimensions)
 

imas/backends/netcdf/nc2ids.py

@@ -317,6 +317,14 @@ def __init__(self, nc2ids, index=()):
         self.index = index
 
     def get_child(self, child):
+        """
+        Retrieves and sets the appropriate context or value for a given
+        child node based on its metadata.
+
+        Args:
+            child: The child IDS node which should be lazy loaded.
+
+        """
         metadata = child.metadata
         path = metadata.path_string
         data_type = metadata.data_type
@@ -366,12 +374,44 @@ def get_child(self, child):
 
 
 class LazyArrayStructContext(LazyContext):
+    """
+    LazyArrayStructContext is a subclass of LazyContext that provides a context for
+    handling structured arrays in a lazy manner. It is initialized with a NetCDF to
+    IDS mapping object, an index, and a size.
+    """
+
     def __init__(self, nc2ids, index, size):
+        """
+        Initialize the instance with nc2ids, index, and size.
+
+        Args:
+            nc2ids: The NetCDF to IDS mapping object.
+            index: The index within the NetCDF file.
+            size: The size of the data to be processed.
+        """
         super().__init__(nc2ids, index)
         self.size = size
 
     def get_context(self):
+        """
+        Returns the current context.
+
+        This method returns the current instance of the class, which is expected
+        to have a 'size' attribute as required by IDSStructArray.
+
+        Returns:
+            The current instance of the class.
+        """
         return self  # IDSStructArray expects to get something with a size attribute
 
     def iterate_to_index(self, index: int) -> LazyContext:
+        """
+        Iterates to a specified index and returns a LazyContext object.
+
+        Args:
+            index (int): The index to iterate to.
+
+        Returns:
+            LazyContext: A LazyContext object initialized with the updated index.
+        """
         return LazyContext(self.nc2ids, self.index + (index,))