feature(data-types): add data-types from https://github.com/zarr-developers/zarr-python/pull/2874 #5

jhamman · 2025-04-15T19:44:54Z

This PR adds the definition of the 4 new data types from zarr-developers/zarr-python#2874. They include:

datetime64
fixed-length-ascii
fixed-length-bytes
fixed-length-ucs4

Closes #4

…-bytes, and fixed-length-ucs4 data-types from zarr-python

d-v-b · 2025-04-15T19:53:50Z

data-types/datetime64/README.md

+
+## Permitted fill values
+
+The value of the `fill_value` metadata key must be a signed 64-bit integer.


If we want to stick with the integer representation, we should clarify that this is a JSON representation of an integer between -2^63 and 2^63 - 1, inclusive. But as this data type is totally new for zarr v3 and we don't have to worry about backwards compatibility in the json format w.r.t. v2, we could consider a more date-like human-readable alternative to an int. I'd love to get feedback from heavy datetime users on what they would expect.

d-v-b · 2025-04-15T19:59:02Z

data-types/datetime64/README.md

+}
+```
+
+## Notes


If numpy compatibility is still the design goal for this dtype, I think it would make sense to say something about compatibility with the numpy datetime dtype, e.g. with a link to the numpy docs .

d-v-b · 2025-04-15T20:09:18Z

data-types/fixed-length-ascii/README.md

@@ -0,0 +1,33 @@
+# Fixed-length ASCII data type
+
+Defines a data type for fixed-length ASCII strings.


I think my initial name for this dtype (fixed-length-ascii) was misleading -- it models the numpy S dtype, which represents any fixed-length byte string, which includes non-ascii characters. So we should probably rename this to fixed-length-bytes or equivalent.

i think the key distinction between this dtype (modelling numpy S*) and the fixed-length bytes dtype defined elsewhere in this PR (modelling numpy V*) is that scalars with this dtype are intended to be interpreted as strings, not opaque sequences of bytes. We should probably decide if we really want two dtypes for this.

Numpy also assumes NUL-padding, i.e. that any NUL characters at the end can be ignored.

d-v-b · 2025-04-15T20:25:46Z

data-types/fixed-length-ascii/README.md

+
+## Permitted fill values
+
+The value of the `fill_value` metadata key must be a string.


we will need more constraints here, such as as length constraint. for reference, in zarr v2 the fill value was a base64-encoded ascii string but it didn't constrain the length.

jbms · 2025-04-15T21:00:42Z

data-types/datetime64/README.md

+
+## Notes
+
+Valid values for the `unit` configuration option include: `["Y", "M", "W", "D", "h", "m", "s", "ms", "us", "μs", "ns", "ps", "fs", "as"]`


This should say something about the epoch (presumably unix epoch) and time standard (presumably UTC) it uses.

d-v-b · 2025-04-15T21:00:45Z

instead of two fixed-length string data types (ascii and ucs4), we could consider a single fixed-length string dtype that's parametrized by the string encoding.

jbms · 2025-04-15T21:03:44Z

data-types/datetime64/README.md

+
+## Notes
+
+Valid values for the `unit` configuration option include: `["Y", "M", "W", "D", "h", "m", "s", "ms", "us", "μs", "ns", "ps", "fs", "as"]`


Note that the units listed after nanoseconds cannot represent dates beyond 1970 within 64-bits and therefore are presumably not very useful.

The first part of this sentence is correct.

The second is not. There is a lot of use cases in the (paleo-)climate space for dates that go much further back in time.

To be clear, the range of 64-bit picoseconds is just a few months forward or backward from the epoch.

Presumably picoseconds would only be useful either with more than 64 bits or for a time duration rather than time point.

is it helpful if we define the goal of this datetime type as "give numpy datetime64 users a zarr data type they can use without thinking"? In this case, we optimize for compatibility and accept whatever decisions the numpy folks have made. We can always define a separate datetime data type that avoids perceived defects in the numpy datetime dtype, and we can point users who don't need maximal numpy compatibility to the better data type.

Another comment regarding datetime: numpy supports a special NaT (not a time) value. Is that intended to be supported here?

i'm asking because, whether it's sensible or not, ps is a unit numpy datetime64 supports. Instead of litigating that decision, or any other numpy decision, we could optimize for compatibility here, which would simplify the task of this PR considerably.

Another comment regarding datetime: numpy supports a special NaT (not a time) value. Is that intended to be supported here?

yes i believe that should be supported, and so it should be defined in this PR

Those units don't hurt anything they just are very unlikely to be used intentionally. Numpy supports the same units for both datetime and timedelta --- are you intending to also add timedelta as well?

Those units don't hurt anything they just are very unlikely to be used intentionally. Numpy supports the same units for both datetime and timedelta --- are you intending to also add timedelta as well?

yes, timedelta is planned.

jbms · 2025-04-15T21:04:12Z

data-types/datetime64/schema.json

+            "type": "object",
+            "properties": {
+              "unit": {
+                "type": "string"


could be listed as enum rather than string

jbms · 2025-04-15T21:17:20Z

Regarding the datetime type generally, I think I'd be happier if that were handled by some sort of unit attribute that was independent of the data type.

For how this is handled by the udunits2 package, see: https://docs.unidata.ucar.edu/udunits/current/udunits2lib.html#Time

In udunits the unit would be specified as "seconds since 1970-01-01".

The complication with making it a data type is that you have to specify how it is handled by every single codec.

jbms · 2025-04-15T21:18:52Z

All of these need to specify which array -> bytes codecs support them (presumably just bytes), and how they are encoded.

jbms · 2025-04-15T21:23:53Z

data-types/fixed-length-ascii/README.md

+
+## Permitted fill values
+
+The value of the `fill_value` metadata key must be a string.


Why is the length specified in bits? What does it mean if you specify a length that is not a multiple of 8?

I think specifying the length in bits is a zarr-python implementation detail that we shouldn't have to deal with in the spec. i think a length in bytes is better here.

jbms · 2025-04-15T21:25:09Z

data-types/fixed-length-ucs4/README.md

@@ -0,0 +1,33 @@
+# Fixed-length Unicode string data type
+
+Defines a data type for fixed-length Unicode strings.


Probably should also include a recommendation NOT to use this type except for compatibility with existing datasets, because UTF-32 encoding is basically never a good choice.

Probably should include a note that it is intended to only contain valid unicode strings, and that implementations may return an error if given invalid data (e.g. utf-16 surrogate pairs, out-of-range characters) if that is what is intended.

Should include a note that NUL characters at the end are to be ignored, if that is what is intended.

jbms · 2025-04-15T21:25:49Z

data-types/fixed-length-ucs4/README.md

+    "data_type": "fixed-length-ucs4",
+    "fill_value": "",
+    "configuration": {
+      "length_bits": 24


Using 24, which is not a multiple of 32, is confusing.

jbms · 2025-04-15T21:26:05Z

data-types/fixed-length-ucs4/schema.json

+        "configuration": {
+          "type": "object",
+          "properties": {
+            "length_bits": {


why is the length in bits, and what happens if it is not a multiple of 32?

jbms · 2025-04-15T21:26:50Z

data-types/fixed-length-ucs4/README.md

@@ -0,0 +1,33 @@
+# Fixed-length Unicode string data type
+
+Defines a data type for fixed-length Unicode strings.


I think utf-32 is now more commonly used and may be a better choice than ucs-4.

To be clear, I mean the name "utf-32".

jhamman · 2025-04-16T04:00:38Z

Thanks @d-v-b and @jbms for the initial reviews. I'll take a second pass tomorrow.

normanrz · 2025-04-16T19:17:09Z

data-types/datetime64/README.md

+    "data_type": "datetime64",
+    "fill_value": -9223372036854775808,
+    "configuration": {
+      "unit": "s"
+    },


Suggested change

"data_type": "datetime64",

"fill_value": -9223372036854775808,

"configuration": {

"unit": "s"

},

"data_type": { "name": "datetime64", "configuration": {"unit":"s"} },

"fill_value": -9223372036854775808,

normanrz · 2025-04-16T19:18:44Z

data-types/datetime64/schema.json

+        "required": ["name", "configuration"],
+        "additionalProperties": false
+      },
+      { "const": "datetime64" }


If the configuration (with its unit field) is mandatory, the short-hand should not be permissible.

normanrz · 2025-05-06T17:15:08Z

Since you mentioned that you are still working on this, I turned this PR into a draft PR, in accordance with #13. Feel free to switch to a regular PR as soon as you think it is ready.

feature(data-types): add datetime64, fixed-length-ascii, fixed-length…

89ea29d

…-bytes, and fixed-length-ucs4 data-types from zarr-python

d-v-b reviewed Apr 15, 2025

View reviewed changes

jbms reviewed Apr 15, 2025

View reviewed changes

jhamman mentioned this pull request Apr 16, 2025

adds extensions that zarr-python defines #1

Merged

normanrz reviewed Apr 16, 2025

View reviewed changes

abarciauskas-bgse mentioned this pull request Apr 25, 2025

ODD PI 25.3 Objective 5: Foundational Zarr-Python and Xarray Contributions NASA-IMPACT/veda-odd#165

Open

3 tasks

mkitti mentioned this pull request Apr 27, 2025

Reserve characters in names #7

Open

normanrz marked this pull request as draft May 6, 2025 17:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature(data-types): add data-types from https://github.com/zarr-developers/zarr-python/pull/2874 #5

feature(data-types): add data-types from https://github.com/zarr-developers/zarr-python/pull/2874 #5

jhamman commented Apr 15, 2025

d-v-b Apr 15, 2025

d-v-b Apr 15, 2025

d-v-b Apr 15, 2025

d-v-b Apr 15, 2025

jbms Apr 15, 2025

d-v-b Apr 15, 2025

jbms Apr 15, 2025

d-v-b commented Apr 15, 2025

jbms Apr 15, 2025

jhamman Apr 17, 2025

jbms Apr 17, 2025

d-v-b Apr 17, 2025

jbms Apr 17, 2025

d-v-b Apr 17, 2025

d-v-b Apr 17, 2025

jbms Apr 17, 2025

d-v-b Apr 17, 2025

jbms Apr 15, 2025

jbms commented Apr 15, 2025

jbms commented Apr 15, 2025

jbms Apr 15, 2025

d-v-b Apr 15, 2025

jbms Apr 15, 2025

jbms Apr 15, 2025

jbms Apr 15, 2025

jbms Apr 15, 2025

jbms Apr 15, 2025

jbms Apr 15, 2025

jbms Apr 15, 2025

jhamman commented Apr 16, 2025

normanrz Apr 16, 2025

normanrz Apr 16, 2025

normanrz commented May 6, 2025


		## Permitted fill values

		The value of the `fill_value` metadata key must be a signed 64-bit integer.

		@@ -0,0 +1,33 @@
		# Fixed-length ASCII data type

		Defines a data type for fixed-length ASCII strings.


		## Notes

		Valid values for the `unit` configuration option include: `["Y", "M", "W", "D", "h", "m", "s", "ms", "us", "μs", "ns", "ps", "fs", "as"]`

		@@ -0,0 +1,33 @@
		# Fixed-length Unicode string data type

		Defines a data type for fixed-length Unicode strings.

feature(data-types): add data-types from https://github.com/zarr-developers/zarr-python/pull/2874 #5

Are you sure you want to change the base?

feature(data-types): add data-types from https://github.com/zarr-developers/zarr-python/pull/2874 #5

Conversation

jhamman commented Apr 15, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

d-v-b commented Apr 15, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbms commented Apr 15, 2025

jbms commented Apr 15, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhamman commented Apr 16, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

normanrz commented May 6, 2025