What happened?
format_items() in xarray/core/formatting.py casts every timedelta64 array to
timedelta64[ns] before splitting it into a whole-day part and a remaining time part:
x = astype(x, dtype="timedelta64[ns]")
day_part = x[~pd.isnull(x)].astype("timedelta64[D]").astype("timedelta64[ns]")
That cast overflows for values that are perfectly valid in a coarser unit, so printing a
timedelta64[s] array whose magnitude exceeds the range of timedelta64[ns] (~292 years)
raises OverflowError instead of displaying the values. This affects repr(), and
therefore also _repr_html_() in Jupyter.
It is not limited to coarse units: the same code also fails for timedelta64[ns] arrays
holding large negative values, because numpy's conversion of a large negative timedelta64
to a coarser unit goes through a value that overflows.
I verified this against current main (commit 84b9d3d7).
What did you expect to happen?
repr() should show the values, in whatever unit the array already uses. The day/time
split only needs floor division and a remainder, neither of which requires converting
between units.
Minimal Complete Verifiable Example
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "xarray[complete]@git+https://github.com/pydata/xarray.git@main",
# ]
# ///
import numpy as np
import xarray as xr
# 10**10 seconds is ~317 years: a valid timedelta64[s], but outside the range
# of timedelta64[ns].
seconds = np.array([0, 10**10], dtype="timedelta64[s]")
ds = xr.Dataset({"duration": ("t", seconds)}, coords={"t": ("t", seconds)})
print(ds) # raises OverflowError
print(ds._repr_html_()) # raises OverflowError
# it also survives a netCDF round trip that xarray itself writes and reads back
ds.to_netcdf("duration.nc")
print(xr.load_dataset("duration.nc")) # raises OverflowError
A value that is valid but out of the nanosecond range is enough:
import numpy as np
import xarray as xr
# large negative values overflow the same way, even in [ns]
a = np.array([-(2**63 - 1), 0, 2**63 - 1], dtype="timedelta64[ns]")
print(xr.Dataset({"v": ("x", a)}, coords={"x": ("x", a)})) # raises OverflowError
Steps to reproduce
Run the script above.
MVCE confirmation
Relevant log output
Traceback (most recent call last):
File "issue.py", line 13, in <module>
print(ds)
File ".../xarray/core/dataset.py", line 2487, in __repr__
return formatting.dataset_repr(self)
...
File ".../xarray/core/formatting.py", line 237, in format_array_flat
relevant_back_items = format_items(last_n_items(array, max_possibly_relevant // 2))
File ".../xarray/core/formatting.py", line 214, in format_items
x = astype(x, dtype="timedelta64[ns]")
File ".../xarray/core/duck_array_ops.py", line 277, in astype
return data.astype(dtype, **kwargs)
OverflowError: Overflow when converting between datetime64 units
Note that ds.to_series() on the same object works fine and prints
115740 days 17:46:40, so the data itself is fine; only the repr path is affected.
Anything else we need to know?
Doing the split in the array's own unit, with // and np.mod against a
"one day expressed in that unit" scalar, avoids the conversion entirely and so cannot
overflow. Both operations use floor semantics, which reproduce the current output
exactly for every input that could be formatted before.
What happened?
format_items()inxarray/core/formatting.pycasts everytimedelta64array totimedelta64[ns]before splitting it into a whole-day part and a remaining time part:That cast overflows for values that are perfectly valid in a coarser unit, so printing a
timedelta64[s]array whose magnitude exceeds the range oftimedelta64[ns](~292 years)raises
OverflowErrorinstead of displaying the values. This affectsrepr(), andtherefore also
_repr_html_()in Jupyter.It is not limited to coarse units: the same code also fails for
timedelta64[ns]arraysholding large negative values, because numpy's conversion of a large negative
timedelta64to a coarser unit goes through a value that overflows.
I verified this against current
main(commit84b9d3d7).What did you expect to happen?
repr()should show the values, in whatever unit the array already uses. The day/timesplit only needs floor division and a remainder, neither of which requires converting
between units.
Minimal Complete Verifiable Example
A value that is valid but out of the nanosecond range is enough:
Steps to reproduce
Run the script above.
MVCE confirmation
Relevant log output
Note that
ds.to_series()on the same object works fine and prints115740 days 17:46:40, so the data itself is fine; only the repr path is affected.Anything else we need to know?
Doing the split in the array's own unit, with
//andnp.modagainst a"one day expressed in that unit" scalar, avoids the conversion entirely and so cannot
overflow. Both operations use floor semantics, which reproduce the current output
exactly for every input that could be formatted before.