Skip to content

repr() raises OverflowError for timedelta64 arrays outside the nanosecond range #11630

Description

@aniruddhaadak80

What happened?

format_items() in xarray/core/formatting.py casts every timedelta64 array to
timedelta64[ns] before splitting it into a whole-day part and a remaining time part:

x = astype(x, dtype="timedelta64[ns]")
day_part = x[~pd.isnull(x)].astype("timedelta64[D]").astype("timedelta64[ns]")

That cast overflows for values that are perfectly valid in a coarser unit, so printing a
timedelta64[s] array whose magnitude exceeds the range of timedelta64[ns] (~292 years)
raises OverflowError instead of displaying the values. This affects repr(), and
therefore also _repr_html_() in Jupyter.

It is not limited to coarse units: the same code also fails for timedelta64[ns] arrays
holding large negative values, because numpy's conversion of a large negative timedelta64
to a coarser unit goes through a value that overflows.

I verified this against current main (commit 84b9d3d7).

What did you expect to happen?

repr() should show the values, in whatever unit the array already uses. The day/time
split only needs floor division and a remainder, neither of which requires converting
between units.

Minimal Complete Verifiable Example

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "xarray[complete]@git+https://github.com/pydata/xarray.git@main",
# ]
# ///
import numpy as np
import xarray as xr

# 10**10 seconds is ~317 years: a valid timedelta64[s], but outside the range
# of timedelta64[ns].
seconds = np.array([0, 10**10], dtype="timedelta64[s]")

ds = xr.Dataset({"duration": ("t", seconds)}, coords={"t": ("t", seconds)})

print(ds)          # raises OverflowError
print(ds._repr_html_())  # raises OverflowError

# it also survives a netCDF round trip that xarray itself writes and reads back
ds.to_netcdf("duration.nc")
print(xr.load_dataset("duration.nc"))  # raises OverflowError

A value that is valid but out of the nanosecond range is enough:

import numpy as np
import xarray as xr

# large negative values overflow the same way, even in [ns]
a = np.array([-(2**63 - 1), 0, 2**63 - 1], dtype="timedelta64[ns]")
print(xr.Dataset({"v": ("x", a)}, coords={"x": ("x", a)}))  # raises OverflowError

Steps to reproduce

Run the script above.

MVCE confirmation

  • Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • Complete example — the example is self-contained, including all data and the text of any traceback.
  • Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • New issue — a search of GitHub Issues suggests this is not a duplicate.
  • Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

Traceback (most recent call last):
  File "issue.py", line 13, in <module>
    print(ds)
  File ".../xarray/core/dataset.py", line 2487, in __repr__
    return formatting.dataset_repr(self)
  ...
  File ".../xarray/core/formatting.py", line 237, in format_array_flat
    relevant_back_items = format_items(last_n_items(array, max_possibly_relevant // 2))
  File ".../xarray/core/formatting.py", line 214, in format_items
    x = astype(x, dtype="timedelta64[ns]")
  File ".../xarray/core/duck_array_ops.py", line 277, in astype
    return data.astype(dtype, **kwargs)
OverflowError: Overflow when converting between datetime64 units

Note that ds.to_series() on the same object works fine and prints
115740 days 17:46:40, so the data itself is fine; only the repr path is affected.

Anything else we need to know?

Doing the split in the array's own unit, with // and np.mod against a
"one day expressed in that unit" scalar, avoids the conversion entirely and so cannot
overflow. Both operations use floor semantics, which reproduce the current output
exactly for every input that could be formatted before.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions