API: date_range, timedelta_range infer unit from start/end/freq #63146

jbrockmendel · 2025-11-18T21:56:04Z

closes ENH/BUG: pd.date_range() still defaults to nanosecond resolution #59031 (Replace xxxx with the GitHub issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.
If I used AI to develop this pull request, I prompted it to follow AGENTS.md.

Waiting to update expected's until after #62801, so this won't pass yet.

jorisvandenbossche · 2025-11-21T21:56:02Z

Should we only do this for pd.date_range for now, and not yet for pd.timedelta_range, since in general timedelta still defaults to nanoseconds? (eg when converting strings or integers in pd.to_timedelta)

jbrockmendel · 2025-11-21T22:47:21Z

I think best case would be to do it all at once (xref #63018 gets everything but the string-parsing inference for td64). But if it'll make a difference in getting a release out soon, I'll happily revert.

jorisvandenbossche · 2025-11-21T23:17:28Z

Ah, I hadn't seen #63018. Yes, my preference would also be to do it all at once. Will try to take a look at that one then.

rhshadrach

Do we have tests for the inference itself? I could have missed it.

rhshadrach · 2025-11-25T03:01:49Z

pandas/core/indexes/datetimes.py

-    unit : {'s', 'ms', 'us', 'ns'}, default 'ns'
+    unit : {'s', 'ms', 'us', 'ns', None}, default None
        Specify the desired resolution of the result.
+        If not specified, this is inferred from the 'start', 'end', and 'freq'


Is it possible to given an indication of how the inference works?

What did you have in mind? I can't think of an explanation that is meaningfully simpler than "read the code"

Accidentally started a separate chain here: #63146 (comment)

rhshadrach · 2025-11-25T03:02:37Z

pandas/core/indexes/timedeltas.py

-    unit : {'s', 'ms', 'us', 'ns'}, default 'ns'
+    unit : {'s', 'ms', 'us', 'ns', None}, default None
        Specify the desired resolution of the result.
+        If not specified, this is inferred from the 'start', 'end', and 'freq'


jbrockmendel · 2025-11-25T17:51:55Z

Do we have tests for the inference itself? I could have missed it.

I didn't add dedicated tests bc existing tests hit all the cases, but will do so now.

rhshadrach

lgtm

rhshadrach · 2025-11-25T21:57:36Z

pandas/core/indexes/datetimes.py

-    unit : {'s', 'ms', 'us', 'ns'}, default 'ns'
+    unit : {'s', 'ms', 'us', 'ns', None}, default None
        Specify the desired resolution of the result.
+        If not specified, this is inferred from the 'start', 'end', and 'freq'


I realized writing this that we also don't document Timestamp inference (at least, in the API docs). So this is perhaps a little incomplete unless that is done, but as long as there is no opposition to doing so I can open an issue for that.

Suggested change

If not specified, this is inferred from the 'start', 'end', and 'freq'

If not specified, this is inferred from the 'start', 'end', and 'freq' using the same inference as :class:`Timestamp` taking the highest resolution of the three that are provided.

Let's track as a separate issue indeed (your above suggestion does not make it much more clear, right now, IMO)

jorisvandenbossche

Actual change looks good to me!

Personally, I would do less of unit="ns" in the tests (to ensure we test the default unit more as well, and it also just makes them a bit less verbose), but it that requires more updates in the tests to get them passing, that's certainly fine for a follow-up issue.

jorisvandenbossche · 2025-11-26T16:45:32Z

pandas/tests/apply/test_frame_apply.py

+    df = DataFrame(
+        {"dt": date_range("2015-01-01", periods=3, tz="Europe/Brussels", unit="ns")}
+    )
    result = df.apply(lambda x: x)
    tm.assert_frame_equal(result, df)

    result = df.apply(lambda x: x + pd.Timedelta("1day"))
    expected = DataFrame(
-        {"dt": date_range("2015-01-02", periods=3, tz="Europe/Brussels")}
+        {"dt": date_range("2015-01-02", periods=3, tz="Europe/Brussels", unit="ns")}


This doesn't (yet) work without specifying the unit in both cases?

ATM the Timedelta("1day") above has "ns" unit, so we need it in the expected line. In order to be robust to the Timedelta PR, need it in the df line too. Can remove both if the Timedelta PR goes in.

jorisvandenbossche · 2025-11-26T16:46:31Z

pandas/tests/apply/test_frame_apply.py

            "B": [1.0, 2.0, 3.0],
            "C": ["foo", "bar", "baz"],
-            "D": date_range("20130101", periods=3),
+            "D": date_range("20130101", periods=3, unit="ns"),


Suggested change

"D": date_range("20130101", periods=3, unit="ns"),

"D": date_range("20130101", periods=3),

and then remove the .as_unit("ns") below?

jorisvandenbossche · 2025-11-26T16:53:20Z

pandas/tests/indexes/datetimes/test_date_range.py

+        start = Timestamp("2025-11-25").as_unit("ms")
+        end = Timestamp("2025-11-26").as_unit("s")
+        dti = date_range(start, end, freq=off)
+        assert dti.unit == "us"


Suggested change

start = Timestamp("2025-11-25").as_unit("ms")

end = Timestamp("2025-11-26").as_unit("s")

dti = date_range(start, end, freq=off)

assert dti.unit == "us"

start = Timestamp("2025-11-25 09:00:00").as_unit("s")

end = Timestamp("2025-11-25 09:00:02").as_unit("s")

dti = date_range(start, end, freq=off)

assert dti.unit == "us"

off = DateOffset(milleconds=2)

dti = date_range(start, end, freq=off)

assert dti.unit == "ms"

off = DateOffset(nanoseconds=2)

dti = date_range(start, end, freq=off)

assert dti.unit == "ns"

to also cover the two other cases? (unless that is already covered elsewhere?)

Sure, though i'll change the nano case to make a smaller array

jbrockmendel · 2025-11-26T17:38:32Z

Personally, I would do less of unit="ns" in the tests (to ensure we test the default unit more as well, and it also just makes them a bit less verbose), but it that requires more updates in the tests to get them passing, that's certainly fine for a follow-up issue.

I'm of mixed opinion on this. Ended up deciding that outside of tests targeted at date_range, I prefer to be a little bit more verbose in order to be robust to e.g. decisions on other PRs

jbrockmendel added 15 commits November 18, 2025 13:54

API: date_range, timedelta_range infer unit from start/end/freq

908a88e

Merge branch 'main' into api-date_range

257f410

update expecteds

df7376e

mypy fixup

6fae9cc

stub fixup

8863e26

stubtest fixup

3d8c0b6

stubtest fixup

3b2cd0d

Merge branch 'main' into api-date_range

4210a9e

update gcs test

affa6f0

update slow test

964c908

update xarray test

6dfc65d

pyright fixup

873ea3a

update doctests

a142fbf

update doctests

966f630

update doctests

615a712

rhshadrach reviewed Nov 25, 2025

View reviewed changes

jbrockmendel added 4 commits November 25, 2025 10:56

dedicated tests

be8e591

Merge branch 'main' into api-date_range

2575866

Merge branch 'main' into api-date_range

b917aa3

no-longer-necessary type ignore

0a47f79

rhshadrach approved these changes Nov 25, 2025

View reviewed changes

rhshadrach added Timedelta Timedelta data type Datetime Datetime data dtype API - Consistency Internal Consistency of API/Behavior labels Nov 25, 2025

rhshadrach added this to the 3.0 milestone Nov 25, 2025

docstring about how unit inference is done

4506bc5

jorisvandenbossche approved these changes Nov 26, 2025

View reviewed changes

suggestd edits

cd160be

mroeschke mentioned this pull request Nov 26, 2025

RLS: 3.0 #57064

Open

Merge branch 'main' into api-date_range

68d058b

	If not specified, this is inferred from the 'start', 'end', and 'freq'
	If not specified, this is inferred from the 'start', 'end', and 'freq' using the same inference as :class:`Timestamp` taking the highest resolution of the three that are provided.

	"D": date_range("20130101", periods=3, unit="ns"),
	"D": date_range("20130101", periods=3),

Uh oh!

API: date_range, timedelta_range infer unit from start/end/freq #63146

Are you sure you want to change the base?

API: date_range, timedelta_range infer unit from start/end/freq #63146

Conversation

jbrockmendel commented Nov 18, 2025

Uh oh!

jorisvandenbossche commented Nov 21, 2025

Uh oh!

jbrockmendel commented Nov 21, 2025

Uh oh!

jorisvandenbossche commented Nov 21, 2025

Uh oh!

rhshadrach left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbrockmendel commented Nov 25, 2025

Uh oh!

rhshadrach left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbrockmendel commented Nov 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants