Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 27 additions & 4 deletions docs/api/gift-eval/gift-eval.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,34 @@
# `timecopilot.gift_eval`
# `timecopilot_gift_eval`

::: timecopilot.gift_eval.eval
GIFT-Eval integration lives in the standalone [`timecopilot-gift-eval`](https://pypi.org/project/timecopilot-gift-eval/) package on PyPI. Install it with TimeCopilot via:

```bash
pip install "timecopilot[gift-eval]"
```

## Migration from `timecopilot.gift_eval`

The in-repo module `timecopilot.gift_eval` was removed in favor of the standalone package. Update imports as follows:

| Before | After |
|--------|-------|
| `from timecopilot.gift_eval.eval import GIFTEval` | `from timecopilot_gift_eval import GIFTEval` |
| `from timecopilot.gift_eval.gluonts_predictor import GluonTSPredictor` | `from timecopilot_gift_eval import GluonTSPredictor` |
| `from timecopilot.gift_eval.utils import DATASETS_WITH_TERMS` | `from timecopilot_gift_eval.utils import DATASETS_WITH_TERMS` |

No compatibility shim is provided — install `timecopilot-gift-eval` (or `timecopilot[gift-eval]`) and switch to the `timecopilot_gift_eval` import path.

::: timecopilot_gift_eval.eval
options:
members:
- GIFTEval

::: timecopilot.gift_eval.gluonts_predictor
::: timecopilot_gift_eval.gluonts_predictor
options:
members:
- GluonTSPredictor

::: timecopilot_gift_eval.protocol
options:
members:
- GluonTSPredictor
- ForecasterProtocol
1 change: 1 addition & 0 deletions docs/changelogs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

Welcome to the TimeCopilot Changelog. Here, you will find a comprehensive list of all the changes, updates, and improvements made to the TimeCopilot project. This section is designed to keep you informed about the latest features, bug fixes, and enhancements as we continue to develop and refine the TimeCopilot experience. Stay tuned for regular updates and feel free to explore the details of each release below.

- [v0.0.31](v0.0.31.md)
- [v0.0.30](v0.0.30.md)
- [v0.0.29](v0.0.29.md)
- [v0.0.28](v0.0.28.md)
Expand Down
22 changes: 22 additions & 0 deletions docs/changelogs/v0.0.31.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
### Breaking Changes

* **GIFT-Eval moved to standalone package**: The in-repo `timecopilot.gift_eval` module was removed. GIFT-Eval integration now lives in the [`timecopilot-gift-eval`](https://pypi.org/project/timecopilot-gift-eval/) PyPI package (`timecopilot_gift_eval` import path). Install via `pip install "timecopilot[gift-eval]"` or `pip install timecopilot-gift-eval>=0.3.0`.

**Migration:**

```text
from timecopilot.gift_eval.eval import GIFTEval
from timecopilot.gift_eval.gluonts_predictor import GluonTSPredictor
from timecopilot.gift_eval.utils import DATASETS_WITH_TERMS
```

```python
from timecopilot_gift_eval import GIFTEval, GluonTSPredictor
from timecopilot_gift_eval.utils import DATASETS_WITH_TERMS
```

See the [GIFT-Eval API docs](../api/gift-eval/gift-eval.md) for details.

---

**Full Changelog**: https://github.com/TimeCopilot/timecopilot/compare/v0.0.30...v0.0.31
15 changes: 7 additions & 8 deletions docs/examples/gift-eval.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@
},
"outputs": [],
"source": [
"%pip install \"timecopilot>=0.0.22\""
"%pip install \"timecopilot[gift-eval]>=0.0.22\""
]
},
{
Expand All @@ -64,7 +64,7 @@
"source": [
"## Dataset Setup\n",
"\n",
"TimeCopilot includes built-in [GIFT-Eval integration](https://timecopilot.dev/api/gift-eval/gift-eval/#timecopilot.gift_eval.eval.GIFTEval) for dataset handling:"
"TimeCopilot includes built-in [GIFT-Eval integration](https://timecopilot.dev/api/gift-eval/gift-eval/#timecopilot_gift_eval.eval.GIFTEval) for dataset handling:"
]
},
{
Expand All @@ -76,7 +76,7 @@
},
"outputs": [],
"source": [
"from timecopilot.gift_eval.eval import GIFTEval\n",
"from timecopilot_gift_eval import GIFTEval\n",
"\n",
"# TimeCopilot's built-in GIFT-Eval dataset downloader\n",
"# Handles the complete benchmark dataset with all 97 configurations\n",
Expand Down Expand Up @@ -156,7 +156,7 @@
"id": "M2YcjoDF5NH7"
},
"source": [
"With TimeCopilot you can evaluate any [Forecaster](https://timecopilot.dev/api/models/utils/forecaster/#timecopilot.models.utils.forecaster.Forecaster) in a standardized way using its [GIFT-Eval](https://timecopilot.dev/api/gift-eval/gift-eval/#timecopilot.gift_eval.eval.GIFTEval) integration."
"With TimeCopilot you can evaluate any [Forecaster](https://timecopilot.dev/api/models/utils/forecaster/#timecopilot.models.utils.forecaster.Forecaster) in a standardized way using its [GIFT-Eval](https://timecopilot.dev/api/gift-eval/gift-eval/#timecopilot_gift_eval.eval.GIFTEval) integration."
]
},
{
Expand All @@ -168,8 +168,7 @@
"outputs": [],
"source": [
"import pandas as pd\n",
"from timecopilot.gift_eval.eval import GIFTEval\n",
"from timecopilot.gift_eval.gluonts_predictor import GluonTSPredictor\n",
"from timecopilot_gift_eval import GIFTEval, GluonTSPredictor\n",
"\n",
"\n",
"def evaluate_forecaster(\n",
Expand Down Expand Up @@ -710,7 +709,7 @@
},
"outputs": [],
"source": [
"from timecopilot.gift_eval.utils import DATASETS_WITH_TERMS"
"from timecopilot_gift_eval.utils import DATASETS_WITH_TERMS"
]
},
{
Expand Down Expand Up @@ -789,7 +788,7 @@
"id": "g0-oVisu6XX3"
},
"source": [
"The TimeCopilot's [GIFT-Eval integration](https://timecopilot.dev/api/gift-eval/gift-eval/#timecopilot.gift_eval.eval.GIFTEval) was designed considering reproducibility as one of its main features. The library can replicate the official results provided by the mantainers of the benchmark for the [`SeasonalNaive`](https://huggingface.co/spaces/Salesforce/GIFT-Eval/tree/main/results/seasonal_naive) method. The following code replicates the Seasonal Naive performance for the datasets evaluated in this notebook. The reproducibility of the results for the rest of the datasets are tested continuously in the [library's repo](https://github.com/TimeCopilot/timecopilot/blob/main/tests/gift_eval/test_evaluation.py)."
"The TimeCopilot's [GIFT-Eval integration](https://timecopilot.dev/api/gift-eval/gift-eval/#timecopilot_gift_eval.eval.GIFTEval) was designed considering reproducibility as one of its main features. The library can replicate the official results provided by the mantainers of the benchmark for the [`SeasonalNaive`](https://huggingface.co/spaces/Salesforce/GIFT-Eval/tree/main/results/seasonal_naive) method. The following code replicates the Seasonal Naive performance for the datasets evaluated in this notebook. The reproducibility of the results for the rest of the datasets are tested continuously in the [library's repo](https://github.com/TimeCopilot/timecopilot/blob/main/tests/gift_eval/test_evaluation.py)."
]
},
{
Expand Down
3 changes: 2 additions & 1 deletion experiments/gift-eval/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,14 @@
dependencies = [
"modal>=1.0.5",
"s3fs>=2023.12.1",
"timecopilot-gift-eval>=0.3.0",
"timecopilot>=0.0.21",
"transformers<4.54",
"transformers==4.40.1 ; python_full_version < '3.12'",
"typer>=0.16.0",
]
description = "TimeCopilot experiments for GIFT-Eval"
name = "timecopilot-gift-eval"
name = "timecopilot-gift-eval-experiments"
readme = "README.md"
requires-python = ">=3.11"
version = "0.2.0"
3 changes: 1 addition & 2 deletions experiments/gift-eval/src/download_results.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,7 @@
from pathlib import Path

import pandas as pd

from timecopilot.gift_eval.utils import DATASETS_WITH_TERMS
from timecopilot_gift_eval.utils import DATASETS_WITH_TERMS

logging.basicConfig(level=logging.INFO)

Expand Down
3 changes: 1 addition & 2 deletions experiments/gift-eval/src/run_modal.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,7 @@ def main():
import logging

import fsspec

from timecopilot.gift_eval.utils import DATASETS_WITH_TERMS
from timecopilot_gift_eval.utils import DATASETS_WITH_TERMS

logging.basicConfig(level=logging.INFO)

Expand Down
3 changes: 1 addition & 2 deletions experiments/gift-eval/src/run_timecopilot.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,8 @@
from typing import Annotated

import typer
from timecopilot_gift_eval import GIFTEval, GluonTSPredictor

from timecopilot.gift_eval.eval import GIFTEval
from timecopilot.gift_eval.gluonts_predictor import GluonTSPredictor
from timecopilot.models.ensembles.median import MedianEnsemble
from timecopilot.models.foundation.chronos import Chronos
from timecopilot.models.foundation.timesfm import TimesFM
Expand Down
Loading
Loading