---
name: ecmwf-open-data-ingestion
description: "How to get ECMWF Open Data (real-time forecasts) data into a pandas DataFrame and Parquet: access, the request, the format, limits, quirks and terms."
---

# Load one ECMWF 2 m temperature field into pandas and Parquet

Source entry: https://signal-lab-jet.vercel.app/datasets/ecmwf-open-data
Provider: https://www.ecmwf.int/en/forecasts/datasets/open-data
Checked against the provider's documentation: 2026-10-30

## Access
- No account or key for ECMWF Open Data.
- Read the JSON-lines .index file for a run, then download only the bytes of the field you need with an HTTP Range request.

## Request
```bash
curl 'https://data.ecmwf.int/forecasts/20261029/00z/ifs/0p25/oper/20261029000000-24h-oper-fc.index'
```

## Format
GRIB2 forecast fields plus JSON-lines .index sidecars that give each field's byte offset and length. The snippet downloads one global 2 m temperature field at forecast step 24 with a Range request, decodes it with cfgrib/xarray, then writes a compact 10-degree box to Parquet.

## Load into pandas and Parquet
Needs Python 3 with requests, pandas and pyarrow.
```python
"""ECMWF Open Data 2 m temperature, step 24 -> pandas -> Parquet. No key."""
import json
from datetime import datetime, timedelta, timezone
import requests
import pandas as pd
import xarray as xr  # GRIB2 decoding needs: pip install xarray cfgrib (cfgrib brings eccodes)

# The portal keeps only the most recent 12 runs, so use yesterday's 00 UTC run.
run = (datetime.now(timezone.utc) - timedelta(days=1)).strftime("%Y%m%d")
base = f"https://data.ecmwf.int/forecasts/{run}/00z/ifs/0p25/oper/{run}000000-24h-oper-fc"
index = requests.get(base + ".index", timeout=60)
index.raise_for_status()
field = next(f for f in map(json.loads, index.text.splitlines()) if f["param"] == "2t")
start, end = field["_offset"], field["_offset"] + field["_length"] - 1
grib = requests.get(base + ".grib2", headers={"Range": f"bytes={start}-{end}"}, timeout=120)
grib.raise_for_status()
with open("ecmwf_2t_step24.grib2", "wb") as f:
    f.write(grib.content)
ds = xr.open_dataset("ecmwf_2t_step24.grib2", engine="cfgrib", backend_kwargs={"indexpath": ""})
# Keep a compact 10-degree box after decoding the single global field.
box = ds.sel(latitude=slice(55, 45), longitude=slice(5, 15))
df = box[["t2m"]].to_dataframe().reset_index()
df["t2m_c"] = df.pop("t2m") - 273.15
df.to_parquet("ecmwf_2t_step24_box.parquet", index=False)
print(len(df), "grid points", "run", str(df["time"].iloc[0]), "valid", str(df["valid_time"].iloc[0]), f"{len(grib.content) // 1024} KB")
```
Run by us on 2026-10-30 (python 3.13 | pandas | pyarrow | requests | xarray 2026.9.0 | cfgrib 0.9.15.1). Result: 1681 grid points run 2026-10-29 00:00:00 valid 2026-10-30 00:00:00 307 KB

## Limits
- Open Data portal: between 500 and 2,000 simultaneous connections.
- Rate limit: Not stated.
- The rolling archive keeps the most recent 12 forecast runs, about 2–3 days across four daily cycles.

## Quirks
- GRIB2 stores run time, forecast step and valid time separately. Retain all three when building point-in-time features.
- Each .index line gives a field's _offset and _length, so one field can be fetched with a byte range. A complete run is much larger than this one-field example.
- Observed 2026-10-30: the 2 m temperature field at step 24 was 307 KB; cfgrib and eccodes decoded it. The official ecmwf-opendata Python client does the same index and byte-range selection.

## Terms and attribution
- ECMWF Open Data are CC BY 4.0 and may be redistributed and used commercially with appropriate attribution.
- Attribute ECMWF and comply with the linked ECMWF Terms of Use.
- Terms page: https://www.ecmwf.int/en/forecasts/datasets/open-data

## Sources
- https://www.ecmwf.int/en/forecasts/datasets/open-data: "Products are provided in GRIB2 format unless stated otherwise."
- https://www.ecmwf.int/en/forecasts/datasets/open-data: "access to the Open-Data Portal is currently limited to between 500 and 2000 simultaneous connections"
- https://www.ecmwf.int/en/forecasts/datasets/open-data: "the data may be redistributed and used commercially"
- https://raw.githubusercontent.com/ecmwf/ecmwf-opendata/master/README.md: "This package is designed for users that want to download a subset of the whole dataset."

Information for educational purposes only. Free tiers, limits and licenses belong to the providers and may change; check the provider's own terms before using the data.