Building a reliable ERCOT API pipeline under rate limits

A script that can download ERCOT data is not necessarily a production data pipeline.

That distinction became very clear in March 2026, when ERCOT added bandwidth-based limits to its Public API. Developers paginating through large LMP datasets started receiving HTTP 429 “Quota Exceeded” responses, and the discussion that followed became one of the busiest threads on ERCOT's API board.

The lesson is larger than one policy change. A production collector should not assume that yesterday's allowable request volume, pagination strategy or API behavior will stay the same. It needs to control how much data it asks for, know exactly what it has already received, resume after an interruption, and prove that the final dataset is complete.

Where this guide sits in the pipeline

1. Source
2. Fetch
3. Raw copy
4. Validate
5. Verify completeness
6. Normalize
7. Promote
8. Client system

What ERCOT said

In a staff reply on March 9, 2026, ERCOT explained that bandwidth-based limits had gone into effect on March 6, in addition to the existing request-based limits. ERCOT said roughly 10% of users had been generating more than 80% of total system load, that the limits are based on data volume rather than request count alone, and that rate-limited requests receive HTTP 429.

ERCOT also said it was not publishing specific thresholds, and that it needs the flexibility to adjust its policies as usage patterns and system capacity evolve. In a follow-up on March 11, ERCOT noted that the 429 error message shows there are two bandwidth-based limits — one hourly and one daily — and advised requesting smaller chunks of data more frequently: an hour, or two, or four at a time, several times a day, instead of a full day of LMP data at once.

Community members in the same thread posted the specific values they saw in error messages. We deliberately don't repeat them. ERCOT hasn't published them as policy, and a pipeline that hardcodes a number it read in an error message will break quietly the next time the number changes.

The design constraint: ERCOT added bandwidth-based limits in March 2026, but developers should not architect around a hardcoded limit unless ERCOT documents one officially. Build a collector that adapts instead.

A separate community thread, Discussion #135, asked ERCOT to reconsider the change and described teams reallocating work and missing delivery commitments. That is one group of users' view, not ERCOT's — but it shows what an upstream policy change costs a pipeline that wasn't built to adapt to one.

Request limits and bandwidth limits are different

A request-count limit asks how often you are calling the service. A bandwidth limit asks how much data you are consuming. They fail in different ways.

A collector that makes a few hundred requests an hour can sit comfortably inside a request limit and still exceed a bandwidth limit, if each request pages through a large nodal dataset. Slowing the request rate — the usual fix for request limits — doesn't help when every request is still enormous. What helps is asking for less data per unit of time.

Avoid the giant daily query

The fragile pattern looks like this:

1. Yesterday
2. One query for the whole day
3. Millions of rows
4. Hundreds of pages
5. HTTP 429 partway through

When that query fails on page 140, the collector has two bad options: start the day again from page one and spend the budget a second time, or keep a partial day it can't easily describe. Pages are also a weak checkpoint, because page boundaries depend on the query, the sort order and the data that existed at the moment you asked.

The sturdier pattern splits time into small windows and treats each one as a unit of work:

00:00–02:00
02:00–04:00
04:00–06:00
...
22:00–24:00

Keep the window size configurable per dataset. Two hours is not a magic number: a high-volume nodal price product may need smaller windows than a system-wide report, and the right size can change when ERCOT changes its limits. Being able to halve the window without a code change is worth more than finding the perfect value today.

from datetime import datetime, timedelta

def windows(start: datetime, end: datetime, size: timedelta):
    """Split [start, end) into consecutive windows of at most `size`."""
    cursor = start
    while cursor < end:
        yield cursor, min(cursor + size, end)
        cursor += size

Generate windows from timezone-aware datetimes and convert to the format the endpoint expects at the last moment. ERCOT's local time has a repeated hour every November, and endpoints represent it explicitly — the SCED LMP endpoint, for example, has a repeatHourFlag parameter in ERCOT's API specification.

Make every ingestion window deterministic

Record each window before it runs, with enough state to answer “what do we have?” without reading the data itself:

CREATE TABLE ingestion_window (
    dataset        text        NOT NULL,
    window_start   timestamptz NOT NULL,
    window_end     timestamptz NOT NULL,
    status         text        NOT NULL DEFAULT 'pending',  -- pending | complete | failed
    attempts       integer     NOT NULL DEFAULT 0,
    rows_received  integer,
    completed_at   timestamptz,
    PRIMARY KEY (dataset, window_start)
);

Two runs over the same dataset and window should ask for exactly the same thing. That makes retries safe, makes gaps easy to find, and makes the pipeline auditable: for any hour of any day you can say when it was collected, how many rows arrived and how many attempts it took.

Make downloads resumable

If ten windows have completed and the eleventh fails, the next run should start at window eleven, not window one:

SELECT dataset, window_start, window_end
FROM ingestion_window
WHERE dataset = %(dataset)s
  AND status <> 'complete'
ORDER BY window_start;

Resumability is also what makes a 429 cheap. When the collector is throttled, it records where it stopped and ends the run. Nothing already collected is requested again, so no bandwidth you've already used is spent twice.

Treat 429 as flow control

A 429 is the API telling you to slow down. Retrying immediately at full speed spends more of the budget you just exhausted and keeps the collector pinned against the limit. Instead:

• Back off before retrying, with jitter so parallel workers don't retry in lockstep. If a response includes a Retry-After header, honor it.
• Reduce the workload, not just the rate: shrink the window size for that dataset and let the scheduler spread the remaining windows out.
• Persist the checkpoint before waiting, so a restart doesn't lose your place.
• Schedule instead of spinning. With an hourly and a daily limit in play, the right response to a 429 may be to stop and resume in the next period.

Other failures need different handling — a 400, a 401, a 500 and a timeout each mean something different. Surviving ERCOT API failures covers the full set.

Separate live ingestion from backfill

Live collection and historical reconstruction compete for the same limits, and they have different priorities. A backfill of last year's nodal prices should never be the reason today's intervals arrive late. Give backfills their own throttle and a lower priority, and pause them when live collection falls behind.

Don't look for extra headroom by spreading work across more accounts. When ERCOT described degraded API performance in January 2026, staff cited attempts to bypass usage limits among the causes (Discussion #124). Historical backfills covers how to run the two side by side.

Validate after collection

A completed HTTP session doesn't prove a complete dataset. A window can return 200 with fewer rows than it should, or with none at all if the upstream data hasn't been processed yet. Completeness is a separate check with its own record — see detecting missing ERCOT market data.

Practical takeaway

• ✓Don't hardcode bandwidth thresholds you saw in an error message.
• ✓Split collection into small, configurable time windows.
• ✓Record every window's state before and after it runs.
• ✓Resume from the first incomplete window after any failure.
• ✓Treat 429 as a signal to back off and shrink the workload.
• ✓Budget backfills separately from live collection.
• ✓Verify completeness after collection, not just HTTP success.