How to make an ERCOT data pipeline survive 400s, 429s, 500s and timeouts

Retry logic is not for i in range(3).

A production market-data collector should respond differently to authentication failures, invalid requests, throttling, temporary upstream failures and incomplete datasets. Treating every error as “sleep five seconds and try again” can make an incident worse — for you, and for everyone else sharing the API.

Where this guide sits in the pipeline

1. Source
2. Fetch
3. Raw copy
4. Validate
5. Verify completeness
6. Normalize
7. Promote
8. Client system

ERCOT failures don't all look alike

ERCOT's public API discussions show several distinct failure modes over the past couple of years:

• Timeouts and 500s while sign-in still worked. In January 2026, users reported data requests hanging for minutes before timing out or returning HTTP 500, even though tokens could still be obtained normally (Discussion #124). ERCOT staff attributed the degradation to usage patterns from a small number of accounts, including excessive and repetitive polling.
• Throttling. Since March 2026, requests beyond ERCOT's bandwidth-based limits receive HTTP 429 (Discussion #132).
• 400s caused upstream. In late May 2026, batch archive downloads began failing with 400 errors across products. ERCOT staff traced the problem to an API update and deployed a fix on June 2 (Discussion #141).
• 403s tied to network origin or client behavior (Discussion #85, Discussion #96, Discussion #136) — covered in why a collector can work locally but fail in the cloud.

Each calls for a different response, and a generic retry loop gets most of them wrong.

A default response for each failure

These are defaults, not rules: a status code narrows down the likely cause, and your logs and the upstream's announcements confirm it. Two rows deserve a closer look.

401. ERCOT tokens last one hour and can't be refreshed (see the Python guide), so an occasional 401 on a long run is expected: get a new token and carry on. A steady stream of 401s is not expected, and retrying won't fix it.

400. In the May 2026 case the requests were valid and the API was at fault — but retrying still wouldn't have helped, because the fix needed a deployment on ERCOT's side, several days later. The right behavior was to stop, alert and resume once fixed. A pipeline that retried every 400 would have spent those days hammering an endpoint that couldn't succeed.

Classify before you retry

import requests


class Retryable(Exception): ...   # worth another attempt after a delay
class Throttled(Exception): ...   # back off and resume in a later run
class Fatal(Exception): ...       # stop and alert; a person needs to look


def check(response: requests.Response) -> None:
    if response.status_code == 429:
        raise Throttled(response.headers.get("Retry-After"))
    if response.status_code in (500, 502, 503, 504):
        raise Retryable(response.status_code)
    if response.status_code >= 400:
        raise Fatal(response.status_code)  # 400, 403, and a 401 that survived a token refresh

Exponential backoff

For failures that are probably temporary, wait longer after each attempt — 2, 4, 8, 16 seconds, up to a ceiling. That gives an overloaded service room to recover instead of adding load at the worst possible moment.

Add jitter

If twenty workers fail at the same moment and back off on exactly the same schedule, they retry at the same moment too. Randomizing the delay spreads them out:

import random


def backoff_delay(attempt: int, base: float = 2.0, cap: float = 300.0) -> float:
    """Full jitter: a random delay between 0 and the exponential ceiling."""
    return random.uniform(0, min(cap, base * 2 ** attempt))

This matters more than it looks. The January 2026 slowdown was attributed to a small number of accounts generating heavy, repetitive load; synchronized retries from many clients produce the same kind of load.

Set a retry budget

Don't retry forever. Give each unit of work a maximum number of attempts, or a maximum total time, and give the whole run a ceiling as well. When the budget is spent, the work is recorded as failed and the run moves on or stops. A collector that retries one window for six hours has stopped collecting everything else.

Retry the unit of work, not the whole job

Retries only work if the collector knows exactly what failed. Persist the state of each unit — a dataset and a time window — so a retry repeats that unit and nothing else, and a restarted process picks up where it stopped. Building under rate limits shows a window table for this.

import time

MAX_ATTEMPTS = 5


def run_window(window) -> None:
    for attempt in range(MAX_ATTEMPTS):
        try:
            payload = fetch_window(window)   # calls check() on every response
            save_raw(payload, window)
            mark_complete(window, rows=count_rows(payload))
            return
        except (Retryable, requests.Timeout) as exc:
            record_attempt(window, attempt, exc)
            time.sleep(backoff_delay(attempt))
        except Throttled:
            mark_pending(window)             # resume in a later run
            raise                            # stop this run; don't push through a 429
    quarantine(window)                       # budget spent: a person looks at it

Fatal isn't caught here on purpose: it stops the run and raises an alert.

Quarantine what keeps failing

When a window exhausts its budget, move it to a failed-window queue with the last error, the attempt count and the time. That queue is what a person reviews, and what a scheduled job retries later — after the upstream incident, not during it. It turns “some data is probably missing” into a specific list.

Validate successful responses too

A 200 is the one response a naive pipeline never questions. It can still carry an empty page, a partial window, or data the upstream hasn't finished processing — ERCOT's March 2026 processing issue produced exactly that (Discussion #133). Every successful window still goes through the completeness check in detecting missing ERCOT market data.

The shape of the whole pipeline

1. Fetch
2. Raw store
3. Validate
4. Check completeness
5. Retry gaps
6. Promote

Retries happen at two levels: the transport level, inside a single fetch, and the data level, where the completeness check finds gaps and schedules targeted re-fetches. A reliable pipeline needs both.

Practical takeaway

• ✓Classify every response before deciding whether to retry.
• ✓Back off exponentially, with jitter, and honor Retry-After.
• ✓Stop on 400 and 403; refresh once on 401; slow down on 429.
• ✓Give every unit of work a retry budget, and quarantine what exceeds it.
• ✓Persist state per window so retries and restarts repeat only what failed.
• ✓Validate successful responses as carefully as failed ones.