How to detect missing ERCOT market data before it reaches your models

Successful HTTP requests do not prove that you have a complete ERCOT dataset.

Your collector can receive 200 OK, parse valid JSON, insert every returned row into PostgreSQL and still have an incomplete operating day. With power-market data that matters a great deal, because a handful of missing intervals disappears easily inside millions of otherwise valid records.

A production pipeline therefore needs a second job after collection: prove that the data you expected actually arrived.

Where this guide sits in the pipeline

1. Source
2. Fetch
3. Raw copy
4. Validate
5. Verify completeness
6. Normalize
7. Promote
8. Client system

What happened in March 2026

In Discussion #133, opened on March 9, 2026, community members reported missing intervals in NP6-788-CD (LMPs by resource nodes, load zones and trading hubs), and others added gaps in related products including NP6-346-CD, NP4-190-CD and NP6-905-CD. A separate community report, Discussion #134, described missing Day-Ahead Market settlement point prices and ancillary service clearing prices for March 7.

On March 11, ERCOT staff replied that issues encountered on March 6 had prevented some of these files from being processed. The affected files had been queued automatically and were being processed along with current report artifacts, and intervals for March 6, 7 and 9 should by then be present for NP6-346-CD, NP6-788-CD and NP4-190-CD. NP6-905-CD was still processing at the time of that reply.

The data wasn't lost — it arrived late, after an upstream processing problem. But a collector that ran on March 9, received 200 responses and moved on would have recorded incomplete days as complete, and nothing at the HTTP layer would have said otherwise. Catching that is the collector's job, not the API's.

Transport success is not data completeness

HTTP 200 means the request was valid and the service answered. It says nothing about whether the answer contains every interval, every settlement point or every hour you needed. An empty page, a short page and a full page all pass raise_for_status().

So completeness needs its own definition, written down per dataset, and its own check that runs after collection.

Define the expected grain for each dataset

ERCOT products don't share one grain, so “complete” has to be defined product by product:

The SCED row is the one that catches people out. SCED results are stamped with the time each run executed — the timestamps quoted in the March 2026 reports look like 00:00:18, not 00:00:00 — and SCED normally runs about every five minutes rather than exactly on the boundary. An exact count of 288 is the wrong test there. A better one is “no gap between consecutive SCED timestamps longer than N minutes,” plus a sanity range on the daily count.

Daylight saving time needs the same explicit treatment. ERCOT reports hour-ending intervals in local time and flags the repeated hour in November: the DAM price endpoint takes a DSTFlag parameter and the SCED LMP endpoint a repeatHourFlag in ERCOT's API specification. Generate expected intervals with timezone-aware datetimes, so the short and long days come out right instead of being patched by hand.

Generate the expected interval set

For fixed-interval products, generate the expected set from the operating day itself:

from datetime import date, datetime, timedelta, timezone
from zoneinfo import ZoneInfo

CENTRAL = ZoneInfo("America/Chicago")

def expected_intervals(day: date, step: timedelta) -> list[datetime]:
    """Interval starts for one operating day, as UTC datetimes, DST-aware."""
    start = datetime(day.year, day.month, day.day, tzinfo=CENTRAL).astimezone(timezone.utc)
    nxt = day + timedelta(days=1)
    end = datetime(nxt.year, nxt.month, nxt.day, tzinfo=CENTRAL).astimezone(timezone.utc)
    intervals, cursor = [], start
    while cursor < end:
        intervals.append(cursor)
        cursor += step
    return intervals

len(expected_intervals(date(2026, 3, 8), timedelta(minutes=15)))   # 92
len(expected_intervals(date(2026, 11, 1), timedelta(minutes=15)))  # 100

Working in UTC internally and converting at the edges avoids the ambiguous repeated hour entirely. Keep the conversion to ERCOT's local representation in one place, next to the code that reads each endpoint's DST flag.

Compare expected with received

With the expected set defined, the check is a set difference: which expected intervals have no data? Record the result per dataset and operating day, and keep the list of missing intervals, not just the count — the list is what the repair job needs.

Check entity coverage too

For nodal datasets, a complete set of timestamps can still hide missing settlement points. Compare the entities present in each interval — resource nodes, load zones, hubs — against the set you expect for that date.

Take the expected set from a maintained reference list rather than from yesterday's data. Settlement points are added and retired over time, and a node that disappears on purpose shouldn't raise the same alarm as one that vanished from a single interval.

Late is different from missing

Some data arrives after its interval for ordinary reasons. ERCOT staff have described publication lag between ercot.com and the Public API for some products. If the completeness check runs too early, every normal delay becomes an alert, and people learn to ignore alerts.

Give each dataset a lateness tolerance. An interval that hasn't arrived is pending until the tolerance passes, and only then missing. Why ERCOT API data can lag behind ercot.com covers how to set that tolerance.

Retry only the missing range

1. Identify the missing intervals
2. Request only that range
3. Validate what came back
4. Update the ledger

Re-downloading a whole day to fix five intervals spends bandwidth you may need elsewhere — ERCOT has limited Public API usage by data volume since March 2026 (see the rate-limits guide) — and rewrites data you had already verified. Requesting only the gap keeps repairs cheap enough to run automatically. In the March 2026 case, re-checking the affected windows after ERCOT finished processing would have filled the gaps without anyone touching the pipeline.

Maintain a completeness ledger

Keep the results of every check in one place:

dataset
operating_date
expected_intervals
received_intervals
missing_intervals     -- the list, not just a count
first_checked_at
last_checked_at
status                -- pending | incomplete | complete | quarantined

The ledger answers the question models and people actually ask — “can I trust March 9?” — without scanning millions of rows. It also keeps history: when a day was first checked, when it became complete, and how long upstream gaps took to fill. Quarantined is for periods that exhausted their retries and need a person.

Promote only verified periods

1. Raw data lands
2. Validate structure and values
3. Complete for this period?
   • No → retry or quarantine
   • Yes → promote
4. Tables that models and reports read

Keep raw data and production tables separate. Collection writes to a raw layer; only periods the ledger marks complete are promoted to the tables that models, reports and trading tools read. Anything downstream can then trust any row it sees, and an incomplete day stays visibly incomplete instead of blending in.

This is the core of how we build pipelines: each stage has to prove its output before the next stage uses it.

Practical takeaway

• ✓Define “complete” per dataset: grain, expected count, daylight-saving behavior.
• ✓For SCED-based data, test for gaps between runs, not an exact count.
• ✓Check entities as well as timestamps, against a maintained reference list.
• ✓Treat data as pending until its lateness tolerance passes.
• ✓Repair only the missing range, then re-check.
• ✓Promote a period only after the ledger marks it complete.