A real-world debugging story in Microsoft Fabric — and what Microsoft just shipped to solve it natively.

This morning I ran into a frustrating issue that I suspect many Fabric users have hit without fully understanding why. Our Power BI semantic model refreshed successfully — green tick, no errors — but the data was from yesterday. The pipeline had run, the files were there, everything looked fine. So what went wrong?

Let me walk you through the diagnosis, the solution we built today, and — importantly — what Microsoft has just released that changes the game going forward.

The setup

We’re running a Microsoft Fabric lakehouse architecture:

  • Spark notebooks write Delta tables to a Gold lakehouse
  • A SQL Analytics Endpoint sits on top of the lakehouse
  • A Power BI semantic model reads from that endpoint in Import mode
  • Everything is orchestrated via a Fabric pipeline

The pipeline runs daily, writes data, refreshes the SQL endpoint, then triggers the semantic model refresh. Simple enough.

What happened this morning

Looking at the lakehouse file view, the parquet files for ft_sales had landed at 4:18 AM. The SQL endpoint refreshMetadata API was called at 4:20 AM — just 2 minutes later.

But here’s the smoking gun: the metadata folder inside ft_sales showed a last-modified time of 7:40 AM — nearly 3 hours after the files landed.

The SQL endpoint refresh ran before the Delta table transaction log had fully committed. The API saw no changes, returned "NotRun" for ft_sales, and the semantic model refreshed against stale data.

Understanding the root cause

The Microsoft Fabric SQL Analytics Endpoint sits on top of the Delta Lake transaction log, not the raw parquet files. When Spark writes a Delta table, the process is:

  1. Parquet files written to storage
  2. Delta transaction log (_delta_log) updated ← this can lag
  3. Metadata folder updated ← confirmation of commit
  4. SQL Endpoint detects change via Delta log
  5. Semantic model sees new data

Calling refreshMetadata between steps 1 and 2 means the endpoint sees nothing new. It’s not a bug — it’s a race condition caused by firing the refresh too eagerly.

There’s also a second problem: there was no mechanism to confirm the SQL endpoint had actually finished syncing before triggering the semantic model refresh. The API returns 202 Accepted asynchronously, but our code was moving on immediately.

Solution 1 — the robust notebook approach (production-ready today)

After debugging this end-to-end, here’s the correct sequence:

Spark writes Delta tables
   ↓
1. Poll _delta_log until commit files (.json) appear
   ↓
2. POST /sqlEndpoints/{id}/refreshMetadata  (with timeout body)
   ↓
3. Read per-table sync status from response
   ↓
4. Safe to trigger semantic model refresh

Here’s the complete notebook, designed to work both standalone and called from a pipeline with parameters.

Resolve which tables to check:

import time
from datetime import datetime, timezone

lakehouse_name = f"{workspace_gold}_LH"
base_path = f"abfss://{workspace_gold}@onelake.dfs.fabric.microsoft.com/{lakehouse_name}.Lakehouse/Tables"

print(f"Base path: {base_path}")

if tables_written_param and tables_written_param.strip():
    # Specific tables passed from pipeline
    tables_to_check = [t.strip() for t in tables_written_param.split(",")]
    print(f"Checking specific tables: {tables_to_check}")
else:
    # Scan all Delta tables in the lakehouse
    print("No tables specified — scanning all Delta tables...")
    tables_to_check = []
    try:
        all_items = mssparkutils.fs.ls(base_path)
        for item in all_items:
            if item.isDir:
                table_name = item.name.rstrip("/")
                delta_log = f"{base_path}/{table_name}/_delta_log"
                try:
                    mssparkutils.fs.ls(delta_log)
                    tables_to_check.append(table_name)
                except:
                    pass
    except Exception as e:
        print(f"Could not scan lakehouse path: {e}")

    print(f"Found {len(tables_to_check)} Delta tables")

Wait until Delta logs are committed — this is the critical step that was missing before:

MAX_WAIT = 300   # 5 minutes
INTERVAL = 15    # poll every 15 seconds

print("Waiting for Delta commits to settle...\n")

for table_name in tables_to_check:
    delta_log = f"{base_path}/{table_name}/_delta_log"
    confirmed = False
    elapsed = 0

    while elapsed < MAX_WAIT:
        try:
            log_files = [
                f for f in mssparkutils.fs.ls(delta_log)
                if f.name.endswith(".json")
            ]
            if log_files:
                latest_ts = max(f.modifyTime for f in log_files)
                last_commit = datetime.fromtimestamp(latest_ts / 1000, tz=timezone.utc)
                print(f"{table_name} — last commit: {last_commit}")
                confirmed = True
                break
            else:
                print(f"[{elapsed}s] {table_name} — no commits yet, retrying...")
        except Exception:
            print(f"[{elapsed}s] {table_name} — not ready, retrying...")

        time.sleep(INTERVAL)
        elapsed += INTERVAL

    if not confirmed:
        raise TimeoutError(f"{table_name} Delta log never settled after {MAX_WAIT}s")

print("\nAll Delta tables committed.")

Refresh SQL Endpoint and read results:

import requests
import json

token = mssparkutils.credentials.getToken("https://api.fabric.microsoft.com")
headers = {
    "Authorization": f"Bearer {token}",
    "Content-Type": "application/json"
}

sql_url = (
    f"https://api.fabric.microsoft.com/v1/workspaces/"
    f"{workspace_id}/sqlEndpoints/{sql_endpoint_id}/refreshMetadata"
)

# Pass a timeout — the API waits synchronously up to this limit
json_body = {
    "timeout": {
        "timeUnit": "Minutes",
        "value": 5
    }
}

print("Refreshing SQL Endpoint...")
response = requests.post(sql_url, headers=headers, json=json_body)
result = response.json()

print(f"Status code: {response.status_code}\n")

# Per-table results
tables = result.get("tables", [])
if tables:
    for t in tables:
        name = t.get("tableName", "unknown")
        status = t.get("status", "unknown")
        error = t.get("error", {}).get("message", "") if status == "Failure" else ""
        print(f"{name}: {status} {f'— {error}' if error else ''}")
else:
    print("No table sync info returned.")

print("\nSQL Endpoint refresh complete — safe to trigger semantic model refresh")

Solution 2 — what Microsoft just shipped (the native path)

While we were debugging this, Microsoft has been working on the exact same problem. Two announcements are directly relevant:

New metadata sync — preview since May 2026

This is the most impactful change. Microsoft released a new metadata sync architecture for the SQL Analytics Endpoint that keeps data available for querying within seconds of it landing in the lakehouse — using a decoupled architecture that detects schema and data changes separately, and triggers an on-demand refresh automatically when an incoming query detects stale data.

In other words: the 3-hour lag we experienced this morning would essentially disappear.

How to enable it:

  1. Go to your Fabric workspace
  2. Open Workspace settings
  3. Navigate to Warehouse settings
  4. Enable New metadata sync (preview)

Important: this only applies to new SQL analytics endpoints created after enabling the setting. Your existing endpoints are not automatically migrated.

Once enabled, you also get a new T-SQL stored procedure to refresh a specific table on demand, without touching the full endpoint:

EXEC sys.sp_dw_refresh_ext_table 'dbo.ft_sales';

And a DMV to check the actual sync status per table:

SELECT * FROM sys.dm_db_external_tables_log_status;
-- Returns: last_update_time_utc, latest_log_version, is_blocked

Native “Refresh SQL Endpoint” pipeline activity — June 2026

Also just released: a first-class Refresh SQL Endpoint activity in Fabric pipelines, part of the new Lakehouse Utility Suite. It replaces the notebook-based API call entirely — no more requests.post, no more manual polling.

How to set it up in your pipeline:

  1. Open your Fabric pipeline
  2. In the Activities pane, search for Refresh SQL Endpoint
  3. Drag it onto the canvas after your Spark notebook activity
  4. In Settings, select your workspace and lakehouse
  5. Connect it with a success dependency from your notebook

What makes it smarter than the raw API call: Fabric automatically determines whether a full refresh, incremental refresh, or no refresh is needed based on actual changes in the underlying data — reducing unnecessary compute.

Your pipeline then looks like this:

Spark notebook (writes Delta tables)
   ↓ [on success]
Refresh SQL Endpoint activity  ← native, no code needed
   ↓ [on success]
Semantic model refresh

Which solution should you use?

  Notebook approach New metadata sync Native pipeline activity
Available today Yes Preview Yes
Requires code Yes No No
Per-table visibility Full Via DMV Partial
Handles lag automatically Via polling Yes, seconds Yes
Works on existing endpoints Yes New only Yes
Production-ready Yes Preview Yes

My recommendation:

  • Right now: use the notebook approach from Solution 1 — it’s battle-tested and gives you full visibility.
  • Short term: replace the refreshMetadata API call in the notebook with the native pipeline activity.
  • When new metadata sync goes GA: enable it on new lakehouses and simplify the whole chain significantly.

Key lessons

  1. The refreshMetadata API is not magic. It detects changes based on the Delta transaction log, not file timestamps. If the log hasn’t committed when you call it, it returns "NotRun" silently.
  2. Polling _delta_log is the right signal. Checking for .json commit files via mssparkutils.fs.ls() confirms a Delta write is fully committed before proceeding.
  3. The timeout body matters. Without it, refreshMetadata may return before the sync completes. Pass {"timeout": {"timeUnit": "Minutes", "value": 5}} to make it wait synchronously.
  4. Read the per-table response. The API returns "Success", "NotRun", or "Failure" per table. This is your audit trail.
  5. Make it parametric. Pass specific table names from the pipeline when you know what changed, or let the notebook auto-scan the full lakehouse.
  6. Microsoft is closing this gap natively. The new metadata sync and native pipeline activity are the long-term answer. Keep an eye on GA dates.

If you’re running Fabric pipelines with Import mode semantic models and have ever had a “successful” refresh that showed yesterday’s data — this is probably why. And now you have both the fix for today and the roadmap for tomorrow.