Incremental Migrations in Drupal: Using high_water_property Properly

A migration that runs once, during a rebuild, gets the brute-force treatment: replay everything, read the report, start over. A migration that runs every day against a live source database is a different job entirely. Replaying 200,000 rows every night to actually import 300 changed ones costs CPU, costs I/O on the source database, and above all eats an execution window that eventually overflows.

The mechanism the Migrate API provides for this is called high_water_property.

What the high water mark actually does

The principle: the migration remembers the highest value of the tracking field (typically a changed / updated_at) among the rows already processed. On the next run, it only picks up beyond that value.

Concretely, on the SqlBase side:

1. The source plugin adds a field > remembered_value condition to the query.
2. It adds an ORDER BY on that field — the marker's progression only makes sense if rows arrive in ascending order.
3. The marker is saved as it goes, row by row, not at the end of the run. A migration interrupted at 60% therefore doesn't start over from zero.

The value is not stored in the map table, but in the key/value store, collection migrate:high_water, indexed by migration ID. It is information held outside the map: that's what explains several of the surprises listed below.

One point that is often misunderstood: the high water condition is not exclusive. It is combined with OR against rows absent from the map table and those flagged STATUS_NEEDS_UPDATE. A row that was never imported therefore still gets imported, even if its changed is old.

Configuration

id: article_incremental
label: 'Articles — incremental import'
source:
  plugin: article_source
  high_water_property:
    name: changed

If the source query has joins and the column name is ambiguous, specify the table alias:

  high_water_property:
    name: changed
    alias: n

On the source plugin side, there is nothing to wire up: SqlBase applies the condition and the sort from this configuration. One thing remains to be done, outside Drupal:

CREATE INDEX idx_node_changed ON node (changed);

Without an index on the high water field, you replace a full scan with a full scan plus a sort. That's the first gain to lock in.

What it is not

track_changes is the direct competitor. It detects more things — including a change in a joined table that doesn't bump the node's changed — but it pays for a full read of the source on every run. Simple rule: high_water_property when the source exposes a reliable modification date, track_changes when it doesn't.

The pitfalls

1. Be careful combining it with --sync. --sync infers deletions by comparing the list of source IDs to the map. But high water is precisely what restricts the source query. The returned list becomes partial, and anything that doesn't show up can be treated as deleted. Test this on a copy before considering the two together — not in production on a Friday evening.

2. Deletions are not detected. High water only sees what moves "upward". Content deleted or unpublished on the source side never surfaces. It needs separate handling: a soft delete on the source side with an updated changed, or a dedicated periodic synchronization pass.

3. The comparison is strict (>), not >=. If several rows share the same changed value down to the second and the run stops in the middle of that batch, the rest of the batch is lost on the next run. On high-volume sources, plan a safety margin (restart from high_water - 60s) rather than trusting second-level granularity.

4. The marker advances even on skipped rows. A row discarded by a MigrateSkipRowException in prepareRow() doesn't block progression. If the reason for the skip disappears later (a reference value finally present), the row will not be picked up again: its changed is now below the marker.

5. Non-SQL sources: a functional gain, not a performance gain. For a JSON or CSV source via migrate_plus, the filtering happens in PHP in SourcePluginBase::next(), after fetching. You avoid useless writes to the destination, not the download or the parsing of the full feed.

6. The field must be monotonic. A changed that the source rewrites downward, or a reused auto-incremented ID, breaks the mechanism silently. Nothing is reported as an error: rows simply stop being imported.

7. Rollback and high water are two distinct things. The map table is purged by the rollback, the key/value not necessarily. A migration that is rolled back and then relaunched can therefore reimport nothing at all. It's the classic symptom of "my migration doesn't do anything anymore".

Inspecting and resetting the marker

# Read the current value
drush php:eval 'var_dump(\Drupal::keyValue("migrate:high_water")->get("article_incremental"));'

# Reset (next run = full import)
drush php:eval '\Drupal::keyValue("migrate:high_water")->delete("article_incremental");'

# Reposition on a specific date (targeted catch-up)
drush php:eval '\Drupal::keyValue("migrate:high_water")->set("article_incremental", strtotime("2026-08-01"));'

The third case is the real production tool: after an incident, you don't replay everything, you move the marker back over the affected window.

Not to be confused with migrate:reset-status, which unblocks a migration stuck in Importing status after a kill, and has no effect whatsoever on high water.

Checklist before going to production

• The high water field is indexed on the source side.
• It is monotonic and updated by every application write, including batches and third-party imports.
• The deletion case is handled by an explicit, documented mechanism.
• The catch-up procedure (repositioning the marker) is written down somewhere, not held in one person's head.
• The daily run is monitored on the number of rows processed, not just on its exit code: a broken incremental migration succeeds perfectly well at importing zero rows.