A migration that runs once, during a rebuild, gets the brute-force treatment: replay everything, read the report, start over. A migration that runs every day against a live source database is a different job entirely. Replaying 200,000 rows every night to actually import 300 changed ones costs CPU, costs I/O on the source database, and above all eats an execution window that eventually overflows.
\nThe mechanism the Migrate API provides for this is called high_water_property.
The principle: the migration remembers the highest value of the tracking field (typically a changed / updated_at) among the rows already processed. On the next run, it only picks up beyond that value.
Concretely, on the SqlBase side:
field > remembered_value condition to the query.ORDER BY on that field — the marker's progression only makes sense if rows arrive in ascending order.The value is not stored in the map table, but in the key/value store, collection migrate:high_water, indexed by migration ID. It is information held outside the map: that's what explains several of the surprises listed below.
One point that is often misunderstood: the high water condition is not exclusive. It is combined with OR against rows absent from the map table and those flagged STATUS_NEEDS_UPDATE. A row that was never imported therefore still gets imported, even if its changed is old.
id: article_incremental\nlabel: 'Articles — incremental import'\nsource:\n plugin: article_source\n high_water_property:\n name: changedIf the source query has joins and the column name is ambiguous, specify the table alias:
\n high_water_property:\n name: changed\n alias: nOn the source plugin side, there is nothing to wire up: SqlBase applies the condition and the sort from this configuration. One thing remains to be done, outside Drupal:
CREATE INDEX idx_node_changed ON node (changed);Without an index on the high water field, you replace a full scan with a full scan plus a sort. That's the first gain to lock in.
\n| Mechanism | \nWhat it does | \nCost | \n
|---|---|---|
high_water_property | \nFilters the source query on > last processed value | \nNear zero if the field is indexed | \n
migrate:import --update | \nFlags every already-imported row as \"needs update\" and replays everything | \nMaximal | \n
migrate:import --sync | \nCompares source IDs to map IDs and deletes destinations whose source has disappeared | \nFull read of the source | \n
track_changes: true | \nComputes a hash of each source row and compares it to the one in the map | \nFull read + a hash per row | \n
track_changes is the direct competitor. It detects more things — including a change in a joined table that doesn't bump the node's changed — but it pays for a full read of the source on every run. Simple rule: high_water_property when the source exposes a reliable modification date, track_changes when it doesn't.
1. Be careful combining it with --sync. --sync infers deletions by comparing the list of source IDs to the map. But high water is precisely what restricts the source query. The returned list becomes partial, and anything that doesn't show up can be treated as deleted. Test this on a copy before considering the two together — not in production on a Friday evening.
2. Deletions are not detected. High water only sees what moves \"upward\". Content deleted or unpublished on the source side never surfaces. It needs separate handling: a soft delete on the source side with an updated changed, or a dedicated periodic synchronization pass.
3. The comparison is strict (>), not >=. If several rows share the same changed value down to the second and the run stops in the middle of that batch, the rest of the batch is lost on the next run. On high-volume sources, plan a safety margin (restart from high_water - 60s) rather than trusting second-level granularity.
4. The marker advances even on skipped rows. A row discarded by a MigrateSkipRowException in prepareRow() doesn't block progression. If the reason for the skip disappears later (a reference value finally present), the row will not be picked up again: its changed is now below the marker.
5. Non-SQL sources: a functional gain, not a performance gain. For a JSON or CSV source via migrate_plus, the filtering happens in PHP in SourcePluginBase::next(), after fetching. You avoid useless writes to the destination, not the download or the parsing of the full feed.
6. The field must be monotonic. A changed that the source rewrites downward, or a reused auto-incremented ID, breaks the mechanism silently. Nothing is reported as an error: rows simply stop being imported.
7. Rollback and high water are two distinct things. The map table is purged by the rollback, the key/value not necessarily. A migration that is rolled back and then relaunched can therefore reimport nothing at all. It's the classic symptom of \"my migration doesn't do anything anymore\".
\n# Read the current value\ndrush php:eval 'var_dump(\\Drupal::keyValue(\"migrate:high_water\")->get(\"article_incremental\"));'\n\n# Reset (next run = full import)\ndrush php:eval '\\Drupal::keyValue(\"migrate:high_water\")->delete(\"article_incremental\");'\n\n# Reposition on a specific date (targeted catch-up)\ndrush php:eval '\\Drupal::keyValue(\"migrate:high_water\")->set(\"article_incremental\", strtotime(\"2026-08-01\"));'The third case is the real production tool: after an incident, you don't replay everything, you move the marker back over the affected window.
\nNot to be confused with migrate:reset-status, which unblocks a migration stuck in Importing status after a kill, and has no effect whatsoever on high water.