Source Stream Settings

Configure incremental scanning with timestamp tracking for efficient delta updates.

Last Scanned Timestamp (SOURCE_STREAM_LAST_SCANNED_TIMESTAMP)

Last timestamp scanned from this data source. Used as part of SCAN_WHERE_CLAUSE for incremental scanning.

How it works:

• Stored after each successful scan

• Automatically updated when scans complete

• Used to filter records in subsequent scans

• Enables delta/incremental scanning

Incremental scanning:

• Set SCAN_WHERE_CLAUSE to include timestamp filter (e.g., WHERE created_date > '<last_scanned_timestamp>')

• Scans only new or changed records

• Faster than full scans

• Schema discovery still runs for new data

Resetting:

• Remove timestamp from configuration to scan entire data source

• Useful for full re-scanning after schema changes

• Can be cleared manually in configuration