Source Stream Settings

Configure incremental scanning with timestamp tracking for efficient delta updates.

Last Scanned Timestamp (SOURCE_STREAM_LAST_SCANNED_TIMESTAMP)

Last timestamp scanned from this data source. Used as part of SCAN_WHERE_CLAUSE for incremental scanning.

How it works:

• Stored after each successful scan
• Automatically updated when scans complete
• Used to filter records in subsequent scans
• Enables delta/incremental scanning

Incremental scanning:

• Set SCAN_WHERE_CLAUSE to include timestamp filter (e.g., WHERE created_date > '<last_scanned_timestamp>')
• Scans only new or changed records
• Faster than full scans
• Schema discovery still runs for new data

Resetting:

• Remove timestamp from configuration to scan entire data source
• Useful for full re-scanning after schema changes
• Can be cleared manually in configuration