Schema Summary Views

Views for data source overview, scan history, configurations, and billing information.

VW_DATASOURCES

Purpose: Comprehensive summary of all data sources with configuration and metadata.

Contents:

• Data source identification (ID, name, tag)

• Source connection details (database, schema, object, attribute)

• Data format and type information

• Scan statistics (last scan, attribute counts, complexity metrics)

• Product tier information (feature flags for different tiers)

• Output configuration (database, schema, prefix names, security policies)

• Metadata table configuration

• Gen AI instructions

Use Cases:

• Data source inventory and overview

• Configuration review

• Product tier verification

• Integration workflows

Example Query:

Example

SELECT 
    datasource_name,
    data_format,
    last_scan,
    attribute_count,
    polymorphic_attribute_count,
    product_tier_attribute_metadata_management,
    product_tier_sql_code_generation
FROM core.vw_datasources
WHERE datasource_status = 'active'
ORDER BY datasource_name;

---

VW_DATASOURCE_SCANS

Purpose: Historical scan execution records with performance metrics.

Contents:

• Scan identification (scan ID, data source, configuration name)

• Scan timing (begin, end, last updated, duration)

• Scan status and batch information

• Record counts (processed, source count, limit)

• Performance metrics (records processed per minute)

• Configuration details (WHERE clause, ORDER BY, horizontal scale type)

• Warehouse information (name, reference ID)

• Thread and batch details

Use Cases:

• Scan history review

• Performance analysis

• Troubleshooting scan issues

• Monitoring scan completion

Example Query:

Example

SELECT 
    datasource_name,
    configuration_name,
    scan_datetime_begin,
    scan_duration_minutes,
    records_processed,
    records_processed_per_minute,
    scan_status
FROM core.vw_datasource_scans
WHERE datasource_id = 'your-datasource-id'
ORDER BY scan_datetime_begin DESC;

---

VW_DATASOURCE_SCANS_IN_PROCESS

Purpose: Real-time monitoring of currently executing scans with progress estimates.

Contents:

• Active scan identification

• Progress tracking (records processed, estimated remaining)

• Time estimates (duration, time remaining, percentage complete)

• Performance metrics (records per minute, estimated completion)

• Thread configuration and multipliers

• Batch status and warehouse information

Use Cases:

• Monitoring active scans

• Progress tracking

• Performance optimization

• Resource utilization monitoring

Example Query:

Example

SELECT 
    datasource_name,
    configuration_name,
    scan_status,
    percentage_complete,
    estimated_records_processed,
    estimated_records_remaining,
    process_time_remaining_in_minutes
FROM core.vw_datasource_scans_in_process
ORDER BY scan_datetime_begin DESC;

---

VW_DATASOURCE_SCAN_CONFIGURATIONS

Purpose: All scan configuration settings for data sources.

Contents:

• Configuration identification (ID, name, data source)

• Record limits and filtering (record limit, WHERE clause, ORDER BY)

• Scheduling settings (enable schedule, cron schedule, timezone)

• Vertical scale settings (number of procedure calls, row limit per call, thread count)

• Warehouse assignment (name, size, reference ID)

Use Cases:

• Configuration review

• Configuration comparison

• Integration workflows

• Audit and compliance

Example Query:

Example

SELECT 
    datasource_name,
    configuration_name,
    enable_schedule,
    cron_schedule,
    number_of_procedure_calls,
    thread_count,
    warehouse_name
FROM core.vw_datasource_scan_configurations
WHERE datasource_id = 'your-datasource-id';

---

VW_DATASOURCE_SCAN_ERRORS

Purpose: Error records from scan operations for diagnostics and troubleshooting.

Contents:

• Error identification and context

• Scan and batch information

• Error messages and details

• Timestamps and status information

Use Cases:

• Error diagnostics

• Troubleshooting scan failures

• Quality monitoring

• Integration error handling

Example Query:

Example

SELECT 
    datasource_name,
    configuration_name,
    error_message,
    scan_datetime_begin
FROM core.vw_datasource_scan_errors
WHERE datasource_id = 'your-datasource-id'
ORDER BY scan_datetime_begin DESC
LIMIT 100;

---

VW_DATASOURCE_SCAN_PROCEDURE_CALLS

Purpose: Detailed procedure call information for scans using multiple procedure calls.

Contents:

• Procedure call identification (procedure number, scan ID)

• Procedure timing (start, end, duration)

• Records processed per procedure call

• Performance metrics (records processed per minute per procedure)

• Thread and batch information

• Version information

Use Cases:

• Performance analysis for multi-call scans

• Procedure call optimization

• Timeout management

• Detailed scan diagnostics

Example Query:

Example

SELECT 
    datasource_name,
    procedure_number,
    procedure_records_processed,
    procedure_records_processed_per_minute,
    procedure_start_time,
    procedure_end_time
FROM core.vw_datasource_scan_procedure_calls
WHERE datasource_scan_id = 'your-scan-id'
ORDER BY procedure_number;

---

VW_DATASOURCE_SCAN_COMPUTE_RESOURCE_USE

Purpose: Active scan compute resource utilization for warehouse management.

Contents:

• Scan and data source identification

• Warehouse information (compute resource name, ID)

• Scan status and progress

• Record counts

Use Cases:

• Warehouse utilization monitoring

• Resource conflict detection

• Capacity planning

Example Query:

Example

SELECT 
    compute_resource_name,
    datasource_name,
    scan_status,
    records_processed
FROM core.vw_datasource_scan_compute_resource_use
WHERE scan_status != 'Completed' AND scan_status != 'Failed';

---

VW_DATASOURCE_BILLING_EVENTS

Purpose: Billing event history for product tier upgrades and scan operations.

Contents:

• Event identification (type, description, timestamp)

• Data source information (current and historical names)

• Product tier information

• Billing details (month, year, pricing tier, price, charge)

• Attribute quantity and version information

• Scan association

Note: Created when the product tier has been upgraded for a data source or a scan has been performed in a given month.

Use Cases:

• Billing review and audit

• Cost analysis

• Product tier history

• Financial reporting

Example Query:

Example

SELECT 
    current_datasource_name,
    product_tier_name,
    event_type,
    billing_month,
    billing_year,
    price,
    charge
FROM core.vw_datasource_billing_events
WHERE billing_year = 2024
ORDER BY billing_month DESC, created_datetime DESC;

---

VW_DATASOURCE_PRODUCT_TIER_AUDIT_LOG

Purpose: Historical record of product tier changes for data sources.

Contents:

• Product tier change history

• Timestamps and user information

• Data source information (current and historical)

• Source connection details at time of change

• Record status information

Note: Created when the product tier has changed for a particular data source.

Use Cases:

• Product tier change audit trail

• Historical configuration review

• Compliance and governance

• Change tracking

Example Query:

Example

SELECT 
    current_datasource_name,
    product_tier_name,
    date_added,
    added_by_user,
    datasource_name_at_time_of_change
FROM core.vw_datasource_product_tier_audit_log
WHERE datasource_id = 'your-datasource-id'
ORDER BY date_added DESC;