Schema Transformation / Consolidation - Bulk Update

Select and export the scema transformation metadata to a .csv file, edit in a spreadsheet like Google Sheets or Excel, then perform a bulk update in DataPancake.

Ensure you have completed Initialization Guide

1. Open a new Snowflake worksheet

2. Run a select statement for DATAPANCAKE.CORE.VW_DATASOURCE_SCHEMA_CONSOLIDATION

SELECT
-- REQUIRED COLUMNS FOR BULK UPDATE
TRANSFORMATION_ID,  -- Unique ID for upsert operations - use {new_id} for new records
DATASOURCE_ID,
-- VIEW ONLY COLUMNS
DATASOURCE_NAME,
DATASOURCE_TAG,
-- EDITABLE COLUMNS FOR BULK UPDATE
TRANSFORMATION_NAME,
TRANSFORMATION_TYPE,
SCHEMA_ATTRIBUTE_PATH_SEARCH,
SCHEMA_REGULAR_EXPRESSION_SEARCH,
SCHEMA_REGULAR_EXPRESSION_REPLACE,
STRING_DELIMITER_VALUE,
STRING_KEY_VALUE_SEPARATOR,
STRING_WRAPPER_VALUE,
STRING_FIELD_WIDTHS
FROM datapancake_documentation.core.vw_datasource_schema_transformation
ORDER BY DATASOURCE_NAME, TRANSFORMATION_NAME;

You can choose which editable fields you wish to include in your select statement.

3. Verify the query successfully returns a list of attribute metadata

4. Download the results as a .csv file

5. Open the .csv file in either Excel or Google Sheets

6. Edit any editable columns as needed

7. Save (Excel) or download (Google Sheets) the file

8. Load the .csv into your stage

9. Run the bulk update stored procedure

When the stored procedure completes, the attribute metadata file will be moved to the archive folder.

call <database>.<schema>.sp_update_datapancake_datasource_schema_transformation_metadata_core(
    '@<stage_name>/<staged_filename>.csv',
    '@<stage_name>/<archive_folder>/',
    '<snowflake_username>'
);

10. Verify the integration completed successfully by running the following query

select * from datapancake.core.vw_integration_process
where user_name = 'BWOLFF'
order by integration_process_date desc;