Core Attributes - Bulk Update

Select and export the attribute metadata to a .csv file, edit in a spreadsheet like Google Sheets or Excel, then perform a bulk update for core attribute metadata in DataPancake.

Ensure you have completed Attribute Metadata Bulk Updates (Core) - Setup

Open a new SQL file Snowflake workspace

Run a select statement for datapancake.core.vw_datasource_all_attributes

The minimum field requirements to select are attribute_id, polymorphic_version_id - for example:

select ATTRIBUTE_ID, POLYMORPHIC_VERSION_ID
from datapancake.core.vw_datasource_attribute_security
where version_status = 'active';

Select statement for updating attributes where the source data type is an array

select
-- VIEW ONLY COLUMNS
DATASOURCE_ID,
ATTRIBUTE_ID,
POLYMORPHIC_VERSION_ID,
DATASOURCE_NAME,
DATASOURCE_TAG,
VERSION_NUMBER,
VERSION_STATUS_DATE,
PARENT_OBJECT,
PARENT_ARRAY,
ATTRIBUTE_PATH,
ATTRIBUTE_NAME,
ATTRIBUTE_LEVEL,
ATTRIBUTE_ORDER,
SOURCE_DATA_TYPE,
POLYMORPHIC_ATTRIBUTE_NAME,
ARRAY_TYPE,
ARRAY_PRIMITIVE_TYPE,
-- EDITABLE COLUMNS (Array-specific)
INCLUDE_ARRAY_IN_CODE_GEN, --TRUE, FALSE
ARRAY_RELATIONSHIP_NAME,
ARRAY_RELATIONSHIP_DESCRIPTION,
ARRAY_RELATIONSHIP_TYPE, --many_to_one, one_to_one
ARRAY_RELATIONSHIP_JOIN_TYPE, --left_outer, inner
SEMANTIC_LAYER_ALIAS_NAME,
FLATTEN_ARRAY, --TRUE, FALSE
-- AUDIT COLUMNS
ATTRIBUTE_LAST_UPDATED_DATETIME,
ATTRIBUTE_LAST_UPDATED_BY
from datapancake.core.vw_datasource_all_attributes
where version_status = 'active'
and SOURCE_DATA_TYPE = 'array'
order by attribute_path;

Select statement for updating attributes where source data types are primitive and object

select
-- VIEW ONLY COLUMNS
DATASOURCE_ID,
ATTRIBUTE_ID,
POLYMORPHIC_VERSION_ID,
DATASOURCE_NAME,
DATASOURCE_TAG,
VERSION_NUMBER,
VERSION_STATUS_DATE,
PARENT_OBJECT,
PARENT_ARRAY,
ATTRIBUTE_PATH,
ATTRIBUTE_NAME,
ATTRIBUTE_LEVEL,
ATTRIBUTE_ORDER,
SOURCE_DATA_TYPE,
POLYMORPHIC_ATTRIBUTE_NAME,
SAMPLE_VALUE,
HAS_EMBEDDED_CONTENT,
DATA_PLATFORM_DATA_TYPE,
-- EDITABLE COLUMNS (Non-array attributes)
ATTRIBUTE_RECORD_STATUS,
INCLUDE_COLUMN_IN_SECURE_VIEW,
NUMERIC_PRECISION,
NUMERIC_SCALE,
USE_DATETIME_FORMAT,
DATETIME_FORMAT,
ATTRIBUTE_RECORD_STATUS,
IS_UNIQUE,
IS_PRIMARY_KEY,
CONTAINS_ENUM_VALUES,
RAW_TRANSFORMATION_TYPE,
RAW_TRANSFORMATION_EXPRESSION,
RAW_TRANSFORMATION_EXPRESSION_COMMENT,
RAW_NULL_VALUE_EXPRESSION,
SEMANTIC_LAYER_TRANSFORMATION_EXPRESSION,
SEMANTIC_LAYER_TRANSFORMATION_EXPRESSION_COMMENT,
RAW_ALIAS_NAME,
SEMANTIC_LAYER_ALIAS_NAME,
CONSOLIDATION_REGEX_SEARCH,
CONSOLIDATION_SQL_EXPRESSION,
-- AUDIT COLUMNS
ATTRIBUTE_LAST_UPDATED_DATETIME,
ATTRIBUTE_LAST_UPDATED_BY
from datapancake.core.vw_datasource_all_attributes
where version_status = 'active'
and SOURCE_DATA_TYPE != 'array'
order by attribute_path;

You can choose which editable fields you wish to include in your select statement.

Verify the query successfully returns a list of attribute metadata

Download the results as a .csv file

Open the .csv file in either Excel or Google Sheets

Edit any editable columns as needed

Save (Excel) or download (Google Sheets) the file

Load the .csv into your stage

Run the bulk update stored procedure

Modify the <placeholders> with the same database and schema you uploaded the stored procedure to.

When the stored procedure completes, the attribute metadata file will be moved to the archive folder.

call <database>.<schema>.sp_update_datapancake_attribute_metadata_core(
    '@<stage_name>/<staged_filename>.csv',
    '@<stage_name>/<archive_folder>/',
    '<snowflake_username>'
);

Verify the integration completed successfully by running the following query

select * from datapancake.core.vw_integration_process
where user_name = 'BWOLFF'
order by integration_process_date desc;