Release Notes
DataPancake™ V1.46 Release Notes
Value- Based Pricing Model for Billing
New features have been added to the DataPancake offering. The billing model for DataPancake has been updated to create billable events monthly based on the feature used and the number of attributes contained in a particular data source.
Features offered are:
Schema Discovery (formerly known as Schema Summary) - Free
Attribute Metadata Management (known formerly as Schema Analysis) - Paid
SQL Code Generation (formerly known as Extract, Relate, and Flatten) - Paid
Data Dictionary Builder - Paid
Security Policy Integration - Paid
Cortex Analyst Semantic Model Code Generation - Paid
Feature
Price per Attribute
Attribute Metadata Management
$0.50
SQL Code Generation
$0.90
Data Dictionary Builder
$0.30
Security Policy Integration
$0.75
Semantic Model Code Generation
$0.50
Data Source Overview
New columns have been added to the data source overview grid to indicate which services have been enabled for each data source.
Scans in Process will now show a more accurate estimate of the amount of time it will take to complete a scan based on the baseline scan settings. The baseline scan settings can be reset in the Data Source screen. The next scan completed will recompute the average number of records processed based on the number of threads used during the scan. Each subsequent scan will use the baseline scan settings to estimate the time needed to complete the scan.
Navigation
A new left hand navigation menu has been added to better organize the available features in the application.
Quick Start Script Builders
Three new script builders have been created.
Individual Data Sources
Multiple Data Sources
Schema Drift Alert
Data Source
DataPancake Services
Users will now have the option to enable specific services for each data source. Services replace the option to select a specific Product Tier.
Column Data Type
Users will now have the option to select a source column data type. Currently the only option is Variant. In a future release users will have the option to select String.
Sample Schema Data
Users can supply a single document to represent the schema for a specific data source. This sample document can be in either a JSON or XML format. For example, users can create a sample XML document from an existing XSD file or create a sample JSON document from an existing Avro Schema. Once the sample has been configured users can configure a scan to create attribute metadata from the provided sample instead of through data discovery.
Output Object Settings
Users can choose the type of output object. Current options include Dynamic Table or Table.
Dynamic Table Metadata
Users can now configure a data source to generate the code necessary to track Dynamic Table metadata including the insert and last updated datetimes. This will allow users to see when a Dynamic Table record was inserted or modified. The generated code will create a new table to store the metadata, a merge statement to initialize the table, a stream for the root level Dynamic Table to track changes, and a task to keep the table updated on users configured schedule.
Baseline Scan Settings
Users can see the average number of records processed based on a specific thread count calculated from scan. These numbers can be recalculated by resetting the values to 0. The next subsequent scan will update these calculations.
Required Fields
When creating or updating a data source the user can see what fields are required if a specific piece of information has not been supplied.
Scan Configuration
Users can now configure the type of attribute creation a scan will use. The Discover attribute creation type will generate attribute metadata based on a discovery of the data located in the database object configured in the data source. The Schema attribute creation type will generate attribute metadata based on the single sample document configured for the data source. Multiple scan configurations can be created for a data source using both attribute creation types to allow a user to compare a schema against the data in a data source.
Scan Data Source
No changes
Data Source Attributes
Users can now configure whether an attribute is unique, is a primary key, or whether the attribute contains enum values only.
Users can now modify the status of an attribute from active to inactive. Inactive attributes are not included in the code generation process. Users can now also see the difference between attributes discovered in the data source vs attributes created from the data source’s sample document.
The Data Governance tab has been renamed to “Semantic Layer - Security Policy”
Arrays
Users can now configure relationship information for each nested array including:
Relationship Name
Relationship Description
Relationship Type
Join Type
This relationship information will be used as part of the semantic model code generation process.
Data Source SQL Code Generation
If a user has configured a data source to create Dynamic Table metadata the code will be generated to allow for the creation of the necessary database objects to maintain metadata that will track the insert and last update datetimes of each row in the root level Dynamic Table.
Data Dictionary Builder
The data dictionary/glossary/synonym builder uses Cortex AI to assist users in building a data dictionary for each data source. Descriptions can be created for the data source. Descriptions and synonyms can be created for each nested array and descriptions, synonyms, and sample values can be created for each attribute. Users can choose which LLM model they want to use to generate responses.
Semantic Model Code Generation
Users can create semantic model yaml files for use with Cortex Analyst. Information generated using the data dictionary builder will be included in the generation of all attribute descriptions and synonyms.
Users can choose which attributes to include and can configure additional properties such as:
W Question Category (Dimension, Time Dimension, Facts, Metric, Filter)
Sample Value (multiple comma separated values)
Enum Values (designates that the attribute contains only enum values)
Semantic Model Description (in addition to the glossary definition)
Semantic Model Expression
Cortex Search Service Name
Cortex Search Database
Cortex Search Schema
Cortex Search Column
Additional sections generated include:
Relationships
Verified Queries
Custom Instructions
Worksheet Commands
No Changes
Last updated