What is DataPancake?

DataPancake is a complete solution for semi-structured data. You can now Pancake your data™ - Flatten, normalize, enrich, and secure complex JSON/XML with a Cortex AI Data Dictionary Builder and a Cortex Analyst Semantic Model Generator - all securely inside of Snowflake.

---

DataPancake's Core Features

🔍 Schema Discovery

• Recursively scans 100% of your semi-structured data (JSON, XML, Avro, Parquet, and ORC) to discover all attributes including nested arrays, objects, and every polymorphic version of each attribute.
• Detects all 7 polymorphic data type variations (4 primitives, 2 types of arrays, and objects).
• Identifies escaped JSON within string fields.
• Scanning and discovery benefits from Snowflake's vertical scaling.
• Infers Snowflake destination data types including correct datetime formats for accurate type conversion.

🛠️ Pipeline Designer

• Enables users to customize how each pipeline SQL DDL will be generated.
• Configure foreign key relationships for nested arrays.
• Apply column-level transformation logic during the materialization process.
• Create virtual attributes for derived fields or semantic model metrics and filters.
• Configure row access and column masking policies integration.
• Configure semantic layer of views including additional column-level transformations.

✨ SQL Code Generation

• Generates SQL DDL code needed to create relational dynamic tables and policy-infused views in Snowflake based on your configured attribute metadata.
• Code-generated Snowflake Dynamic Table SQL DDL using DataPancake ITDCs (Immutable Typed Derived Columns) to create technical-debt-free pipelines.
• Reflects configured transformations, foreign keys, and virtual attributes allowing for post-normalized table joins.
• Code-generated views selecting data from normalized dynamic tables that incorporate row-access and column-masking security policies and additional column-level transformations.
• Code-generated streams, tasks, and tables to track dynamic table metadata including insert and last updated datetime.

🚨 Schema Drift Monitoring

• Continuously monitors and alerts you when your semi-structured data source schema changes.
• Detects schema drift in semi-structured data sources like JSON and XML.
• Flags changes in data types, structure, and new attributes.
• Alerts users to configure newly discovered attributes and regenerate pipeline code.
• Optionally generates updated pipeline SQL DDL upon schema change detection.

📚 Data Dictionary Builder

• Creates a comprehensive data dictionary that includes definitions, synonyms, and sample values for every attribute with integration to the Semantic Model Generator for Cortex Analyst.
• Uses your preferred LLM to generate definitions, synonyms, and sample values.
• Extends DataPancake’s system prompt with your own custom context for greater clarity and improved responses.
• Generates descriptions for the data source, nested arrays, and attributes.

🧠 Cortex Analyst Semantic Model Generator

• Generates Cortex Analyst–ready YAML files that define your complete semantic model.
• Automatically includes relationship metadata based on selected columns.
• Integrated with the Pipeline Designer and Data Dictionary Builder.
• Configures custom metrics, facts, and filters through virtual attributes, then adds verified queries and custom instructions.