# DataPancake™ ## Documentation - [Home](https://docs.datapancake.com/readme.md) - [What is DataPancake?](https://docs.datapancake.com/getting-started/what-is-datapancake.md) - [Understanding the Challenges of Semi-Structured Data](https://docs.datapancake.com/getting-started/understanding-the-challenges-of-semi-structured-data.md) - [Install & Setup](https://docs.datapancake.com/getting-started/install-and-setup.md): This guide will walk you through installing the free version of DataPancake, upgrading it to the paid version, and completing necessary admin setup steps. - [Data Source Quick Start](https://docs.datapancake.com/getting-started/data-source-quick-start.md): This quick start will help you implement your first DataPancake pipeline for semi-structured data. - [JSON Tutorial](https://docs.datapancake.com/tutorials/json-tutorial.md): This detailed tutorial will walk you through creating and configuring a JSON DataPancake pipeline using an example dataset. - [How to Guides (UI)](https://docs.datapancake.com/guides/how-to-guides-ui.md) - [Creating & Configuring Data Sources](https://docs.datapancake.com/guides/how-to-guides-ui/creating-and-configuring-data-sources.md) - [How to Create a Data Source for Data Streamed from Kafka](https://docs.datapancake.com/guides/how-to-guides-ui/creating-and-configuring-data-sources/how-to-create-a-data-source-for-data-streamed-from-kafka.md): Create a data source for data streamed from Kafka in DataPancake. - [How to Configure Materialization & Dynamic Table Settings](https://docs.datapancake.com/guides/how-to-guides-ui/creating-and-configuring-data-sources/how-to-configure-output-object-and-dynamic-table-settings.md): Configure Output Object Settings, Dynamic Table Settings, and Dynamic Table Metadata Settings for a data source. - [How to Configure the Secure View Layer](https://docs.datapancake.com/guides/how-to-guides-ui/creating-and-configuring-data-sources/how-to-configure-the-secure-semantic-view-layer.md): Configure the Semantic Layer Settings for a data source. - [How to Configure Schema Filters](https://docs.datapancake.com/guides/how-to-guides-ui/creating-and-configuring-data-sources/how-to-configure-schema-filters.md): Configure Schema Filters for a data source. - [How to Configure a Data Source for a Single Semi-Structured Data Column in a Table or View](https://docs.datapancake.com/guides/how-to-guides-ui/creating-and-configuring-data-sources/how-to-configure-a-data-source-for-a-single-semi-structured-data-column-in-a-table-or-view.md): Configure a table or view data source for semi-structured data. - [How to Configure a Data Source for a Structured Table or View](https://docs.datapancake.com/guides/how-to-guides-ui/creating-and-configuring-data-sources/how-to-configure-a-data-source-for-a-structured-table-or-view.md): Configure a table or view data source for structured data. - [How to Configure an External Table Data Source (JSON Only)](https://docs.datapancake.com/guides/how-to-guides-ui/creating-and-configuring-data-sources/how-to-configure-an-external-table-data-source-json-only.md): Configure an External Table data source. - [How to Configure an Iceberg Data Source (JSON Only)](https://docs.datapancake.com/guides/how-to-guides-ui/creating-and-configuring-data-sources/how-to-configure-an-iceberg-data-source-json-only.md): Configure an Iceberg data source. - [How to Manage a Scan Configuration](https://docs.datapancake.com/guides/how-to-guides-ui/creating-and-configuring-data-sources/how-to-manage-a-scan-configuration.md): Create or modify a scan configuration for a data source. - [Page](https://docs.datapancake.com/guides/how-to-guides-ui/creating-and-configuring-data-sources/page.md) - [How to Configure Schema Consolidation](https://docs.datapancake.com/guides/how-to-guides-ui/creating-and-configuring-data-sources/how-to-configure-schema-consolidation.md): Configure Schema Consolidation for a data source. - [How to Configure Baseline Scan Settings](https://docs.datapancake.com/guides/how-to-guides-ui/creating-and-configuring-data-sources/how-to-configure-baseline-scan-settings.md): Configure Baseline Scan Settings for a data source. - [How to Modify the Schema](https://docs.datapancake.com/guides/how-to-guides-ui/creating-and-configuring-data-sources/how-to-modify-the-schema.md) - [Modify Enabled Features for a Data Source](https://docs.datapancake.com/guides/how-to-guides-ui/modify-enabled-features-for-a-data-source.md): Enable or disable the Pipeline Designer, SQL Code Generation, Security Policy Integration, Data Dictionary Builder, and Semantic Model Generator as needed for each data source. - [How to Scan the Data Source](https://docs.datapancake.com/guides/how-to-guides-ui/how-to-scan-the-data-source.md) - [Pipeline Designer](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer.md) - [How to Create Virtual Attributes](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-create-virtual-attributes.md) - [How to Add Column Materialization Rules](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-add-column-materialization-rules.md) - [How to Apply Column-Level Schema Transformations](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-apply-column-level-schema-transformations.md) - [How to Merge Polymorphic Attribute Versions](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-merge-polymorphic-attribute-versions.md): This feature allows you to automatically merge all numeric attributes into either the float or string polymorphic version of the attribute by creating a sql expression and an alias. - [How to Add Aliases](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-add-aliases.md) - [How to Configure Array Relationships \[Coming Soon\]](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-configure-array-relationships-coming-soon.md) - [How to Configure Security Policies](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-configure-security-policies.md) - [How to Configure Foreign Keys](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-configure-foreign-keys.md) - [How to Generate Dynamic SQL Statement Code](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-generate-dynamic-sql-statement-code.md) - [Data Dictionary Builder](https://docs.datapancake.com/guides/how-to-guides-ui/data-dictionary-builder.md) - [How to Generate a Data Source Description](https://docs.datapancake.com/guides/how-to-guides-ui/data-dictionary-builder/how-to-generate-a-data-source-description.md) - [How to Generate Array Descriptions & Synonyms](https://docs.datapancake.com/guides/how-to-guides-ui/data-dictionary-builder/how-to-generate-array-descriptions-and-synonyms.md) - [How to Generate Attribute Descriptions, Synonyms, & Sample Values](https://docs.datapancake.com/guides/how-to-guides-ui/data-dictionary-builder/how-to-generate-attribute-descriptions-synonyms-and-sample-values.md) - [How to Deploy & Validate Generated Code](https://docs.datapancake.com/guides/how-to-guides-ui/how-to-deploy-and-validate-generated-code.md) - [Data Dictionary](https://docs.datapancake.com/guides/how-to-guides-ui/data-dictionary.md) - [Semantic Model Generator](https://docs.datapancake.com/guides/how-to-guides-ui/semantic-model-generator.md) - [How to Guides (Script Builder)](https://docs.datapancake.com/guides/how-to-guides-script-builder.md) - [Admin Guides](https://docs.datapancake.com/guides/how-to-guides-script-builder/admin-guides.md) - [How to Grant Account Level Privileges to DataPancake](https://docs.datapancake.com/guides/how-to-guides-script-builder/admin-guides/how-to-grant-account-level-privileges-to-datapancake.md): Build a script to grant DataPancake the necessary account-level and Cortex AI privileges required for scanning and schema discovery the Cortex AI assisted data dictionary builder. - [How to Configure Available Warehouses in DataPancake](https://docs.datapancake.com/guides/how-to-guides-script-builder/admin-guides/how-to-configure-available-warehouses-in-datapancake.md): Build a script to configure virtual warehouse access for DataPancake by granting usage on existing warehouses or provisioning new, optimized warehouses for scanning and discovery workloads. - [How to Manage Access to DataPancake](https://docs.datapancake.com/guides/how-to-guides-script-builder/admin-guides/how-to-manage-access-to-datapancake.md): To allow additional users to access DataPancake, you can grant the APP\_PUBLIC application role to any existing system or custom role in your Snowflake account. - [Data Source Guides](https://docs.datapancake.com/guides/how-to-guides-script-builder/data-source-guides.md) - [How to Create a Single DataPancake Data Source (Script Builder)](https://docs.datapancake.com/guides/how-to-guides-script-builder/data-source-guides/how-to-create-a-single-datapancake-data-source-script-builder.md): Build a script to create and initiate a scan for a single data source in DataPancake and optionally generate SQL code. - [Alert Guides](https://docs.datapancake.com/guides/how-to-guides-script-builder/alert-guides.md) - [How to Create Schema Drift Alerts (Script Builder)](https://docs.datapancake.com/guides/how-to-guides-script-builder/alert-guides/how-to-create-schema-drift-alerts-script-builder.md): Build a script to create schema drift email alerts for all connected data sources in DataPancake. - [How to Guides (Bulk Upsert via CSV)](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv.md) - [Database Object Initialization](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/database-object-initialization.md) - [Core Attributes - Initialization](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/database-object-initialization/core-attributes-initialization.md): Create a new stage and upload the stored procedure to enable bulk updates for core attribute metadata. - [Core + Security Attributes - Initialization](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/database-object-initialization/core-+-security-attributes-initialization.md): Create a new stage and upload the stored procedure to enable bulk updates for both core and security attribute metadata. - [Core + Data Dictionary Attributes - Initialization](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/database-object-initialization/core-+-data-dictionary-attributes-initialization.md): Select a stage and upload the stored procedure to enable bulk updates for both core and data dictionary attribute metadata or to generate descriptions, synonyms, and sample values via Cortex AI. - [Datasource Metadata - Initialization](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/database-object-initialization/datasource-metadata-initialization.md): Select a stage and upload the stored procedure to enable bulk updates for datasource metadata. - [Array Foreign Key - Initialization](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/database-object-initialization/array-foreign-key-initialization.md): Select a stage and upload the stored procedure to enable bulk updates for array foreign key metadata metadata. - [Schema Transformation / Consolidation - Intialization](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/database-object-initialization/schema-transformation-consolidation-intialization.md): Select a stage and upload the stored procedure to enable bulk updates for schema transformation metadata. - [Data Source + CodeGen + Data Dictionary](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/database-object-initialization/data-source-+-codegen-+-data-dictionary.md) - [Scan Configuration](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/database-object-initialization/scan-configuration.md) - [Virtual Attributes](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/database-object-initialization/virtual-attributes.md) - [Semantic Model](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/database-object-initialization/semantic-model.md) - [Usage](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/usage.md) - [Core Attributes - Bulk Update](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/usage/core-attributes-bulk-update.md): Select and export the attribute metadata to a .csv file, edit in a spreadsheet like Google Sheets or Excel, then perform a bulk update for core attribute metadata in DataPancake. - [Core + Security Attributes - Bulk Update](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/usage/core-+-security-attributes-bulk-update.md): Select and export the attribute metadata to a .csv file, edit in a spreadsheet like Google Sheets or Excel, then perform a bulk update for core and security attribute metadata in DataPancake. - [Core + Data Dictionary Attributes - Bulk Update](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/usage/core-+-data-dictionary-attributes-bulk-update.md): Export the attribute metadata to a .csv file, edit in a spreadsheet like Google Sheets or Excel, then use to perform a bulk update on core & data dictionary attribute metadata in DataPancake. - [Datasources - Bulk Update](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/usage/datasources-bulk-update.md): Select and export the datasource metadata to a .csv file, edit in a spreadsheet like Google Sheets or Excel, then perform a bulk update in DataPancake. - [Array Foreign Key - Bulk Update](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/usage/array-foreign-key-bulk-update.md): Select and export the array foreign key metadata to a .csv file, edit in a spreadsheet like Google Sheets or Excel, then perform a bulk update in DataPancake. - [Schema Transformation / Consolidation - Bulk Update](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/usage/schema-transformation-consolidation-bulk-update.md): Select and export the scema transformation metadata to a .csv file, edit in a spreadsheet like Google Sheets or Excel, then perform a bulk update in DataPancake. - [temp](https://docs.datapancake.com/guides/how-to-guides-bulk-upsert-via-csv/temp.md) - [Miscellaneous Guides](https://docs.datapancake.com/guides/miscellaneous-guides.md) - [How to Install DataPancake](https://docs.datapancake.com/guides/miscellaneous-guides/how-to-install-datapancake.md): Purchase and Install the DataPancake Snowflake Native App into an existing Snowflake account. - [How to Upgrade DataPancake from the Free Trial](https://docs.datapancake.com/guides/miscellaneous-guides/how-to-upgrade-datapancake-from-the-free-trial.md): Upgrade DataPancake from a free trial to use all of its features. - [JSON Tutorial UDFs & Security Policies](https://docs.datapancake.com/guides/miscellaneous-guides/json-tutorial-udfs-and-security-policies.md): Copy and deploy these in a workspace for the JSON Tutorial. - [FAQ](https://docs.datapancake.com/guides/faq.md): Frequently Asked Questions - [Warehouses](https://docs.datapancake.com/core-concepts/warehouses.md): Snowflake virtual warehouses configured for DataPancake scanning operations, with specific requirements for Snowpark-Optimized warehouses and concurrency settings. - [Data Sources](https://docs.datapancake.com/core-concepts/data-sources.md): Introduction to data sources and navigation hub for all data source documentation. - [Adding Data Sources](https://docs.datapancake.com/core-concepts/data-sources/adding-data-sources.md) - [Baseline Scan Settings](https://docs.datapancake.com/core-concepts/data-sources/baseline-scan-settings.md) - [Basic Configuration Settings](https://docs.datapancake.com/core-concepts/data-sources/basic-configuration-settings.md) - [Data Source Types](https://docs.datapancake.com/core-concepts/data-sources/data-source-types.md) - [Dynamic Table Settings](https://docs.datapancake.com/core-concepts/data-sources/dynamic-table-settings.md) - [Materialization Settings](https://docs.datapancake.com/core-concepts/data-sources/materialization-settings.md) - [Product Tiers & Features](https://docs.datapancake.com/core-concepts/data-sources/product-tiers-and-features.md) - [Schema Filters](https://docs.datapancake.com/core-concepts/data-sources/schema-filters.md) - [Schema Transformations](https://docs.datapancake.com/core-concepts/data-sources/schema-transformations.md) - [Secure View Settings](https://docs.datapancake.com/core-concepts/data-sources/secure-view-settings.md) - [Source Object Settings](https://docs.datapancake.com/core-concepts/data-sources/source-object-settings.md) - [Scan Configurations](https://docs.datapancake.com/core-concepts/scan-configurations.md): Introduction to scan configurations - learn and control how DataPancake scans your data. - [Basic Configuration Settings](https://docs.datapancake.com/core-concepts/scan-configurations/basic-configuration-settings.md): Core settings for naming, attribute discovery method, status, and auto code generation. - [Scheduling Settings](https://docs.datapancake.com/core-concepts/scan-configurations/scheduling-settings.md): Configure automated scanning with cron schedules and time zones. - [Data Source & Warehouse Settings](https://docs.datapancake.com/core-concepts/scan-configurations/data-source-and-warehouse-settings.md): Configure record limits, WHERE clauses, and warehouse selection for scan operations. - [Vertical Scale Settings](https://docs.datapancake.com/core-concepts/scan-configurations/vertical-scale-settings.md): Control thread count, procedure calls, and memory management for optimal scan performance. - [Source Stream Settings](https://docs.datapancake.com/core-concepts/scan-configurations/source-stream-settings.md): Configure incremental scanning with timestamp tracking for efficient delta updates. - [Common Patterns & Best Practices](https://docs.datapancake.com/core-concepts/scan-configurations/common-patterns-and-best-practices.md): Pre-configured patterns and optimization recommendations for common scan scenarios. - [Attribute Metadata](https://docs.datapancake.com/core-concepts/attribute-metadata.md) - [Attribute Discovery Process](https://docs.datapancake.com/core-concepts/attribute-metadata/attribute-discovery-process.md): How DataPancake discovers attributes during scanning, including polymorphic detection and recursive parsing of stringified JSON. - [Attribute Metadata Types](https://docs.datapancake.com/core-concepts/attribute-metadata/attribute-metadata-types.md): Overview of the three attribute types and where they come from: Discovered (from scans), Schema (from samples), and Virtual (user-created). - [Attribute Metadata Details](https://docs.datapancake.com/core-concepts/attribute-metadata/attribute-metadata-details.md): Complete reference for attribute metadata, covering discovered source schema fields and configurable extended metadata that controls SQL transformation, security policies, and code generation. - [Source Schema](https://docs.datapancake.com/core-concepts/attribute-metadata/attribute-metadata-details/source-schema.md): Read-only metadata discovered during scanning, including attribute paths, nesting levels, data types, structure information, and sample values representing what DataPancake found in your source data. - [Destination Schema](https://docs.datapancake.com/core-concepts/attribute-metadata/attribute-metadata-details/destination-schema.md): Configuration for mapping source attributes to Snowflake data types, including precision, scale, datetime formats, primary key identification, and data quality indicators for Dynamic Tables. - [Transformation Security](https://docs.datapancake.com/core-concepts/attribute-metadata/attribute-metadata-details/transformation-security.md): Configuration for attribute transformations during materialization, semantic layer expressions, column naming, null handling, schema consolidation, and security policies including row-level access and - [Arrays](https://docs.datapancake.com/core-concepts/attribute-metadata/attribute-metadata-details/arrays.md): Configuration for array attributes in code generation, including table creation, relationship types and joins, custom aliases for tables and views, and security policies for array tables. - [Foreign Keys](https://docs.datapancake.com/core-concepts/attribute-metadata/attribute-metadata-details/foreign-keys.md): Configuration for foreign key relationships between array tables and parent tables, including parent attribute selection, column aliases, and row-level security policy integration. - [Polymorphic Versions](https://docs.datapancake.com/core-concepts/attribute-metadata/polymorphic-versions.md): How DataPancake proactively creates all 7 polymorphic versions for every attribute upfront, then activates only the versions that match discovered data types. - [DataPancake Metadata Fields](https://docs.datapancake.com/core-concepts/attribute-metadata/datapancake-metadata-fields.md): Reference for all extended metadata fields that control how attributes are transformed into SQL, including data types, transformations, and security policies. - [Arrays & Nested Structures](https://docs.datapancake.com/core-concepts/attribute-metadata/arrays-and-nested-structures.md): How DataPancake handles object arrays, primitive arrays, and nested structures, including foreign key relationships and table generation. - [Attribute Lifecycle](https://docs.datapancake.com/core-concepts/attribute-metadata/attribute-lifecycle.md): How attributes are created, versioned, and managed over time, including schema evolution and version activation/deactivation. - [Attributes in Code Generation](https://docs.datapancake.com/core-concepts/attribute-metadata/attributes-in-code-generation.md): How active polymorphic versions become SQL columns in Dynamic Tables, including transformation application and array table generation. - [Best Practices](https://docs.datapancake.com/core-concepts/attribute-metadata/best-practices.md): Configuration recommendations for attributes, including data type selection, transformation expressions, and performance optimization. - [Integration & API](https://docs.datapancake.com/core-concepts/attribute-metadata/integration-and-api.md): Programmatic access to attributes via system views and stored procedures for automation and external system integration. - [Foreign Keys](https://docs.datapancake.com/core-concepts/attribute-metadata/foreign-keys.md) - [SQL Expressions](https://docs.datapancake.com/core-concepts/attribute-metadata/sql-expressions.md) - [Scan Processing](https://docs.datapancake.com/core-concepts/scan-processing.md) - [Views](https://docs.datapancake.com/core-concepts/views.md): System views for monitoring, querying, and integrating with DataPancake metadata and operations. - [Schema Summary Views](https://docs.datapancake.com/core-concepts/views/schema-summary-views.md): Views for data source overview, scan history, configurations, and billing information. - [Schema Analysis Views](https://docs.datapancake.com/core-concepts/views/schema-analysis-views.md): Views for analyzing attributes, polymorphic versions, and schema evolution. - [Security & Governance Views](https://docs.datapancake.com/core-concepts/views/security-and-governance-views.md): Views for security policy configuration, data dictionary metadata, and transformation settings. - [Integration & Code Generation Views](https://docs.datapancake.com/core-concepts/views/integration-and-code-generation-views.md): Views for semantic layer configuration, foreign key relationships, and generated SQL code. - [Monitoring & Diagnostic Views](https://docs.datapancake.com/core-concepts/views/monitoring-and-diagnostic-views.md): Views for integration process tracking and access control. - [Attribute Consolidation](https://docs.datapancake.com/core-concepts/attribute-consolidation.md): Configure how attributes extract values from source data when schema transformations have consolidated paths during discovery. Attribute consolidation bridges the gap between consolidated discovery pa - [Reference Architecture](https://docs.datapancake.com/core-concepts/reference-architecture.md): These diagrams illustrate the architecture and security of DataPancake as a Snowflake Native App. - [Generated SQL DDL Deployment Options](https://docs.datapancake.com/core-concepts/reference-architecture/generated-sql-ddl-deployment-options.md): Each generated DataPancake object can be deployed to its own DB and Schema or they can be deployed to a single DB and Schema. - [Release Notes](https://docs.datapancake.com/development/release-notes.md) - [Roadmap](https://docs.datapancake.com/development/roadmap.md) - [Issues](https://docs.datapancake.com/development/issues.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on a page URL with the `ask` query parameter: ``` GET https://docs.datapancake.com/readme.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.