How to Create a Single DataPancake Data Source (Script Builder)

Build a script to create and initiate a scan for a single data source in DataPancake and optionally generate SQL code.

Please ensure the Admin Guides have been completed before creating your first data source.

This script builder is not available for the following scenarios:

• Data sources with a pre-configured schema

• Data sources that contain stream data such as Kafka

See Creating & Configuring Data Sources guides for more details.

1. Navigate to the "Data Source" script builder

2. Enter the custom data source name

3. (Optional) Change the app database name

The default is DATAPANCAKE

4. Enter the role used to install DataPancake

5. Enter the role which has the select privilege for the database object to be scanned

6. Select the Snowflake Database Object Type

7. Select the Column Data Type (Semi-Structured Only)

8. Select the Data Source Type

9. (Semi-Structured Only) Select the Data Format Type

10. Select the Attribute Create Type

"Discover" means DataPancake will discover the all attributes for you during the scan. "Schema" means you will provide a schema for all known attributes, on top of which DataPancake can find additional discovered attributes.

11. Enter the Snowflake database name that contains the data

If the name is case sensitive, put the name in double quotes such as "EXAMPLE_NAME"

12. (Optional) Indicate if the database is shared

13. Enter the name of the schema which contains the database object to be scanned

If the name is case sensitive, put the name in double quotes such as "EXAMPLE_NAME"

14. Enter the name of the database object to be scanned

If the name is case sensitive, put the name in double quotes such as "EXAMPLE_NAME"

15. (Semi-Structured Only) Enter the name of the column to be scanned

If the name is case sensitive, put the name in double quotes such as "EXAMPLE_NAME"

16. Select the warehouse to be used by DataPancake to scan

17. (Optional) Prevent the scan from starting automatically

If you switch this off, you will need to manually trigger the data source scan from the "Scan Data" page.

18. (Optional) Enter a record limit for the initial scan

If you want to scan all records, set the limit to 0

19. (Optional) Enable Additional Paid Features

Explore https://www.datapancake.com/pricing to learn more about DataPancake's transparent, attribute-based pricing model.

20. (Optional) Disable Using Case Sensitivity

The "Generate Code After Scan" will automatically create the output SQL statements based on the discovered schema at the completion of the scan.

SQL Code Generation is a paid feature available once you purchase the application or as part of an Enterprise license. Code Generation in the Trial version is not enabled.

If you choose not to generate code, skip to the "Generate Script" step below.

21. (Optional) Enable generating SQL code after the scan completes

With the SQL Code Generation feature enabled, you can also generate SQL code on scanned data sources from the "Generated SQL" page as needed.

22. Enter the Materialized Object Type

The type of materialized object to create.

23. Enter the user defined name for the root dynamic table, such as 'dt_<object_name>'

The name used for the materialized object's root name and as a prefix for all nested materialized objects in the code generation process.

24. Enter the name of the warehouse used to refresh dynamic tables

This warehouse name is included as part of the create dynamic SQL table statements.

"Why is this not a dropdown with options like the scan data warehouse one above?"

The previous dropdown is the warehouse used to scan data, which must be connected to DataPancake.

The warehouse here is used to refresh dynamic tables and can be any warehouse, including ones not connected to DataPancake.

25. Select the Semantic Layer - Object Type

The type of semantic layer object to create.

26. Enter the user defined name for the root level view, such as 'vw_<object_name>'

27. Click "Generate Script"

28. Copy the script

29. Open a new Snowflake SQL file and paste the script

30. Scroll Down to Review the Optional Parameters as Part of the Code Generation

31. Execute the Script

Use the blue dropdown arrow at the top right and click "Run All"

32. Ensure successful execution by examining the call result column in the result set

If you mistyped any of the database connection information, the resulting error message will show you what was incorrect.

If the warehouse used has not been previously added to DataPancake, then the scan will not be initiated.

If the scan is initiated, the result message will show that the scan has started.

33. View the scan status

On the "Scans In Process" page you can view the progress of the scan. Click the "Refresh" button to update view the latest status. When the scan is finished, you are ready to start designing your pipeline.