JSON Tutorial

This detailed tutorial will walk you through creating and configuring a JSON DataPancake pipeline using an example dataset.

How to use this tutorial:

Reference the screenshots and code blocks for the input values you should use as you follow along.

You can open the linked guides in new tabs to easily return to this tutorial for the next steps in the process.

Create the table and load the data which can be downloaded here:

12MB

Additionally, copy and deploy the UDFs and security policies.

Note: The Enterprise version of DataPancake is required to use the security policies used in this tutorial.

Make sure to check "Start Scan", this will discover the raw schema and attributes.

Re-scan the data source with reset attributes = true.

Note: After manual metadata edits, reset attributes is no longer available.

This is where you can add surrogate primary keys, create calculated fields, and add ordering attributes for dedupe logic.

Add column materialization rules in the dynamic table layer and the secure view layer.

You can also normalize data types and formats, and perform schema consolidation at the field level.

Define parent/child relationships

Note: this is required for semantic model generation.

Configure security policies such as row-level access rules, and attribute-level security tags.

Add relationships between flattened entities by configuring foreign keys.

Make sure to perform data quality checks and validate security enforcement.

Repeat steps 3 - 13 as many times as needed.

Last updated 7 months ago

Was this helpful?