# JSON Tutorial

{% hint style="info" %}
**How to use this tutorial:**

Reference the screenshots and code blocks for the input values you should use as you follow along.

You can open the linked guides in new tabs to easily return to this tutorial for the next steps in the process.
{% endhint %}

{% stepper %}
{% step %}

### **Deploy Data Objects & Load Data**

Create the table and [load the data](https://docs.snowflake.com/en/guides-overview-loading-data) which can be downloaded here:

{% file src="<https://201309747-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkX4nSXGvrFIEggg2Drr1%2Fuploads%2Foj5uzb9MzIWMr8lmjUqi%2Fpharma_manufacturing_demo_json_data.json?alt=media&token=83677d44-e710-45f9-b96c-57f7d44f479e>" %}

Additionally, copy and deploy the [UDFs and security policies](https://docs.datapancake.com/guides/miscellaneous-guides/json-tutorial-udfs-and-security-policies).

Note: The Enterprise version of DataPancake is required to use the security policies used in this tutorial.
{% endstep %}

{% step %}

### **Create the Data Source**

Use the [Script Builder to generate the initial source](https://docs.datapancake.com/guides/how-to-guides-script-builder/data-source-guides/how-to-create-a-single-datapancake-data-source-script-builder).

Make sure to check "Start Scan", this will discover the raw schema and attributes.
{% endstep %}

{% step %}

### **Finalize the Schema & Rescan**

[Add schema transformations](https://docs.datapancake.com/guides/how-to-guides-ui/creating-and-configuring-data-sources/how-to-modify-the-schema).

[Re-scan the data source](https://docs.datapancake.com/guides/how-to-guides-ui/how-to-scan-the-data-source) with **reset attributes = true**.

Note: After manual metadata edits, reset attributes is no longer available.
{% endstep %}

{% step %}

### **Define Virtual Attributes**

[Create the new virtual attributes](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-create-virtual-attributes)

This is where you can add surrogate primary keys, create calculated fields, and add ordering attributes for dedupe logic.
{% endstep %}

{% step %}

### **Configure Column Materialization**

[Add column materialization rules](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-add-column-materialization-rules) in the dynamic table layer and the secure view layer.
{% endstep %}

{% step %}

### **Apply Column-Level Schema Transformations**

[Apply column-level schema transformations](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-apply-column-level-schema-transformations).

You can also normalize data types and formats, and perform schema consolidation at the field level.
{% endstep %}

{% step %}

### **Merge Polymorphic Versions**

[Merge string, float, and integer variants](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-merge-polymorphic-attribute-versions) into unified attributes.
{% endstep %}

{% step %}

### **Add Aliases**

[Add user-friendly alias names for columns and arrays](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-add-aliases) where needed.
{% endstep %}

{% step %}

### **Configure Array Relationships \[Coming Soon]**

Define parent/child relationships

Note: this is required for semantic model generation.
{% endstep %}

{% step %}

### **Add Security Policies**

[Configure security policies](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-configure-security-policies) such as row-level access rules, and attribute-level security tags.
{% endstep %}

{% step %}

### **Configure Foreign Keys**

Add relationships between flattened entities by [configuring foreign keys](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-configure-foreign-keys).
{% endstep %}

{% step %}

### **Apply Final Metadata Modifications \[Coming Soon]**

{% endstep %}

{% step %}

### **Generate Code**

[Generate the dynamic SQL statements](https://docs.datapancake.com/guides/how-to-guides-ui/pipeline-designer/how-to-generate-dynamic-sql-statement-code) using your latest configurations.
{% endstep %}

{% step %}

### **Deploy and Validate**

Review and [deploy the generated code](https://docs.datapancake.com/guides/how-to-guides-ui/how-to-deploy-and-validate-generated-code).

Make sure to perform data quality checks and validate security enforcement.
{% endstep %}
{% endstepper %}

### Pipeline Maintenance

Repeat steps 3 - 13 as many times as needed.