Schema Filters

Configuration for excluding parts of the schema from code generation and attribute discovery.

Overview

Schema Filters allow you to exclude specific attributes or attribute patterns from the code generation process using regular expressions. Filtered attributes are discovered during scanning but marked as inactive, preventing them from being included in generated SQL code. Filters can be reactivated at any time by removing the filter or changing attribute status.

---

Filter Configuration

Filter Name

Purpose: Descriptive name for the filter (required field for all filters).

Filter Regular Expression

Purpose: Regular expression pattern matching attribute paths to exclude (required field for all filters).

Pattern Matching:

• Matches against complete attribute paths (e.g., customer.metadata.debug)

• Uses standard regular expression syntax

• Case-sensitive by default

• Can match partial paths or full paths

Examples:

• ^customer\.internal_id$ - Excludes customer.internal_id only

• ^metadata\..* - Excludes all metadata.* attributes

• .*\.debug$ - Excludes any attribute ending with .debug

• .*_temp$ - Excludes attributes ending with _temp

• ^temp\..* - Excludes all attributes under temp.*

• .*\[0\]\.internal.* - Excludes internal attributes in first array element

Note: Use multiple filters for different patterns—each filter can target different attribute groups.

---

How Filters Work

During Scanning

• Filters do not prevent attribute discovery—all attributes are still discovered during scanning

• Attributes matching filter patterns are marked as inactive

• Attributes not matching filters remain active (or keep existing status)

• Status can be changed manually after scanning

During Code Generation

• Only active attributes included in generated SQL

• inactive attributes excluded from code generation

• Filtered attributes not in Dynamic Tables, semantic layer views, or generated SQL code

Reactivation

Removing Filters:

• Delete filter from configuration

• Run scan to update attribute statuses

• Previously filtered attributes become active

Manual Reactivation:

• Change attribute status to active in Attributes management

• Attribute included in future code generation

• No need to remove filter if manually reactivated

---

Filter Use Cases

Exclude Internal Fields:

• ^.*\._internal.*$ - Exclude internal fields

• ^metadata\..* - Exclude metadata fields

• .*\.system_.* - Exclude system fields

Exclude Debug/Temporary Fields:

• .*_debug$ - Exclude debug fields

• .*_temp$ - Exclude temporary fields

• ^temp\..* - Exclude temp namespace

Exclude Sensitive Data:

• .*password.* - Exclude password fields

• .*ssn.* - Exclude SSN fields

• .*pii\..* - Exclude PII namespace

Exclude Large/Unused Attributes:

• .*\.binary_data$ - Exclude binary data

• .*\.large_text$ - Exclude large text fields

• ^unused\..* - Exclude unused namespace

Exclude Specific Paths:

• ^customer\.address\.coordinates$ - Exclude specific nested attribute

• .*\[.*\]\.internal.* - Exclude internal fields in arrays

• ^root\.metadata\..* - Exclude metadata under root

---

Managing Filters

Adding Filters:

• Navigate to data source management

• Scroll to Schema Filters section

• Click + symbol to add new row

• Enter Filter Name (required)

• Enter Filter Regular Expression (required)

• Click "Save Schema Filters"

Editing Filters:

• Find filter in grid

• Edit Filter Name or Regular Expression directly

• Changes saved when "Save Schema Filters" clicked

• System validates regular expression syntax

Deleting Filters:

• Move mouse to leftmost column

• Checkbox appears for row selection

• Check box for filter to delete

• Press Delete key

• Click "Save Schema Filters" to confirm deletion

Filter Validation:

• System validates regex syntax on save

• Invalid regex patterns cause save to fail

• Error message indicates regex issue

---

Filter vs. Transformation

When to Use Filters:

• You want to exclude attributes completely

• Attributes are not needed in materialized objects

• You want to reduce object complexity

• Attributes should remain in source only

When to Use Transformations:

• You want to modify attribute names

• You need to change data types

• You want to restructure schema

• You need to normalize data

Order of Application:

1. Transformations applied first (consolidation, type changes)

2. Filters applied after transformations

3. Final schema used for code generation

---

Summary

Core Configuration:

• Filter Name (required)

• Filter Regular Expression (required)

Functionality:

• Excludes attributes from code generation

• Marks matching attributes as inactive

• Can be reactivated by removing filter or changing attribute status

Use Cases:

• Exclude internal/debug fields

• Remove sensitive data

• Optimize object size

• Fine-grained attribute control

Important: Filters are applied after transformations. Order matters when combining filters and transformations.

For schema transformations, see Schema Transformations. For attribute management, see the Attributes documentation.