# Schema Filters

Configuration for excluding parts of the schema from code generation and attribute discovery.

## Overview

Schema Filters allow you to exclude specific attributes or attribute patterns from the code generation process using regular expressions. Filtered attributes are discovered during scanning but marked as `inactive`, preventing them from being included in generated SQL code. Filters can be reactivated at any time by removing the filter or changing attribute status.

***

## Filter Configuration

### Filter Name

**Purpose:** Descriptive name for the filter (required field for all filters).

### Filter Regular Expression

**Purpose:** Regular expression pattern matching attribute paths to exclude (required field for all filters).

**Pattern Matching:**

* Matches against complete attribute paths (e.g., `customer.metadata.debug`)
* Uses standard regular expression syntax
* Case-sensitive by default
* Can match partial paths or full paths

**Examples:**

* `^customer\.internal_id$` - Excludes `customer.internal_id` only
* `^metadata\..*` - Excludes all `metadata.*` attributes
* `.*\.debug$` - Excludes any attribute ending with `.debug`
* `.*_temp$` - Excludes attributes ending with `_temp`
* `^temp\..*` - Excludes all attributes under `temp.*`
* `.*\[0\]\.internal.*` - Excludes internal attributes in first array element

**Note:** Use multiple filters for different patterns—each filter can target different attribute groups.

***

## How Filters Work

### During Scanning

* Filters do not prevent attribute discovery—all attributes are still discovered during scanning
* Attributes matching filter patterns are marked as `inactive`
* Attributes not matching filters remain `active` (or keep existing status)
* Status can be changed manually after scanning

### During Code Generation

* Only `active` attributes included in generated SQL
* `inactive` attributes excluded from code generation
* Filtered attributes not in Dynamic Tables, semantic layer views, or generated SQL code

### Reactivation

**Removing Filters:**

* Delete filter from configuration
* Run scan to update attribute statuses
* Previously filtered attributes become active

**Manual Reactivation:**

* Change attribute status to `active` in Attributes management
* Attribute included in future code generation
* No need to remove filter if manually reactivated

***

## Filter Use Cases

**Exclude Internal Fields:**

* `^.*\._internal.*$` - Exclude internal fields
* `^metadata\..*` - Exclude metadata fields
* `.*\.system_.*` - Exclude system fields

**Exclude Debug/Temporary Fields:**

* `.*_debug$` - Exclude debug fields
* `.*_temp$` - Exclude temporary fields
* `^temp\..*` - Exclude temp namespace

**Exclude Sensitive Data:**

* `.*password.*` - Exclude password fields
* `.*ssn.*` - Exclude SSN fields
* `.*pii\..*` - Exclude PII namespace

**Exclude Large/Unused Attributes:**

* `.*\.binary_data$` - Exclude binary data
* `.*\.large_text$` - Exclude large text fields
* `^unused\..*` - Exclude unused namespace

**Exclude Specific Paths:**

* `^customer\.address\.coordinates$` - Exclude specific nested attribute
* `.*\[.*\]\.internal.*` - Exclude internal fields in arrays
* `^root\.metadata\..*` - Exclude metadata under root

***

## Managing Filters

**Adding Filters:**

* Navigate to data source management
* Scroll to Schema Filters section
* Click `+` symbol to add new row
* Enter Filter Name (required)
* Enter Filter Regular Expression (required)
* Click "Save Schema Filters"

**Editing Filters:**

* Find filter in grid
* Edit Filter Name or Regular Expression directly
* Changes saved when "Save Schema Filters" clicked
* System validates regular expression syntax

**Deleting Filters:**

* Move mouse to leftmost column
* Checkbox appears for row selection
* Check box for filter to delete
* Press Delete key
* Click "Save Schema Filters" to confirm deletion

**Filter Validation:**

* System validates regex syntax on save
* Invalid regex patterns cause save to fail
* Error message indicates regex issue

***

## Filter vs. Transformation

**When to Use Filters:**

* You want to exclude attributes completely
* Attributes are not needed in materialized objects
* You want to reduce object complexity
* Attributes should remain in source only

**When to Use Transformations:**

* You want to modify attribute names
* You need to change data types
* You want to restructure schema
* You need to normalize data

**Order of Application:**

1. Transformations applied first (consolidation, type changes)
2. Filters applied after transformations
3. Final schema used for code generation

***

## Summary

**Core Configuration:**

* Filter Name (required)
* Filter Regular Expression (required)

**Functionality:**

* Excludes attributes from code generation
* Marks matching attributes as inactive
* Can be reactivated by removing filter or changing attribute status

**Use Cases:**

* Exclude internal/debug fields
* Remove sensitive data
* Optimize object size
* Fine-grained attribute control

**Important:** Filters are applied after transformations. Order matters when combining filters and transformations.

For schema transformations, see [Schema Transformations](https://docs.datapancake.com/core-concepts/data-sources/schema-transformations). For attribute management, see the Attributes documentation.