# Schema Filters Configuration for excluding parts of the schema from code generation and attribute discovery. ## Overview Schema Filters allow you to exclude specific attributes or attribute patterns from the code generation process using regular expressions. Filtered attributes are discovered during scanning but marked as `inactive`, preventing them from being included in generated SQL code. Filters can be reactivated at any time by removing the filter or changing attribute status. *** ## Filter Configuration ### Filter Name **Purpose:** Descriptive name for the filter (required field for all filters). ### Filter Regular Expression **Purpose:** Regular expression pattern matching attribute paths to exclude (required field for all filters). **Pattern Matching:** * Matches against complete attribute paths (e.g., `customer.metadata.debug`) * Uses standard regular expression syntax * Case-sensitive by default * Can match partial paths or full paths **Examples:** * `^customer\.internal_id$` - Excludes `customer.internal_id` only * `^metadata\..*` - Excludes all `metadata.*` attributes * `.*\.debug$` - Excludes any attribute ending with `.debug` * `.*_temp$` - Excludes attributes ending with `_temp` * `^temp\..*` - Excludes all attributes under `temp.*` * `.*\[0\]\.internal.*` - Excludes internal attributes in first array element **Note:** Use multiple filters for different patterns—each filter can target different attribute groups. *** ## How Filters Work ### During Scanning * Filters do not prevent attribute discovery—all attributes are still discovered during scanning * Attributes matching filter patterns are marked as `inactive` * Attributes not matching filters remain `active` (or keep existing status) * Status can be changed manually after scanning ### During Code Generation * Only `active` attributes included in generated SQL * `inactive` attributes excluded from code generation * Filtered attributes not in Dynamic Tables, semantic layer views, or generated SQL code ### Reactivation **Removing Filters:** * Delete filter from configuration * Run scan to update attribute statuses * Previously filtered attributes become active **Manual Reactivation:** * Change attribute status to `active` in Attributes management * Attribute included in future code generation * No need to remove filter if manually reactivated *** ## Filter Use Cases **Exclude Internal Fields:** * `^.*\._internal.*$` - Exclude internal fields * `^metadata\..*` - Exclude metadata fields * `.*\.system_.*` - Exclude system fields **Exclude Debug/Temporary Fields:** * `.*_debug$` - Exclude debug fields * `.*_temp$` - Exclude temporary fields * `^temp\..*` - Exclude temp namespace **Exclude Sensitive Data:** * `.*password.*` - Exclude password fields * `.*ssn.*` - Exclude SSN fields * `.*pii\..*` - Exclude PII namespace **Exclude Large/Unused Attributes:** * `.*\.binary_data$` - Exclude binary data * `.*\.large_text$` - Exclude large text fields * `^unused\..*` - Exclude unused namespace **Exclude Specific Paths:** * `^customer\.address\.coordinates$` - Exclude specific nested attribute * `.*\[.*\]\.internal.*` - Exclude internal fields in arrays * `^root\.metadata\..*` - Exclude metadata under root *** ## Managing Filters **Adding Filters:** * Navigate to data source management * Scroll to Schema Filters section * Click `+` symbol to add new row * Enter Filter Name (required) * Enter Filter Regular Expression (required) * Click "Save Schema Filters" **Editing Filters:** * Find filter in grid * Edit Filter Name or Regular Expression directly * Changes saved when "Save Schema Filters" clicked * System validates regular expression syntax **Deleting Filters:** * Move mouse to leftmost column * Checkbox appears for row selection * Check box for filter to delete * Press Delete key * Click "Save Schema Filters" to confirm deletion **Filter Validation:** * System validates regex syntax on save * Invalid regex patterns cause save to fail * Error message indicates regex issue *** ## Filter vs. Transformation **When to Use Filters:** * You want to exclude attributes completely * Attributes are not needed in materialized objects * You want to reduce object complexity * Attributes should remain in source only **When to Use Transformations:** * You want to modify attribute names * You need to change data types * You want to restructure schema * You need to normalize data **Order of Application:** 1. Transformations applied first (consolidation, type changes) 2. Filters applied after transformations 3. Final schema used for code generation *** ## Summary **Core Configuration:** * Filter Name (required) * Filter Regular Expression (required) **Functionality:** * Excludes attributes from code generation * Marks matching attributes as inactive * Can be reactivated by removing filter or changing attribute status **Use Cases:** * Exclude internal/debug fields * Remove sensitive data * Optimize object size * Fine-grained attribute control **Important:** Filters are applied after transformations. Order matters when combining filters and transformations. For schema transformations, see [Schema Transformations](/core-concepts/data-sources/schema-transformations.md). For attribute management, see the Attributes documentation. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.datapancake.com/core-concepts/data-sources/schema-filters.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.