Attribute Consolidation
Use Attribute Consolidation to unify or standardize attributes from multiple data sources. This helps streamline paths, reduce duplication, and ensure consistent naming.
Overview
Attribute Consolidation allows you to:
Identify and replace parts of the attribute path using a regular expression.
Apply a specialized SQL expression to unify or modify attribute values.
Work alongside existing attribute settings (such as transformation expressions, foreign keys, and null value expressions).
Consolidation Settings
Consolidated Search
A regular expression used to locate the portion of the path or attribute name you want to replace. This is particularly useful when:
Multiple sources share similar attribute paths with minor differences.
You want to rename or reorganize attributes that have inconsistent naming conventions.
Example Usage
If multiple schemas have an attribute path like data.customerInfo and you want to standardize it to data.customer, you might set the regular expression to search for customerInfo and replace it via the Consolidation SQL Expression.
Consolidation SQL Expression
A SQL expression that uses the regular expression matches from Consolidated Search to produce the new, unified attribute path or value.
Use {attribute_name} to reference the original attribute within the expression (if needed).
Example Usage
If {attribute_name} contains data.customerInfo, and you want to consolidate it to data.customer, you can set a consolidation expression that replaces customerInfo with customer.
This tells DataPancake how to rewrite the attribute path during dynamic table generation.
Interaction with Other Attribute Settings
Null Value Expression
You can still set a Null Value Expression to handle cases where an attribute is missing or not applicable.
If the consolidation modifies the path, ensure your null expression still references the correct consolidated attribute name if needed.
Transformation Type and Expression
Transformation Type can be used for additional data conversions beyond simple path rewriting.
Transformation Expression can incorporate the consolidated attribute path. If you reference {attribute_name}, it should reflect the consolidated path.
Foreign Keys
If you have defined Foreign Keys, double-check whether the new, consolidated attribute paths change any relationships or constraints. • Adjust the foreign key definitions if they rely on the old attribute paths.
Use DateTime Format
Consolidation does not typically affect how date/time formats are applied, unless your schema changes the attribute type.
If you’re consolidating datetime fields, confirm that your new paths still match the appropriate date/time formats if using the Use DateTime Format setting.
Attributes Grid Integration
Include In Code Gen: Decide whether the consolidated attribute remains active in the generated dynamic table. • Alias: If you are renaming or unifying attributes, consider using an Alias to clarify the final name.
Best Practices
1. Test Your Regular Expressions Validate your pattern on a subset of data or in an isolated environment before applying it to an entire production data source.
2. Combine with Transformation Wisely If you’re also applying a transformation expression, make sure the consolidated result is still valid input for your transformation logic.
3. Maintain Documentation Keep a record of which attributes are being consolidated and why, especially if multiple teams manage different data sources.
4. Check Performance Impact Large-scale regex operations can impact performance. Optimize your pattern and consider partial or targeted consolidation only where necessary.
Last updated