Steps can perform many types of functions, such as transform data, filter and categorize data, and read and write data. This section describes some of the most frequently used steps and their functions.
The following table lists and describes frequently used flow control steps in iWay DQS.
|
Icon |
Step |
Description |
|---|---|---|
|
|
Condition |
Directs the data flow. A true condition directs the data flow to the right. A false condition directs the data flow to the left. |
|
|
Filter |
Directs the data flow. A true condition directs data out of the flow. |
|
|
Extract filter |
Directs the data flow. A true condition directs the data flow to the right. An all condition directs the data flow to the left. |
|
|
Multiplicator |
Multiplies the data flow without performing any modification. |
|
|
Trash |
Discards the data flow. |
|
|
Join |
Identical to a SQL table join operation. |
|
|
Union |
Identical to a SQL table union operation. |
|
|
Union Same |
Similar to the Union step, but is applied only if the data flows are identical. |
|
|
Alter format |
Adds or removes columns. |
The following table lists and describes frequently used data parsing steps in iWay DQS.
|
Icon |
Step |
Description |
|---|---|---|
|
|
Regex Matching |
Parses the input string based upon regular expression capturing groups. |
|
|
Pattern Parser |
Parses the input text based on patterns that are provided. You must define all components and optional validations against dictionaries. |
|
|
GuessNameSurname |
Predefined version of Generic Parser used for parsing of names. |
|
|
Strip Titles |
Extracts strings found in dictionary from the input. For example, James White PhD becomes James White, PhD. |
|
|
Apply Replacements |
Replaces values (and even substrings) found in the input with their standardized value. For example, 5th Ave becomes 5th Avenue. |
|
|
Lookup |
Performs a lookup and validation against the dictionary. |
The following table lists and describes frequently used analysis steps in iWay DQS.
|
Icon |
Step |
Description |
|---|---|---|
|
|
Profiling |
Writes comprehensive analysis to a file (.profile). |
|
|
Character Group Analyzer |
Calculates masks. For example digit becomes # and letter becomes A. |
|
|
Word Analyzer |
Substitutes words found in reference dictionaries with symbols. |
|
|
Relational Analysis |
Calculates the number of missing foreign keys for both source flows. |
|
|
Data Quality Indicator |
Calculate statistics for a given set of business rules. Adds a set of boolean flags to each record. |
The following table lists and describes frequently used match and merge steps in iWay DQS.
|
Icon |
Step |
Description |
|---|---|---|
|
|
Unification |
Assigns groups IDs (client, candidate, unification roles). Able to perform the incremental process using the repository. |
|
|
Representative Creator |
Creates new record from the defined group (records already have group ID). Able to add calculated values into the original data flow. |
|
|
Simple Group Classifier |
Calculates the quality of groups (A for automatic processing, U for unique, M for manual processing, and C for additional data cleansing). |
|
|
Unification extended |
Able to run the match process in mixed mode (online and batch in parallel). xx |
