In this section: |
The following parameters are advanced settings only. There is an advanced way to decide the right value, and therefore, there is no guarantee of improved performance when a parameter is used. To use some of these parameters, just add them as a java parameter, for example, -dnme.copyColumns.skipUnchanged=true.
The following table describes the parameters for Process Control.
Parameter |
Default Value |
Description |
---|---|---|
nme.copyColumns.skipUnchanged |
false |
If true, a CopyColumns operation will skip its execution in certain situations when the data is unchanged in a way that recomputing the aggregation should yield the same result. A skip happens when sourceEntity has no changes and targetEntity has only UPDATE (no INSERT or DELETE instances). Note: A DELETE instance is produced only by a load operation with SourceDeletionStrategy DELETE or by calling RW native services with changeType X. SourceDeletionStrategy, DEACTIVATE, and calling RW native services with changeType D are considered as UPDATE. Must be turned off for reprocessing if aggregation logic has changed. |
nme.persistence.overwriteUnchanged |
false |
If true, records are unchanged, and a reprocess operation will be updated in the storage, even though no change was actually necessary. From a logical point of view, only metadata (last change date and ID) will be different. |
The following table shows the parameters for Performance.
Parameter |
Default Value |
Description |
---|---|---|
nme.clean.parallelism |
1 |
Default parallelism setting for cleansing plans. May be overridden by .perf files. |
nme.multiLoad.parallel |
1 |
Number of delta load operations that will be executed in parallel during a MultiLoad operation. |
nme.delta.parallel |
1 |
Number of threads that will be used to load entities in delta load operations. |
nme.delta.parallel.threshold |
10000 |
Minimum number of incoming records (all entities together) to trigger parallel processing of delta load operation (only if nme.delta.parallel > 1). |
nme.consolidation.parallel |
1 |
Number of threads (and database connections) that will be used in consolidation phase. |
nme.consolidation.parallel.threshold |
10000 |
Minimum number of modified records (all entities together) after delta detection to trigger parallel consolidation (only if nme.consolidation.parallel > 1). |
nme.commit.parallel |
1 |
Number of threads (and database connections) that will be used in committing phase. |
nme.commit.parallel.threshold |
10000 |
Minimum number of modified records (all entities together) to trigger parallel committing (only if nme.commit.parallel > 1). |
nme.taskExecutor.parallel |
20 |
Number of tasks (loads, exports, and so on) being executed by Task Executor in parallel (as long as synchronization conditions allow that). |
nme.export.read.parallel |
1 |
The number of data sources that will be read in parallel in a single export operation. |
nme.export.read.parallel.nameFilter |
List of comma-separated names of export operations that will support parallel retrieval of its data sources (asterisk wildcard is supported). | |
nme.io.storage.memSize |
10000 |
Number of records kept in memory in temporary storages. Any further records are written to the disk and de-serialized when needed. |
nme.consolidation.pool.minSize |
1 |
Minimum size of pool of plan runners for Identify (Cleanse and Match plan). |
nme.consolidation.pool.maxSize |
5 |
Maximum size of pool of plan runners for Identify (Cleanse and Match plan). |
The following table describes the parameters for Services.
Parameter |
Default Value |
Description |
---|---|---|
nme.services.range.maxCount |
100 |
Maximum allowed value of count element in range expression in list and search services. If range expression is not given in request it is default value. |
The following table describes the other parameters.
Parameter |
Default Value |
Description |
---|---|---|
nme.taskExecutor.taskLogLevel |
DETAIL_ALL |
By default start and finish of all operations (load, export, reprocess, processDelta) and their subtasks are logged to standard logger (usually StdOut). This can be switched off by setting this parameter to NO_DETAIL. In that case, only the start and finish of whole operations will be logged. |
nme.taskExecutor.taskPersistLevel |
PERSIST_ALL |
If set to PERSIST_ALL, batch operations and processDelta monitoring (/console/nmeMonitoring) is persisted between server restarts (this is not supported on XmlPersistence). If set to NO_PERSIST, this monitoring is kept only in memory and lost with server restart. |
nme.taskExecutor.debug=true |
false |
If true, monitoring of the iWay MDS process will contain more detailed information, especially about CopyColumns, Match, and Merge. |
The following table describes the parameters for debug.
Parameter |
Default Value |
Description |
---|---|---|
nme.tmpStorage.checkDuplicates |
false |
If true, data written in each execution step during the Master Data Consolidation phase will be checked for duplicate values in the ID column. This additional checking will cause performance degradation (false by default), but can be useful when searching for internal problems. |
This section describes the database storage.
The following parameters are usable only when DB persistence is used.
Parameter |
Default Value |
Description |
---|---|---|
nme.db.inList.batchSize |
1000 |
Batch size for inList input data. |
nme.db.inList.commitSize |
infinity |
Commit size for inList input data. |
nme.db.batchSize |
100 |
Batch size for writing data in the Committing phase. |
nme.db.commitSize |
infinity |
Commit size for writing data in the Committing phase. |
nme.db.smallDataThreshold |
10000 |
Threshold for number of records deciding if LARGE or SMALL data hints should be used. |
nme.db.alterTableLevel |
addColumns |
Defines what alter table commands will be executed if a table in a database does not match the model. Permitted values include none, addColumns, syncColumns, all. |
The following parameters are usable only when VLDB persistence is used.
Parameter |
Default Value |
Description |
---|---|---|
nme.vldb.inList.batchSize |
1000 |
Batch size for inList input data. |
nme.vldb.inList.commitSize |
infinity |
Commit size for inList input data. |
nme.vldb.batchSize |
100 |
Batch size for writing data in the Committing phase. |
nme.vldb.commitSize |
infinity |
Commit size for writing data in the Committing phase. |
nme.vldb.readonly |
false |
If true, the transactions cannot change the data in repository. In cluster installations (several NME instances are running over same DB storage), this flag should be set to true on all read-only nodes. At most, one node can have this parameter set to false. |
nme.vldb.smallDataThreshold |
200000 |
Threshold for number of inList records determining if LARGE or SMALL data hints should be used. |
nme.vldb.prefetch.large |
(unset) |
If set, the value of this argument will be used as prefetch size for SELECT statements for LARGE expected data sizes. |
nme.vldb.prefetch.small |
(unset) |
If set, the value of this argument will be used as prefetch size for SELECT statements for SMALL expected data sizes. |
nme.vldb.alterTableLevel |
addColumns |
Defines what alter table commands will be executed if a table in database does not match the model. Permitted values include none, addColumns, syncColumns, and all. |
Unification Extended Hint Parameter
The following parameter is usable only when VLDB persistence and the Extended Unification step are used.
Parameter |
Default Value |
Description |
---|---|---|
nmeRepositoryCommandsFile |
(unset) |
Defines Extended Unification hints path to repository_nme.xml file (default location is runtime/repository_nme.xml) |
Data Collector (LTC) Parameters
The LTC stands for Logical Transaction Collector, and it is responsible for deleting obsolete records (similar to Java garbage collector but deals with obsolete data of transactions). The following settings belong to an LTC feature related to the VLDB persistence/Logical Transaction.
Parameter |
Default Value |
Description |
---|---|---|
nme.vldb.collector.enabled |
true |
If true, the LTC process is enabled. |
nme.vldb.collector.batchSize |
100 |
Batch size (#deleted records in one execute() call) for the LTC process. |
nme.vldb.collector.batchWait |
0 |
Delay (in ms) between executing individual deletion batches in the LTC process. |
nme.vldb.collector.commitSize |
10000 |
Commit size (#deleted records in deletion commit sets) for the LTC process. |
nme.vldb.collector.commitWait |
100 |
Delay (in ms) between processing individual deletion commit sets in the LTC process. |
nme.vldb.collector.separation.strategy |
NONE |
Determines the separation between RW transactions and collector operations:
|
nme.vldb.collector.separation.timeout |
10000 |
The time (in ms) between the last occurrence of the separation condition and resuming of the LTC. It is unreasonable if the strategy is set to NONE. Since the underlying implementation does not actually monitor individual SQL statements, the values must be large enough to allow completion of a batched sequence of write commands and database commit commands. The actual wait time will be somewhere between timeout and 2*timeout. |
Hint Parameters
The following parameters make it possible to apply hints for some special computations performed during processing.
Note: The following hint examples are usable for Oracle 11+ database only
Parameter |
Default Value |
Description |
---|---|---|
VldbUnorderedInList.match.small |
/*+ ORDERED USE_NL(l d x) FULL(l) INDEX(d) INDEX(x) */ |
Hint for retrieving matched records for SMALL inLists in unordered lists. |
VldbUnorderedInList.match.large |
/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */ |
Hint for retrieving matched records for LARGE inLists in unordered lists. |
VldbUnorderedInList.unmatch |
/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL(x) SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */ |
Hint for retrieving unmatched records for inLists in unordered lists. Note: For unmatch operations, there is no difference between SMALL and LARGE inputs. |
VldbOrderedInList.match.small |
/*+ ORDERED USE_NL(l d x) FULL(l) INDEX(d) INDEX(x) */ |
Hint for retrieving matched records for SMALL inLists in ordered lists. |
VldbOrderedInList.match.large |
/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL(x) SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */ |
Hint for retrieving matched records for LARGE inLists in ordered lists. |
VldbOrderedInList.unmatch |
/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL(x) SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */ |
Hint for retrieving unmatched records for inLists in ordered lists. Note: For unmatch operations, there is no difference between SMALL and LARGE inputs. |
VldbTableAccessor.find.small |
/*+ORDERED USE_NL(x d) INDEX(x) */ |
Hint for retrieving data using SELECT WHERE for SMALL datasets. |
VldbTableAccessor.find.large |
/*+ORDERED USE_HASH(x d) FULL(d) FULL(x) SWAP_JOIN_INPUTS(x) */ |
Hint for retrieving data using SELECT WHERE for LARGE datasets. |
iWay Software |