NME Runtime Parameters

In this section:

The following parameters are advanced settings only. There is an advanced way to decide the right value, and therefore, there is no guarantee of improved performance when a parameter is used. To use some of these parameters, just add them as a java parameter, for example, -dnme.copyColumns.skipUnchanged=true.


Top of page

x
Process Control

The following table describes the parameters for Process Control.

Parameter

Default Value

Description

nme.copyColumns.skipUnchanged

false

If true, a CopyColumns operation will skip its execution in certain situations when the data is unchanged in a way that recomputing the aggregation should yield the same result. A skip happens when sourceEntity has no changes and targetEntity has only UPDATE (no INSERT or DELETE instances).

Note: A DELETE instance is produced only by a load operation with SourceDeletionStrategy DELETE or by calling RW native services with changeType X. SourceDeletionStrategy, DEACTIVATE, and calling RW native services with changeType D are considered as UPDATE.

Must be turned off for reprocessing if aggregation logic has changed.

nme.persistence.overwriteUnchanged

false

If true, records are unchanged, and a reprocess operation will be updated in the storage, even though no change was actually necessary. From a logical point of view, only metadata (last change date and ID) will be different.



x
Performance

The following table shows the parameters for Performance.

Parameter

Default Value

Description

nme.clean.parallelism

1

Default parallelism setting for cleansing plans. May be overridden by .perf files.

nme.multiLoad.parallel

1

Number of delta load operations that will be executed in parallel during a MultiLoad operation.

nme.delta.parallel

1

Number of threads that will be used to load entities in delta load operations.

nme.delta.parallel.threshold

10000

Minimum number of incoming records (all entities together) to trigger parallel processing of delta load operation (only if nme.delta.parallel > 1).

nme.consolidation.parallel

1

Number of threads (and database connections) that will be used in consolidation phase.

nme.consolidation.parallel.threshold

10000

Minimum number of modified records (all entities together) after delta detection to trigger parallel consolidation (only if nme.consolidation.parallel > 1).

nme.commit.parallel

1

Number of threads (and database connections) that will be used in committing phase.

nme.commit.parallel.threshold

10000

Minimum number of modified records (all entities together) to trigger parallel committing (only if nme.commit.parallel > 1).

nme.taskExecutor.parallel

20

Number of tasks (loads, exports, and so on) being executed by Task Executor in parallel (as long as synchronization conditions allow that).

nme.export.read.parallel

1

The number of data sources that will be read in parallel in a single export operation.

nme.export.read.parallel.nameFilter

List of comma-separated names of export operations that will support parallel retrieval of its data sources (asterisk wildcard is supported).

nme.io.storage.memSize

10000

Number of records kept in memory in temporary storages. Any further records are written to the disk and de-serialized when needed.

nme.consolidation.pool.minSize

1

Minimum size of pool of plan runners for Identify (Cleanse and Match plan).

nme.consolidation.pool.maxSize

5

Maximum size of pool of plan runners for Identify (Cleanse and Match plan).



x
Services

The following table describes the parameters for Services.

Parameter

Default Value

Description

nme.services.range.maxCount

100

Maximum allowed value of count element in range expression in list and search services. If range expression is not given in request it is default value.



x
Other

The following table describes the other parameters.

Parameter

Default Value

Description

nme.taskExecutor.taskLogLevel

DETAIL_ALL

By default start and finish of all operations (load, export, reprocess, processDelta) and their subtasks are logged to standard logger (usually StdOut). This can be switched off by setting this parameter to NO_DETAIL. In that case, only the start and finish of whole operations will be logged.

nme.taskExecutor.taskPersistLevel

PERSIST_ALL

If set to PERSIST_ALL, batch operations and processDelta monitoring (/console/nmeMonitoring) is persisted between server restarts (this is not supported on XmlPersistence). If set to NO_PERSIST, this monitoring is kept only in memory and lost with server restart.

nme.taskExecutor.debug=true

false

If true, monitoring of the iWay MDS process will contain more detailed information, especially about CopyColumns, Match, and Merge.



x
Debug

The following table describes the parameters for debug.

Parameter

Default Value

Description

nme.tmpStorage.checkDuplicates

false

If true, data written in each execution step during the Master Data Consolidation phase will be checked for duplicate values in the ID column. This additional checking will cause performance degradation (false by default), but can be useful when searching for internal problems.



x
Database Storage

This section describes the database storage.



x
DB Persistence Parameters

The following parameters are usable only when DB persistence is used.

Parameter

Default Value

Description

nme.db.inList.batchSize

1000

Batch size for inList input data.

nme.db.inList.commitSize

infinity

Commit size for inList input data.

nme.db.batchSize

100

Batch size for writing data in the Committing phase.

nme.db.commitSize

infinity

Commit size for writing data in the Committing phase.

nme.db.smallDataThreshold

10000

Threshold for number of records deciding if LARGE or SMALL data hints should be used.

nme.db.alterTableLevel

addColumns

Defines what alter table commands will be executed if a table in a database does not match the model. Permitted values include none, addColumns, syncColumns, all.



x
VLDB Persistence Parameters

The following parameters are usable only when VLDB persistence is used.

Parameter

Default Value

Description

nme.vldb.inList.batchSize

1000

Batch size for inList input data.

nme.vldb.inList.commitSize

infinity

Commit size for inList input data.

nme.vldb.batchSize

100

Batch size for writing data in the Committing phase.

nme.vldb.commitSize

infinity

Commit size for writing data in the Committing phase.

nme.vldb.readonly

false

If true, the transactions cannot change the data in repository. In cluster installations (several NME instances are running over same DB storage), this flag should be set to true on all read-only nodes. At most, one node can have this parameter set to false.

nme.vldb.smallDataThreshold

200000

Threshold for number of inList records determining if LARGE or SMALL data hints should be used.

nme.vldb.prefetch.large

(unset)

If set, the value of this argument will be used as prefetch size for SELECT statements for LARGE expected data sizes.

nme.vldb.prefetch.small

(unset)

If set, the value of this argument will be used as prefetch size for SELECT statements for SMALL expected data sizes.

nme.vldb.alterTableLevel

addColumns

Defines what alter table commands will be executed if a table in database does not match the model. Permitted values include none, addColumns, syncColumns, and all.

Unification Extended Hint Parameter

The following parameter is usable only when VLDB persistence and the Extended Unification step are used.

Parameter

Default Value

Description

nmeRepositoryCommandsFile

(unset)

Defines Extended Unification hints path to repository_nme.xml file (default location is runtime/repository_nme.xml)

Data Collector (LTC) Parameters

The LTC stands for Logical Transaction Collector, and it is responsible for deleting obsolete records (similar to Java garbage collector but deals with obsolete data of transactions). The following settings belong to an LTC feature related to the VLDB persistence/Logical Transaction.

Parameter

Default Value

Description

nme.vldb.collector.enabled

true

If true, the LTC process is enabled.

nme.vldb.collector.batchSize

100

Batch size (#deleted records in one execute() call) for the LTC process.

nme.vldb.collector.batchWait

0

Delay (in ms) between executing individual deletion batches in the LTC process.

nme.vldb.collector.commitSize

10000

Commit size (#deleted records in deletion commit sets) for the LTC process.

nme.vldb.collector.commitWait

100

Delay (in ms) between processing individual deletion commit sets in the LTC process.

nme.vldb.collector.separation.strategy

NONE

Determines the separation between RW transactions and collector operations:

  • NONE. No separation. The collector and updates can run concurrently.
  • TABLE. When transaction writes data to a particular table, the collector cannot modify the table. It can, however, modify other tables.
  • WRITE. When transaction writes data to any table, the collector is suspended.
  • TRANSACTION. The collector is suspended for the whole duration of a RW transaction.

nme.vldb.collector.separation.timeout

10000

The time (in ms) between the last occurrence of the separation condition and resuming of the LTC. It is unreasonable if the strategy is set to NONE. Since the underlying implementation does not actually monitor individual SQL statements, the values must be large enough to allow completion of a batched sequence of write commands and database commit commands. The actual wait time will be somewhere between timeout and 2*timeout.

Hint Parameters

The following parameters make it possible to apply hints for some special computations performed during processing.

Note: The following hint examples are usable for Oracle 11+ database only

Parameter

Default Value

Description

VldbUnorderedInList.match.small

/*+ ORDERED USE_NL(l d x) FULL(l) INDEX(d) INDEX(x) */

Hint for retrieving matched records for SMALL inLists in unordered lists.

VldbUnorderedInList.match.large

/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */

Hint for retrieving matched records for LARGE inLists in unordered lists.

VldbUnorderedInList.unmatch

/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL(x) SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */

Hint for retrieving unmatched records for inLists in unordered lists.

Note: For unmatch operations, there is no difference between SMALL and LARGE inputs.

VldbOrderedInList.match.small

/*+ ORDERED USE_NL(l d x) FULL(l) INDEX(d) INDEX(x) */

Hint for retrieving matched records for SMALL inLists in ordered lists.

VldbOrderedInList.match.large

/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL(x) SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */

Hint for retrieving matched records for LARGE inLists in ordered lists.

VldbOrderedInList.unmatch

/*+ORDERED USE_HASH(l d x) FULL(l) FULL(d) FULL(x) SWAP_JOIN_INPUTS(l) SWAP_JOIN_INPUTS(x) */

Hint for retrieving unmatched records for inLists in ordered lists.

Note: For unmatch operations, there is no difference between SMALL and LARGE inputs.

VldbTableAccessor.find.small

/*+ORDERED USE_NL(x d) INDEX(x) */

Hint for retrieving data using SELECT WHERE for SMALL datasets.

VldbTableAccessor.find.large

/*+ORDERED USE_HASH(x d) FULL(d) FULL(x) SWAP_JOIN_INPUTS(x) */

Hint for retrieving data using SELECT WHERE for LARGE datasets.


iWay Software