x

The configuration file of event handlers is a list of all event handlers that are to be used.

<?xml version="1.0" encoding="utf-8"?>
<handlers>
	<handler class="first.desired.handler.Class">
		... handler configuration ...
	</handler>
	<handler class="second.desired.handler.Class">
		... handler configuration ...
	</handler>
	...
</handlers>

x

Asynchronous event handler; during NME operation it stores all necessary data to a specified directory. Only when the NME operation is completed, it initiates publishing of the data. If publishing of an event fails for any publisher, the publishing is stopped. When resumed, the event at which the failure occurred is published again with all publishers (that means some publishers may have distributed it two or more times). The preceding events are not published again.

<handler class="com.ataccama.nme.engine.event.handler.EventHandlerAsync">
	<persistenceDir>/persistence/directory</persistenceDir>
	<filter>
		...
	</filter>
	<publishers>
		<publisher class="first.desired.publisher.Class">
			... publisher configuration ...
		</publisher>
		<publisher class="second.desired.publisher.Class">
			... publisher configuration ...
		</publisher>
		...
	</publishers>
</handler>

where:

persistenceDir

Is the path to a directory where all the persisted data will be stored (temporarily, before they are published). This directory should not be used to store anything else and must not be shared by multiple event handlers.

filter

Can pre-select events that will be persisted (and, consequently, published).

publishers

Is the list of all publishers which will distribute the data into various systems, depending on their nature and configuration.

x

Filter is an element that allows pre-filtering the events so only certain events are actually accepted.

<filter>
	<filterExpression>meta.event_type = "UPDATE"</filterExpression>
	<entities>
		<entity name="party" layer="instance">
			<filterExpression>new.src_name != old.src_name</filterExpression>
		</entity>
		<entity name="party" layer="master" masterView="MasterView" />
	</entities>
</filter>

where:

filterExpression

Is a boolean iWay DQS expression that can be specified here. In that case, only the events for which the expression evaluates to true are accepted. It can be omitted to accept all events. Only metadata columns are available here.

entities

Is the second filtering tool that allows you to specify entities only from which the events will be accepted (all other events of entities will be filtered out). Each entity is identified by its name and layer (instance/master). For master entities (and only for master entities), they can be identified by its masterView. This has to be an existing entity that has not been already filtered out earlier (usually by a filter on the parent element). Each entity can have its own filterExpression that works exactly like the filterExpression on the filter, except all the data columns are available (old and new values of the changed records as well as metadata columns).

x

The delegating publisher can be used wherever a publisher is expected, but not doing the actual publishing. Instead, it is adding another kind of functionality, and for the publishing itself, using another publisher (delegate). This delegate can also be a delegating publisher, cascading this event handling chain one level further.

Filtering Publisher

Adds the possibility to filter the flow of incoming events to the underlying publisher.

<publisher class="com.ataccama.nme.engine.event.handler.publishers.
FilteringPublisher">
	<filter>
		...
	</filter>
	<delegate class="delegate.publisher.Class">
		...
	</delegate>
</publisher>

where:

filter

Is a filter.

delegate

Is another publisher that will be used for the actual publishing of the events that pass through the specified filter (this publisher can also be another delegating publisher).

Retrying Publisher

The Retrying Publisher adds the possibility to retry a failed attempt to publish an event. Each time a publishing of an event fails, one global retry is consumed to retry the publishing of this event after the specified retry delay. If there are no global retries left, the publishing fails like it would without this Retrying Publisher. Additionally, after the specified amount of consecutive publishing attempts that were all successful, one additional global retry is generated. This way, the reserve of global retries can grow (up to the specified maximum amount of retries) to be used later.

<publisher class="com.ataccama.nme.engine.event.handler.publishers.
RetryingPublisher">
	<globalRetries>5</globalRetries>
	<retryDelay>20</retryDelay>
<numberOfConsecutiveSuccessesGrantingRetry>10
</numberOfConsecutiveSuccessesGrantingRetry>
	<maximumRetries>30</maximumRetries>
	<delegate class="delegate.publisher.Class">
		...
	</delegate>
</publisher>

where:

globalRetries

Is the initial amount of retries.

retryDelay

Is the delay after a failed attempt to publish an event before retry (in seconds).

numberOfConsecutiveSuccessesGrantingRetry

Is the amount of consecutive successes that will generate another retry. 0 means no retries will be generated.

maximumRetries

Is the limit of retries in reserve.

x

Publishers do the actual publishing, and send the events in the way specified by their nature and configuration.

The following list describes the publishers.

• StdOutPublisher. This publisher outputs events to StdOut. It is useful for debugging and presenting.

  <publisher class="com.ataccama.nme.engine.event.handler.publishers.
  StdOutPublisher">
  	<transformer>
  		...
  	</transformer>
  </publisher>

• EventHttpSoapPublisher. This type of publisher constructs a SOAP message and then sends it through an HTTP protocol to a specified destination.

  <publisher class="com.ataccama.nme.engine.event.handler.publishers.
  EventHttpSoapPublisher">
  	<url>http://target:port/local/path</url>
  	<soapAction>SOAP_ACTION</soapAction>
  	<username>joe</username>
  	<password>crypted:AES:324r3kfs3opewir3mm2340</password>
  	<soapVersion>SOAP_1_1</soapVersion>
  	<timeout>5000</timeout>
  	<encoding>UTF-8</encoding>
  	<delayBetweenRequestsMs>0</delayBetweenRequestsMs>
  	<transformer>
  		...
  	</transformer>
  </publisher>

  where:

  username

  Is an optional entry for HTTP authentication.

  password

  Is an optional entry for HTTP authentication.

  soapVersion

  Is an optional entry, defaulting to SOAP 1.1. The allowed values are SOAP_1_1 and SOAP_1_2.

  timeout,

  Is an optional entry, with defaults listed in the sample configuration.

  encoding

  Is an optional entry, with defaults listed in the sample configuration.

  delayBetweenRequestsMs

  Is an optional entry, with defaults listed in the sample configuration.

  transformer

  Is the specification of a transformer element, an element that will transform the event into text (in this instance, the SOAP message).

• EventJmsPublisher. This type of publisher sends the message through Java Message Service.

  <publisher class="com.ataccama.nme.engine.event.handler.publishers.
  EventJmsPublisher">
  	<connectionName>JMS_CONNECTION</connectionName>
  	<destination>dynamicQueues/target_queue</destination>
  	<contentType>text/xml</contentType>
  	<headers>
  		<header name="HEADER_NAME" value="HEADER_VALUE" type="STRING/>
  	</headers>
  	<transformer>
  		...
  	</transformer>
  </publisher>

  where:

  connectionName

  Is the name of the JMS connection configured in the server configuration.

  destination

  Is the target queue/topic.

  contentType

  Sets the info header of content type. The default value is text/plain.

  headers

  Are the optional, set of headers for the JMS messages. The allowed types are: STRING, INT, LONG, and BOOLEAN.

  transformer

  Is the specification of a transformer element, an element that transforms the event into text.

• iSM Publisher. Compiles an XML message from every event it receives. Every XML message then prefixes with the length of the message in a form of a 4-byte binary integer, and sends it separately to a specified host and port.

  <publisher class="com.ataccama.nme.engine.event.handler.publishers.IsmPublisher">
  	<host>localhost</host>
  	<port>8888</port>
  	<targetSystem>SYSTEM</targetSystem>
  </publisher>

  where:

  host

  Is the target host.

  port

  Is the target port.

  targetSystem

  Is the targetSystem property in the message. This property is a template of general nested iWay DQS expressions.

  For example:

  <?xml version='1.0' encoding='UTF-8'?>
  <mds.Event targetSystem="SYSTEM">
  	<party changeType="UPDATE">
  		<attributes>
  			<source_id>100</source_id>
  			<master_id>2</master_id>
  			<src_name>John Smith</src_name>
  			... more data columns ...
  			<eng_active>true</eng_active>
  		</attributes>
  	</party>
  </mds.Event>

• EventSqlPublisher. This type of publisher executes an SQL statement on a database.

  <publisher class="com.ataccama.nme.engine.event.handler.publishers.
  EventSqlPublisher">
  	<commitSize>1000</commitSize>
  	<dataSource>DATA_SOURCE</dataSource>
  	<templates>
  		<t>
  			<entity name="party" layer="master" masterView="MasterView" />
  			<template>insert into TABLE values (${new.src_name},
  ${old.src_name})</template>
  		</t>
  		...
  	</templates>
  </publisher>

  where:

  commitSize

  Is the optional entry, defaulting to the value in the example. It specifies the number of events after which a database commit should be called. It also defines the batch size.

  dataSource

  Specifies the iWay MDS data source, which is a named database connection defined in server configuration.

  templates

  Unlike other publishers, the SQL publisher does not feature a transformer for creating target text messages. Instead, a set of SQL-specific templates is used to create SQL statements directly. This increases the performance of the publishing considerably. SQL templates have some additional constraints over the common templates: they only allow to bind DQS expressions to SQL variables. They do not allow changing the SQL statement itself.

  The following example is an acceptable template.

  insert into TEST (column1, column2) values (${old.src_name}, ${new.src_name})
  delete from TEST where column1=${
  	case (
  	meta.eventType is 'INSERT', 1,
  	meta.eventType is 'UPDATE', 2,
  		3) }
  { call PACKAGE_NAME.PL_SQL_FUNCTION_NAME(${parameter1}, ${parameter2}) }

  The following example is restricted.

  ${'INSERT'} into TEST values (1)
  insert into ${old.table_name} values (1)
  { call ${'PACKAGE_NAME'}.${PL_SQL_FUNCTION_NAME()} }

x

A transformer is an element that transforms an event into a string, which is a message that will be distributed by a publisher.

The following list describes the different Transformer types.

• Simple XML Transformer. This transformer transforms events to simple predefined XML. It does not require nor provide any template configuration.

  <transformer
  class="com.ataccama.nme.engine.event.handler.publishers.transformers.SimpleXmlTransformer">
  	<indent>true</indent>
  	<includeOldValues>true</includeOldValues>
  </transformer>

  where:

  indent

  Is optional, and if true (default), the XML is pretty-printed, or indented using \t.

  includeOldValues

  Is optional, and if true (default), the XML contains old values of attributes.

  The following XML is produced.

  <event>
  	<metadata>
  		<id>1001</id>
  		<event_type>UPDATE</event_type>
  		<entity>party</entity>
  		...
  	</metadata>
  	<attributes>
  		<source_id>306</source_id>
  		<src_name>John Smith</src_name>
  		...
  	</attributes>
  	<oldAttributes>
  		<source_id>306</source_id>
  		<src_name>John Smith Jr.</src_name>
  		...
  	</oldAttributes>
  </event>

• Expression Template Transformer. A transformer based on a template.

  <transformer
  class="com.ataccama.nme.engine.event.handler.publishers.transformers.
  ExpressionTemplateTransformer">
  	<entity name="party" layer="master" masterView="MasterView" />
  	<template>Some text with ${'DQS expressions'} inside it.</template>
  </transformer>

  where:

  entity

  Is an optional entry that declares the entity that flows into this transformer (and if there is an incoming event from a different entity, an exception is thrown). If omitted, all entities are accepted, but no data columns are available.

  template

  Is the intended text with the nested iWay DQS expressions.

  begin

  Is an optional entry, and can change the default begin mark of nested iWay DQS expressions.

  end

  Is an optional entry, and can change the default end mark of nested iWay DQS expressions.

• Multi Transformer. The multi transformer can join several transformers, each handling one entity to a single transformer that can handle all of them. This is the preferred way to deal with transforming several entities where the contents of the data columns are necessary.

  <transformer
  class="com.ataccama.nme.engine.event.handler.publishers.transformers.
  MultiTransformer">
  	<transformers>
  		<transformer class="first.transformer.Class">
  			...
  		</transformer>
  		<transformer class="second.transformer.Class">
  			...
  		</transformer>
  		...
  	</transformers>
  </transformer>

x

A template is an attribute type. It is basically a string type, but allows nested iWay DQS expressions that give you all the tools and expressive power of the iWay DQS engine. Expressions are denoted by ${ ... } notation by default.

<template>
	<entity name="party" layer="master" masterView="MasterView" />
	<template>This is a common string and ${'this will be evaluated as DQS
expression'}</template>
</template>

where:

entity

Specifies the entity this template applies to. You can use all the entity specific data columns in the expressions of the template.

template

Is the text template itself, with nested iWay DQS expressions. Since the expressions are part of a string, they have to be of a STRING type. This includes converting any columns of a non-string type to string (by toString() iWay DQS function) if necessary.

x

There are two types of iWay DQS expressions in event handling: general and entity specific.

• general. This type of iWay DQS expression only allows access to metadata of the event. These are available through the dot-source metadata. The available columns appear in the following table.

  Name	Type	Description
  id	LONG	Internal record ID.
  event_type	STRING	Type of event (INSERT/UPDATE/DELETE).
  entity	STRING	Name of the entity this event originated from.
  layer	STRING	The layer of the entity this event originated from (instance/master).
  master_view	STRING	Name of the entity master view this event originated from if it is a master entity, otherwise it is null.
  origin	STRING	The origin of the data for an instance entity, otherwise it is null.
  source_system	STRING	The source system of the data for an instance entity, otherwise it is null.
  activation_date creation_date deactivation_date deletion_date last_update_date last_source_update_date	DAY	The date the record was activated, created, deactivated, deleted, last updated, and last updated from source, respectively.
  activation_tid creation_tid deactivation_tid deletion_tid last_update_tid last_source_update_tid	LONG	ID of the logical transaction that activated, created, deactivated, deleted, last updated, and last updated from source the record, respectively.

  entity specific. This type of iWay DQS expression allows access to all metadata, such as general iWay DQS expressions. Furthermore, because it is entity specific, it allows access to all data columns of the record on which the change occurred (both old and new values). These are available through dot-sources old and new. In addition to all columns of that specific entity, the internal column eng_active is also available through these dot-sources.

x

EventHandler stores events in persistent storage to deliver at-least-once semantics. Such an approach requires disk space (this should be taken into account for disk sizing) that can be computed as follows:

required_disk_space = event_size x event_count.