Incremental Integration (Change Data Capture)

In this section:

The iWay Integration Suite also provides efficient solutions for incremental data load requirements at the enterprise level, including:


Top of page

x
iWay Change Data Capture (CDC) Solutions

In this section:

Using the iWay Integration Suite, you can take advantage of iWay Change Data Capture (CDC) solutions for managing incremental data loads. iWay CDC offers effective solutions to the challenges encountered when performing incremental loads from data source tables, systems, and data warehouses.

iWay CDC solutions are available for:



x
iWay Change Data Capture (CDC) Solution for Informix

The iWay Informix CDC Agent is a key component of the iWay CDC Solution for Informix, which enables integration between IBM Informix Dynamic Server (IDS) and other iWay accessible enterprise systems. The agent accomplishes this task by exploiting the IDS 11.5 CDC API, a significantly improved technology for Change Data Capture (CDC) over previous methods. The second component of this solution is the iWay Log Event Listener that runs on the target Oracle database server.

The combination of iWay Integration Technology and the IDS 11.5 CDC API provides:

This architecture exploits the IDS logging facilities, so that overall impact upon existing IDS systems is minimal. Security is ensured by limiting CDC data capture to only preselected schemas, tables and columns with an existing DBMS.



x
For More Information

See the following manual at http://ecl.informationbuilders.com/iway/index.jsp:

  • iWay Change Data Capture Solution for Informix


x
iWay Change Data Capture (CDC) Solution for Microsoft SQL Server

The iWay CDC Solution for Microsoft SQL Server consists of the iWay Log Event Adapter for Microsoft SQL Server, which is used to integrate a Microsoft SQL 2000/2005/2008 database with an Oracle and SQL database with near real-time data replication. The iWay Log Event Adapter for Microsoft SQL Server consists of several components. The key component is the SQL Server wrapper (C# .NET client), which interacts with the server-side components.



x
For More Information

See the following manual at http://ecl.informationbuilders.com/iway/index.jsp:

  • iWay Log Event Adapter for Microsoft SQL Server


x
iWay Change Data Capture (CDC) Solution for Oracle

The iWay CDC Solution for Oracle consists of the iWay Log Event Adapter for Oracle. This adapter is designed to replicate data changes from Oracle 10g/11g to Oracle 10g/11g or SQL 2000, 2005, or 2008. The iWay Log Event Adapter for Oracle uses a listener to capture the changed data from the Oracle redo logs using Oracle LogMiner.

The following types of supported operations can be captured from the source Oracle 10g/11g database table:



x
For More Information

See the following manual at http://ecl.informationbuilders.com/iway/index.jsp:

  • iWay Log Event Adapter for Oracle

Top of page

x
iWay High Water Mark (HWM) Listeners

In this section:

iWay High Water Mark (HWM) listeners provide the ability to read new or modified data based on changes that take place over a specified time interval.

iWay HWM listeners are available for:



x
iWay Relational Database High Water Mark Listener

The iWay Relational Database High Water Mark listener (RDBHWM) provides the ability to read records from an RDBMS if a column or set of columns is defined with increasing values, such as a timestamp of last change.

The iWay RDBHWM operates in real-time to acquire newly inserted or updated rows from a relational database. In order to identify the desired rows, a column (or concatenation of columns) must exist, which contains an increasing high water mark value.

The listener saves the highest HWM value read with each SELECT, and constructs the subsequent query using the saved value. Unlike the iWay RDBMS listener, no deletes or updates are necessary to prevent the rereading of processed rows. Because the iWay RDBHWM listener can access the database as read-only, the performance of the iWay RDBHWM listener is faster than the iWay RDBMS listener.

Support for UNION

The iWay RDBHWM listener can also address more sophisticated SQL in which the HWM predicate appears multiple times in the SELECT statement. This situation may arise in situations in which the SQL addresses multiple databases in JOIN or UNION statements. The net effect is to allow support for more complex SQL and as a consequence, simplify the resulting application.



x
For More Information

See the following manual at http://ecl.informationbuilders.com/iway/index.jsp:

  • iWay Service Manager Protocol Guide


x
iWay LDAP High Water Mark Listener

The iWay LDAP High Water Mark listener provides the ability to read new or modified entries from an LDAP server, provided there is an attribute which steadily increases over time. Attributes such as an employee number or a timestamp may serve this purpose.

The iWay LDAP High Water Mark (HWM) listener follows the HWM design pattern by acquiring new or changed records (LDAP entries) from a data store (the LDAP server) that meet a specific, evolving criteria. The data that is returned is determined by the scope and the specified filter. To avoid retrieving duplicate entries, the iWay LDAP HWM listener keeps track of the highest value of the filter attribute which has been read. Usually the filter value is the creation or modification time of the entry, but any unique and steadily increasing value can be used. Each entry satisfying the filter is returned as an iWay XML document for further processing in an iWay Service Manager channel.

The iWay LDAP HWM listener is triggered (begins processing messages) when an entry is inserted or modified in the LDAP directory that is based on an increasing value of an attribute. The iWay LDAP HWM listener works similar to the iWay Relational Database High Water Mark listener.

A diagnostic tool (testldap), is provided in the extension to manually access and explore the LDAP directory from the iSM command console.



x
For More Information

See the following manual at http://ecl.informationbuilders.com/iway/index.jsp:

  • iWay Service Manager Protocol Guide

Top of page

x
iWay Schedule and Calendar Providers

In this section:

This section describes the iWay Schedule and Calendar providers that are available in iWay Service Manager (iSM).



x
Schedule Provider

The Schedule provider allows you to schedule defined iSM processes and tasks for execution (separate from the channel architecture) at specific times of the day, week, or month.



x
For More Information

See the following manual at http://ecl.informationbuilders.com/iway/index.jsp:

  • iWay Service Manager User's Guide


x
Calendar Provider

The Calendar provider allows you to define 18 months (starting with the current month) of Holiday/Skip dates, which when used with the Schedule provider, allows you to fine tune your scheduler execution.

Once configured, the Calendar provider can be:

  1. Exported to a centralized file system.
  2. Imported from a centralized file system and used within this iWay Service Manager (iSM) instance.
  3. Imported from a centralized iSM and used within this iSM instance.
  4. Accessed from a centralized iSM.

This allows the Calendar provider to be shared in real time by any iSM instance to which it has access.



x
For More Information

See the following manual at http://ecl.informationbuilders.com/iway/index.jsp:

  • iWay Service Manager User's Guide

Top of page

x
iWay Real Time Data Replication (RTDR) Extension

In this section:

The iWay Real Time Data Replication (RTDR) extension is composed of the SQL Batch Insert Iterator object and the Data Integration object.

In the simplest use case, the iWay RTDR extension can be used to replicate data from a source database to a target database. The source and target databases can be the same type or different types. Both databases must exist prior to replication.

In a more complex use case, the iWay RTDR extension can be used to extract data one row at a time from a source database, cleanse or transform the data, and then insert the row into another database. The source and target databases must exist prior to execution.



x
For More Information

See the following manual at http://ecl.informationbuilders.com/iway/index.jsp:


iWay Software