Direct Indexing Examples

The following example shows iWay Service Manager (iSM) retrieving records from a relational database and feeding each row to the search appliance as an XML document.

Top of page

Example: Indexing RDBMS Data

When the search appliance returns one of the relational database documents in a search result, the URL of the result will require the database and return an XML document with the record, as shown in the following illustration.

There are two tables in the database:

GDITestData contains the data you are indexing. Field1 is a unique key that increases by increments of 1 for each new record. The following table illustrates the GDITestData table. It has three columns, which represent Field1, Field2, and Field3, and three rows of related data.

Field1

Field2

Field3

1

xyz

abc

2

abc

xyz

3

pdq

asap

Field1	Field2	Field3
1	xyz	abc
2	abc	xyz
3	pdq	asap

Use a second table, GDIField1Max, so that the RDBMS listener can read rows from GDITestData non-destructively. The table below illustrates the GDIFieldMax table, which contains one column, entitled maxField1, and one row containing its data.

maxField1

0

maxField1
0

The RDBMS Listener is configured to retrieve all rows from GDITestData with a Field1 value greater than the value of maxField1. As a post-query, update maxField1 with the last value of field1.

Configure an RDBMS listener named RDBMS-to-Index, as shown in the following table:

Property

Value

Notes

Driver

com.microsoft.jdbc.sqlserver.SQLServerDriver

JDBC driver appropriate for your data source. SQL Server is shown here as an example.

URL

jdbc:microsoft:sqlserver://localhost:1433;
databasename=testdb;selectmethod=cursor

For connection to JDBC data source. The syntax of the URL depends on your data source.

User

You must have permission to read GDITestData and to read and update GDIField1Max.

Password

Password for the user ID given.

Table

Leave blank, because you will use a statement instead.

Maximum Rows

1

Treats each row of the result set as an individual document.

SQL Query

SELECT field1, field2, field3 FROM GDITestData WHERE field1 > (SELECT MAX(maxField1) from GDIField1Max)

Retrieves all rows where the key is greater than the last maximum value.

SQL Post-query

UPDATE GDIField1Max SET maxField1 = ^field1

^field1 indicates the value of Field1 in the row currently being processed.

Delete Keys

Leave blank.

Generated XML Format.

field

Root Name

If left blank, defaults to listener name.

Property	Value	Notes
Driver	com.microsoft.jdbc.sqlserver.SQLServerDriver	JDBC driver appropriate for your data source. SQL Server is shown here as an example.
URL	jdbc:microsoft:sqlserver://localhost:1433; databasename=testdb;selectmethod=cursor	For connection to JDBC data source. The syntax of the URL depends on your data source.
User		You must have permission to read GDITestData and to read and update GDIField1Max.
Password		Password for the user ID given.
Table		Leave blank, because you will use a statement instead.
Maximum Rows	1	Treats each row of the result set as an individual document.
SQL Query	SELECT field1, field2, field3 FROM GDITestData WHERE field1 > (SELECT MAX(maxField1) from GDIField1Max)	Retrieves all rows where the key is greater than the last maximum value.
SQL Post-query	UPDATE GDIField1Max SET maxField1 = ^field1	^field1 indicates the value of Field1 in the row currently being processed.
Delete Keys		Leave blank.
Generated XML Format.	field
Root Name		If left blank, defaults to listener name.

Note: For more information, see your RDBMS listener documentation.

When the RDBMS-to-Index listener runs, it creates three XML documents, one for each of the three rows in the sample data:

<RDBMS-to-Index table="GDITestData">
     <row>
           <Field1 type="2">1</Field1>
           <Field2 type="12">xyz</Field2>
           <Field3 type="12">abc</Field3>
     </row>
</RDBMS-to-Index>

Note: The above text may vary slightly depending on your version of iEI.

To send these documents to the search appliance for indexing, you must use an iEI Feeder service as part of your channel configuration.

Each indexed record must be identified by a unique URL. The Field1 value from the table is used to create this unique URL for each indexed record. Select Registry, then under Variables, select Parameters to create a new parameter that can contain the Field1 value.

The Parameter information appears in the right pane, as shown in the following image.

Click Add to add a new parameter section.

The Parameters information appears in the right pane, as shown in the following image.

Add a source parameter to identify the process that sends this document to the index. Add a second parameter, recordkey, which identifies the record being indexed and makes the URL unique.

Click Next.

The Registers information appears in the right pane, as shown in the following image.

Provide a name for the parameter group and optional description, then click Finish.

Configure the iEI Feeder Service as shown in the following table.

Property

Value

Notes

Search Appliance URL *

http://server:port/directory

For example,

http://server1:19900/xmlfeed

URL at which the search appliance receives XML feeds.

Feed Datasource *

hostname

For example, server1.

Search appliance uses data source names to group feeds logically. Use a single data source name for related processes that send feeds to the search appliance.

This is the name of the directory where your index files are created.

Base URL *

http://host:port/query string

For example,

http://*:9996/audit

Is the base of the URLs that iEI assigns to the documents it sends to the search appliance. Ordinarily, it refers to a listener.

Batch Size *

3

Adds three documents to the feed before sending. Override this using the Idle Limit parameter.

Secure Search? *

true or false

Secures feeds with HTTP Basic Authentication. You must have an authorization exit configured.

Feed Type *

FULL or INCREMENTAL

Tells the search appliance to add these rows to an existing data source, rather than replacing rows from that data source. Use a FULL feed to overwrite data in the index.

Action *

add or delete

Action to be applied to this record in the search appliance. If add (default) is selected, the URL and content will be added to the index. If delete is selected, the URL is removed from the index.

Content-Type *

text/plain

Even though this process sends XML documents to the search appliance, strip carets and quotes from XML tags to improve indexing.

Idle Limit *

60

Number of seconds a feed containing fewer records than the limit set by the batch size parameter waits before being sent.

Strip XML Tags? *

true or false

The search appliance does not index data inside of XML tags, such as element names, attributes, and attribute values. When you want to index this data, strip tags and set Content-Type to text/plain.

Output Type *

input or status

Determines whether the agent will output its input, or whether it should output an XML status document.

Property	Value	Notes
Search Appliance URL *	http://server:port/directory For example, http://server1:19900/xmlfeed	URL at which the search appliance receives XML feeds.
Feed Datasource *	hostname For example, server1.	Search appliance uses data source names to group feeds logically. Use a single data source name for related processes that send feeds to the search appliance. This is the name of the directory where your index files are created.
Base URL *	http://host:port/query string For example, http://*:9996/audit	Is the base of the URLs that iEI assigns to the documents it sends to the search appliance. Ordinarily, it refers to a listener.
Batch Size *	3	Adds three documents to the feed before sending. Override this using the Idle Limit parameter.
Secure Search? *	true or false	Secures feeds with HTTP Basic Authentication. You must have an authorization exit configured.
Feed Type *	FULL or INCREMENTAL	Tells the search appliance to add these rows to an existing data source, rather than replacing rows from that data source. Use a FULL feed to overwrite data in the index.
Action *	add or delete	Action to be applied to this record in the search appliance. If add (default) is selected, the URL and content will be added to the index. If delete is selected, the URL is removed from the index.
Content-Type *	text/plain	Even though this process sends XML documents to the search appliance, strip carets and quotes from XML tags to improve indexing.
Idle Limit *	60	Number of seconds a feed containing fewer records than the limit set by the batch size parameter waits before being sent.
Strip XML Tags? *	true or false	The search appliance does not index data inside of XML tags, such as element names, attributes, and attribute values. When you want to index this data, strip tags and set Content-Type to text/plain.
Output Type *	input or status	Determines whether the agent will output its input, or whether it should output an XML status document.

Note: Parameters with an asterisk (*) are required.

The iEI Feeder Service creates a URL for each document it processes using user-defined parameters. Each parameter is appended with its value to the base URL as part of the query string. The URL must contain all the data that iSM requires to create the result document when a user clicks a search result.

In the Parms column, click parms for the newly created iEI Feeder Service, as shown in the following image.

Click Add to add a parameter group to the iEI Feeder Service, as shown in the following image.

Select RDBMS_parms, and then click Finish, as shown in the following image.

Before the search appliance can process your feed, you must add the base URL to the search appliance crawl list.
For more information, see Configuring the Search Appliance.

When you start the RDBMS-to-Index listener, the three rows of data are processed, added to a feed, and sent to the search appliance. In approximately half an hour, the documents begin to appear in search results. Their URLs will look like the following: http://iSM hostname:9996/iEI?source=RDBMS-to-Index&recordkey=1.
Next, you must configure a listener that can process requests for these documents.
Configure the listener on port 9996, to match the port you assigned to the Base URL. Since you are not using SSL or basic authentication in this example, you can use the default values for all other parameters.
When the listener receives a request, you want to retrieve the requested row from the GDITestData table. You can use the iwSQL agent (XDSQLAgent), which executes an SQL statement using parameters from input and returns the result as an XML document.
The listener converts the URL query string to a simple XML document:
```
<post>
      <source>RDBMS-to-Index</source>
      <recordkey>1</recordkey>
</post>
```
Thus, the SQL agent can use the key from the URL as a parameter for the query.

Configure the SQL Agent as shown in the following table.

Property

Value

Notes

SQL DML

SELECT field1, field2, field3 FROM GDITestData WHERE field1 = ?key

?key refers to a user-defined parameter, whose value is set to the record key in the input document.

Output Format

Field

JDBC Driver

com.microsoft.jdbc.sqlserver.SQLServerDriver

This value depends on your data source.

Data Source URL

jdbc:microsoft:sqlserver://localhost:1433;
databasename=testdb;selectmethod=cursor

This value depends on your data source.

Property	Value	Notes
SQL DML	SELECT field1, field2, field3 FROM GDITestData WHERE field1 = ?key	?key refers to a user-defined parameter, whose value is set to the record key in the input document.
Output Format	Field
JDBC Driver	com.microsoft.jdbc.sqlserver.SQLServerDriver	This value depends on your data source.
Data Source URL	jdbc:microsoft:sqlserver://localhost:1433; databasename=testdb;selectmethod=cursor	This value depends on your data source.

You must create another Parameter Group that contains your key variable with XPATH to the recordkey value of the incoming post document. This parameter group has to be added to the iwSQL Service that you are creating and will be used to retrieve the information from the underlying DB table.

Save your changes and start the listener.

iWay Software