Input and Output Formats

CSV Format

When you use CSV format, data is organized in the same way as CSV files (comma-separated values). Data records are separated by rows that contain fields separated by a special character.

CSV format has the following parameters:

contentId. Used as the identifier of the data flow in multipart messages.
encoding. Used for the CSV-formatted input/output. The default value is UTF-8.
useHeader. Boolean value indicating whether the CSV header (the first line containing column names) is expected or not. The default value is True.
stepId. Not required. Defines the default name of the Integration Input Step or Integration Output Step (depending on whether it is used in the input or output section). It is used in case no stepId is specified for the individual column.
lineSeparator. Contains the character sequence used as the line separator in the CSV file (for example, "\n\r").
fieldSeparator. Contains the character used as the separator between individual fields on the same line, that is, in the same record (for example, “;”).
stringQualifier. Not required. Contains the character used to define string data.
stringQualifierEscape. Not required. Contains the character used to escape the stringQualifier.
Maximal Length. Required. Sets the maximal length of the input line. If the input contains a line that has more characters, processing will end with an error. Supply the value 0 to ignore this setting.
columns. Element that contains the list of csvColumn elements with CSV column definitions.

csvColumn

The element column defines the binding of the fields from the CSV stream and columns of the input and output steps.

It has the following parameters:

name. Contains the name of the column in the defined Integration Input Step or Integration Output Step (depending on whether it is used in the input or output section).
csvName. Not required. Defines the name of the field (column) in the CSV file. If it is not specified, the csvName will have the same value as name.
stepId. Not required. Is the name of the Integration Input Step or Output Step that is used to read or write the column value. If it is not specified, the global stepId defined in the format element will be used.

Example:

<format class="cz.adastra.cif.online.config.CsvFormat" fieldSeparator=";" lineSeparator="\r\n" stepId="test_in">
     <columns>
       <column name="iparam1" csvName="param1" />
       <column name="iparam2" csvName="param2" />
       <column name="iparam3" csvName="param3" />
     </columns>
</format>

XML Format

XML format describes the structure of input and output data. The content of the element basically describes the structure of the XML document that is read from the input (or written to the output). The description may contain section and column elements. Sections may contain other section and column elements. The column element describes the mapping between an element or attribute content from the XML document on input (or output) and the column defined in the step of the configuration Plan or Component.

When the XML document is parsed, new data records are created. A new data record is always connected to a step identified by the stepId attribute. Data records are bounded by multiple enabled sections. As the document is parsed each time the element repeats, new data records are created and sent to the step identified by the stepId attribute. The section does not have to define the stepId attribute when it is already defined in one of the parent section elements.

XML format has the following parameters:

contentId. Used as an identifier of data flow in multipart messages.
namespace. Defines the namespace URI used for all elements in the XML input (or output) document.
rootSection. Element that defines the XML element that is the only child of the root element of the XML input (or output) document being processed. This element contains subelements with all the data from the request (or response).

rootSection (XML and SOAP)

The rootSection element differs from the section element only in name. It is used to describe the top element of the request (or response) content structure. When it is used inside an XML format definition, it describes the top-level element of the XML document. When used in a SOAP format definition, it describes the only allowed child of the SOAP Body element.

The inner structure of this element is the same as described in the section element.

section (XML and SOAP)

The section element describes non-leaf elements in the XML document. It may contain other nested section elements or column definitions. To describe an XML element with repeating occurrence, set the attribute multiple to true. The whole structure (including subsections and columns) may be repeated. When you use multiple enabled sections, a new data record is created for each repetition of the section in the source document.

The section element (XML and SOAP) has the following parameters:

name. Required. Defines the name of the XML element.
multiple. Boolean value indicating whether the XML element described by this section may be used repeatedly or not. The default value is False.
ignore. Boolean value indicating whether the section should be ignored (skipped) or not. The default value is False.
stepId. Not required. Name of the Integration Input Step (or Output Step) in the referenced configuration Plan or Component, which is used to read (or write) data values. If stepId is not specified, the parent stepId will be used.
primaryKeyColumn. Name of the column in which the value of the internal identifier (generated when the document is processed) should be stored. This column must be defined in the referenced configuration and must be of type long.
You can use this column value to correctly compose and decompose XML documents with two or more nested data structures. For example, you may have a list of people, and each person may have one or more addresses. You can reference the primaryKeyColumn in a record from one of the foreignColumns when defined in the address subelement.
columns. Contains a list of column definitions.
foreignColumns. Contains a list of foreignColumn definitions.
sections. Contains a list of nested section definitions.

column (XML and SOAP)

Defines the XML element or XML attribute containing the value that will be mapped to the column value of the Integration Input Step (or Output Step) in the referenced configuration Plan or Component.

It has the following parameters:

name. Required. Name of the column defined in the step.
nodeName. Not required. Name of the XML element or XML attribute where the value is read from (or written to).
type. Required. Defines the column data type.
attribute. Required. Boolean value that specifies whether the column value is stored in the attribute or as content of the element.
multiple. Boolean value that specifies whether the column may appear repeatedly or not (that is, whether it is an array or not).
There are some restrictions when you use multiple columns:
- The parent section cannot contain any other columns or sections.
- The parent's parent section must have a defined primaryKeyColumn.
- The Integration Input Step/Output Step corresponding to the multiple columns must define a long column named fkId, where the value from the parent's parent primaryKeyColumn will be stored.
ignore. Boolean value that specifies whether the column should be ignored (skipped) or not. The default value is False.

foreignColumn (XML and SOAP)

Foreign columns are used with an XML document that has multiple sections, and a value defined outside of a multiple section needs to be inserted in the record. For example, you need to insert the parent record identifier in the nested records in order to correctly compose the XML output document.

You can define foreignColumns only when describing the input data.

It has the following parameters:

name. Required. Name of the target column defined in the step.
stepId. Required. Identification of the step where the foreign value is read from.
column. Required. Name of the foreign source column.

Example

This example includes a sample XML configuration file, an XML request document, and data records.

XML Configuration File

<format class="cz.adastra.cif.online.config.XmlFormat"
  namespace="http://www.ataccama.com/ws/dqc">
  <rootSection name="request_CUSTOMER" multiple="false">
    <sections>
      <section name="request_FO" multiple="true" stepId="person_in"
        primaryKeyColumn="person_id">
         <columns>
           <column name="src_first_name" nodeName="first_name"
             attribute="false" type="string" />
           <column name="src_last_name" nodeName="last_name"
             attribute="false" type="string" />
         </columns>
         <sections>
           <section name="request_ADDR" multiple="true" stepId="address_in">
             <columns>
               <xmlColumn name="src_zip" nodeName="zip" attribute="false"
                 type="integer" />
               <xmlColumn name="src_city" nodeName="city" attribute="false"
                 type="string" />
               <xmlColumn name="src_street" nodeName="street" attribute="false" 
                 type="string" />
               <xmlColumn name="src_lrn" nodeName="lrn" attribute="false"
                 type="integer" />
               <xmlColumn name="src_sn" nodeName="sn" attribute="false"
                 type="integer" />
             </columns>
             <foreignColumns>
               <foreignColumn name="person_id" stepId="person_input"
                 column="person_id" />
               <foreignColumn name="lastname" stepId="person_input"
                 column="src_last_name" />
             </foreignColumns>
           </section>
         </sections>
      </section>
    </sections>
  </rootSection>
</format>

XML Request Document

<request_CUSTOMER xmlns="http://www.ataccama.com/ws/dqc">
  <request_FO>
    <src_first_name>Ferda</src_first_name>
    <src_last_name>Mravenec</src_last_name>
    <request_ADDR>
      <src_zip>18600</src_zip>
      <src_city>Praha</src_city>
      <src_street>Karolínská</src_street>
      <src_lrn>654</src_lrn>
      <src_sn>2</src_sn>
    </request_ADDR>
    <request_ADDR>
      <src_zip>18000</src_zip>
      <src_city>Praha</src_city>
      <src_street>Husitská</src_street>
      <src_lrn>1252</src_lrn>
      <src_sn>28</src_sn>
    </request_ADDR>
  </request_FO>
  <request_FO>
    <src_first_name>Brouk</src_first_name>
    <src_last_name>Pytlík</src_last_name>
    <request_ADDR>
      <src_zip>18600</src_zip>
      <src_city>Praha</src_city>
      <src_street>Karolínská</src_street>
      <src_lrn>654</src_lrn>
      <src_sn>2</src_sn>
    </request_ADDR>
  </request_FO>
</request_CUSTOMER>

Data Records

 person_in                 address_in 
1 | Ferda | Mravenec      18600 | Praha | Karolínská |  654 |  2 | 1 | Mravenec
2 | Brouk | Pytlik        18000 | Praha | Husitská   | 1252 | 28 | 1 | Mravenec
                          18600 | Praha | Karolínská |  654 |  2 | 2 | Pytlík

SOAP Format

SOAP format is almost the same as XML format. It is extended only by the soapAction parameter to define the header parameter according to the SOAP specification. Other content, for example, the rootSection element structure, is the same as that described in XML Format.

It has the following parameters:

soapAction. Required when used inside the input element. Otherwise, it is ignored. The value defines the name of the SOAP action according to the SOAP standard.
rootSection. Element that defines the XML element that is the only child of the SOAP body being processed. This element contains subelements with all the data from the request (or response). By convention, the rootSection element in the input should have the same name as soapAction.

Example:

<format class="cz.adastra.cif.online.config.SoapFormat" soapAction="customerAction">
  <rootSection name="request_CUSTOMER" multiple="false"
    namespace="http://www.ataccama.com/ws/dqc">
    <sections>
      <section name="request_FO" multiple="true" stepId="person_in"
        primaryKeyColumn="person_id">
         <columns>
           <xmlColumn name="src_first_name" nodeName="first_name" attribute="false" 
             type="string" />
           <xmlColumn name="src_last_name" nodeName="last_name" attribute="false" 
             type="string" />
         </columns>
         <sections>
           <section name="request_ADDR" multiple="true" stepId="address_in">
             <columns>
               <column name="src_zip" nodeName="zip" attribute="false"
                 type="integer" />
               <column name="src_city" nodeName="city" attribute="false"
                 type="string" />
               <column name="src_street" nodeName="street" attribute="false"
                 type="string" />
               <column name="src_lrn" nodeName="lrn" attribute="false"
                 type="integer" />
               <column name="src_sn" nodeName="sn" attribute="false"
                 type="integer" />
             </columns>
             <foreignColumns>
               <foreignColumn name="person_id" stepId="person_in"
                 column="person_id" />
               <foreignColumn name="lastname" stepId="person_in"
                 column="src_last_name" />
             </foreignColumns>
           </section>
         </sections>
      </section>
    </sections>
  </rootSection>
</format>

Multipart Format

Multipart is a special format used to execute multiple requests as a single request. It has one or more parts, and each part is handled as a separate request. Unlike several simple requests, all multipart parts (requests) use the same processing context. This feature is used to process data directly from a database, where each part is used for one database table.

It has the following parameter:

partFormats. Element that contains a list of partFormat elements.

Content of the element partFormat is the same as any other format element. Each part may have different data structures (for example, CSV, XML). The kind of format used is defined by the class attribute.

The element partFormat has only a different name and one more element parameter called contentId. The contentId parameter correctly identifies parts in the input and output that correspond to each other.

Example:

<format class="cz.adastra.cif.online.config.MultipartFormat">
  <partFormats>
    <partFormat class="cz.adastra.cif.online.config.CsvFormat" 
      contentId="first_part" fieldSeparator=";" lineSeparator="\n">
      <columns>
        <column name="param" stepId="multiecho1_in"/>
      </columns>
    </partFormat>
    <partFormat class="cz.adastra.cif.online.config.CsvFormat" 
      contentId="second_part" fieldSeparator=";" lineSeparator="\n">
      <columns>
        <column name="param" stepId="multiecho2_in"/>
      </columns>
    </partFormat>
  </partFormats>
</format>