x

This section provides an overview of the environment and the iSM application that will be deployed.

1. Ensure that Slider is installed on your Hadoop cluster.
2. Ensure that Java is installed on your Hadoop cluster.
3. Install iSM on the Hadoop node.

   You can use any node within a cluster for the iSM installation.

4. Add the required libraries (.jar files) for iSM to run Hadoop Distributed File System (HDFS)/Avro components.

   For example, on the Cloudera Hadoop cluster that is used for this demo, you must add /opt/cloudera/parcels/CDH/jars/* to the iSM post classpath to enable HDFS/Avro services.

   You may not require all of the .jar files, but it is a good practice to simply add all of the .jar files just to be sure. Ensure to add the path that is specific to your environment as the above example is provided only for reference purposes.

5. Start your iSM instance on the Hadoop node, ensure it starts successfully and you can access the iSM Administration Console and SOAP ports.
6. Use your local iWay Integration Tools (iIT) running outside of the Hadoop cluster to connect to the remote iSM instance running on the Hadoop node.

   You can test run your process flows against this remote iSM instance and also deploy your applications to this instance.

7. If you plan to do HDFS write operations, ensure that you confirm that the directory is available.

   For example, you can run the hadoop fs -ls /user command to retrieve a list of the home directories, as shown in the following image.

   

8. The following is a simple process flow that writes a file to the HDFS on a Hadoop cluster.

   

   The write_file object is configured for the HDFS Emit Service, as shown in the following image.

   

   The following HDFS Emit Service parameters are critical in this configuration:

   • Hadoop Configuration. Location of the core-site.xml file, which contains the information on the Controller node and other relevant settings. You should locate this file first and then provide its path here.
   • Default File System. Pointer to the Controller node in your Hadoop cluster. For example, in this demo iSM is running on the iwhadoop3 node, but the Controller node is iwhadoop2. As a result you must specify iwhadoop2 as the value for this parameter.
9. When this process flow is executed, you can search for the files on the Hadoop cluster using the following command:

   hadoop fs -ls | grep file_pattern

   The file written can be on any node or split across nodes in the Hadoop cluster.

   As shown in the following image, two files with demo_### pattern have been written to the cluster.

   

10. For the simple application, we have packaged this process flow to be callable through a Schedule Provider into an iWay Integration Application (iIA).

    Note: The Path setting to include HDFS .jar files had to be added to an application and separately, as it is not inherited from the base configuration due to a different deployment template.

11. Deploy and test your application on the Hadoop node following the standard process of deploying, starting, and testing an iIA.

x

The iWay Service Manager (iSM) application must first be packaged to be deployable under YARN using Slider. For more information on how to create a Slider package, see Creating an iWay Service Manager Slider Package.

You will need to:

1. Create all of the scripts.
2. Update the values in each script according to your environment.
3. Create a .zip file with the described structure.

The final .zip file must either be created on your Hadoop node or moved to your Hadoop node.

Note: If you copy the sample scripts that are provided in this documentation, you will need to reformat them slightly to be compatible with Python syntax. You can also refer to the sample scripts provided within the zip file (iway7master.zip) along with this documentation.

When creating a tarball (tar archive) for iSM, you can reduce its size by removing any unused .jar files and directories. For example, if you are not using iWay Trading Partner Manager (TPM), then you can remove TPM components. If you are not using iWay Data Quality (DQ), then you can remove DQ components, and so on. This strategy can be used to reduce the overall size of the .zip file (package) for smaller cluster deployments.

x

This section describes how to install and deploy the sample Slider package.

1. To install the sample Slider package, run the following command:

   slider install-package --name IWAY7 --package iway7master.zip

   Note: This command has changed in the latest Slider documentation.

   To replace an existing package, add the --replacepkg attribute, as shown in the following example.

   slider install-package --name IWAY7 --package iway7master.zip --replacepkg

   If you do not have a Slider environment configured through the slider-client.xml and slider-env.sh files, then you must define additional properties using a command line.

   By default the slider-client.xml file contains all of the properties, which are commented by default. You will need to uncomment the properties if you are setting the Slider configuration in the file. Minimally, you will need to have the following property uncommented and set according to your environment.

   <property>
      <name>hadoop.registry.zk.quorum</name>
      <value>iwhadoop8:2181,iwhadoop9:2181,iwhadoop2:2181</value>
   </property>

   Note: The specific value should match your environment. This example is only for reference purposes.

   If the Slider is not configured, then you must add the corresponding manager and filesystem commands as shown in the following example. Ensure that your Slider is properly configured.

   slider install-package --name IWAY7 --package iway7master.zip --
   replacepkg --manager iwhadoop2:8033 --filesystem hdfs://iwhadoop2:8020

   Note: This example is only for reference purposes. The port and host values will vary for each system.

2. To create a Slider package, run the following command. Note that If you do not have the Slider configured, then you must define a set of properties on the command line.
   • If the Slider is configured, then run the following command:

     slider create iway7_1 --template appConfig-default.json --resources
     resources-default.json

   • If the Slider is not configured, then run the following command:

     slider create iway7_1 --template appConfig-default.json --resources
     resources-default.json --manager iwhadoop2:8032 --filesystem
     hdfs://iwhadoop2:8020

3. Navigate to the application monitoring console, which should be accessed by the URL for the control node in the cluster and using port 8088 (default).

   For example:

   http://iwhadoop2:8088/cluster

   In the application monitoring console you can see your installed application listed, which you can select to view more details, as shown in the following image.

   

   You can see on which node the application was started, as shown in the following image.

   

   You can also review the running statistics for the application, as shown in the following image.