Avro RPC HTTP Emit Service

Syntax:

com.ibi.agents.XDAvroRpcHttpEmitAgent

Description:

This service converts an Avro RPC request from XML into Avro binary format, sends it over HTTP, and optionally returns the response. An Avro Protocol file describes the possible interactions with the Avro server. The Message Name parameter selects which interaction to execute.

The path to the Avro Protocol file can be a regular path in the file system, or a URL starting with hdfs://, which indicates the file is in the Hadoop file system. When the Hadoop file system is used, the parameters Hadoop Configuration and Default File System can be optionally specified, otherwise they are ignored.

The Avro Protocol declares a request as an array of named parameters with their types. From this, Avro derives the request schema as an anonymous record, each field of the record being one of the parameters. The input document must be an XML document that obeys that schema. For more information on the conventions of the XML representation of an Avro record, see the Avro File Read service.

The service can return the response, a status document, or the input document. The last two options make more sense for one-way messages.

The output will obey the response schema of the message if the service is configured to return the response and the call is successful. This is simpler than the request because the Avro Protocol defines the response schema directly with no derivation. For one-way messages, the response schema is equivalent to null. If the call is not successful, the response will obey the errors schema. This is a union that implicitly always includes the string schema. The appropriate schema will be stored as extra metadata in the output document. For example, this can serve as a default for the schema in the Avro File Emit service.

The service refers to an HTTP Client Provider which is used to manage the HTTP client connections. The Provider must be created prior to the service. This can be done in the iSM console by selecting Pooling Providers in the left menu of the Server pane.

Parameters:

The following table lists and describes the parameters of the Avro RPC HTTP Emit service.

Parameter
Descriptions
Target URL
URL to post this information to, for example:
http://thehost:9876
Avro Protocol
Path to the Avro Protocol file. The standard extension is .avpr
Hadoop Configuration
Path to the Hadoop configuration file, normally core-site.xml
Default File System
In some Hadoop environments, this should be specified as the URI of the namenode. For example:
hdfs://[your namenode]
Message Name
Selects the message within the Avro Protocol.
HTTP Client Provider
HTTP Client Provider (Pooling Provider) that manages HTTP connections for this emitter.
Cookie Store Name
Allows thread-specific management of cookies. If a name is not specified, a cookie store global to the HTTP Client provider will be used.
User ID
User ID for authentication challenges.
Password
Password for authentication challenges.
Domain
Domain for NTLM authentication challenges. Note that to use NTLM, you must enable connection persistence.
Request Header Namespace
Special register namespace from which HTTP headers for the outgoing request will be taken. Select Default Namespace to send HDR type registers with no namespace prefix, or you can supply a namespace prefix here. The None option means that no special registers will be sent as HTTP headers.
Response Header Namespace
Special register namespace into which HTTP headers from the incoming response will be saved. Select Default Namespace to create special registers with no namespace prefix, or you can supply a namespace prefix here.
Excluded Headers
Comma delimited list (case insensitive) of headers that should not be sent with a request, even if it is found in a request header namespace.
Ask for Compressed Response
If set, requests will set the Accept-Encoding header to indicate that the client can accept a compressed response, as described in RFC-2616. If the response has a compressed content encoding, the client will automatically inflate.
Compress Request
If set, the HTTP request entity will be compressed using the selected encoding and the Content-Encoding header will be set accordingly.
Replace Connection?
If false, the connection will not be returned to the connection pool immediately. The identifier of the connection will be stored in the httpclient-key special register and the connection can be handled by the HTTP Client Manager agent.
Maximum HTTP Client Manager Delay
Maximum time the HTTP Client Manager can take to deal with a particular connection before it is automatically aborted. The format is [xxh][xxm]xx[s]. The default is 60 seconds.
Try Expect/Continue Handshake?
If selected, the client will send the HTTP Expect: 100-continue header and await HTTP 100 response before sending the request body.
Chunk Encoded Request?
If true, the request entity will be sent with chunk encoding.
Maximum Request Size
Maximum size, after compression, of a request entity that can be sent with this emitter. 0 means no maximum, and a blank field will default to 256KB.
Maximum Response Size
Maximum size of a response entity that can be received by this emitter. 0 means no maximum, and a blank field will default to 256KB
Persistence
If selected, ask the server to maintain the connection.
Response Timeout value in Seconds
Seconds to wait for a response before signaling an error.
Return
Return from this agent. Select input to return an input document, status for an XML document with transaction parameters and status, or response to capture output from the server.

Parameter	Descriptions
Target URL	URL to post this information to, for example: http://thehost:9876
Avro Protocol	Path to the Avro Protocol file. The standard extension is .avpr
Hadoop Configuration	Path to the Hadoop configuration file, normally core-site.xml
Default File System	In some Hadoop environments, this should be specified as the URI of the namenode. For example: hdfs://[your namenode]
Message Name	Selects the message within the Avro Protocol.
HTTP Client Provider	HTTP Client Provider (Pooling Provider) that manages HTTP connections for this emitter.
Cookie Store Name	Allows thread-specific management of cookies. If a name is not specified, a cookie store global to the HTTP Client provider will be used.
User ID	User ID for authentication challenges.
Password	Password for authentication challenges.
Domain	Domain for NTLM authentication challenges. Note that to use NTLM, you must enable connection persistence.
Request Header Namespace	Special register namespace from which HTTP headers for the outgoing request will be taken. Select Default Namespace to send HDR type registers with no namespace prefix, or you can supply a namespace prefix here. The None option means that no special registers will be sent as HTTP headers.
Response Header Namespace	Special register namespace into which HTTP headers from the incoming response will be saved. Select Default Namespace to create special registers with no namespace prefix, or you can supply a namespace prefix here.
Excluded Headers	Comma delimited list (case insensitive) of headers that should not be sent with a request, even if it is found in a request header namespace.
Ask for Compressed Response	If set, requests will set the Accept-Encoding header to indicate that the client can accept a compressed response, as described in RFC-2616. If the response has a compressed content encoding, the client will automatically inflate.
Compress Request	If set, the HTTP request entity will be compressed using the selected encoding and the Content-Encoding header will be set accordingly.
Replace Connection?	If false, the connection will not be returned to the connection pool immediately. The identifier of the connection will be stored in the httpclient-key special register and the connection can be handled by the HTTP Client Manager agent.
Maximum HTTP Client Manager Delay	Maximum time the HTTP Client Manager can take to deal with a particular connection before it is automatically aborted. The format is [xxh][xxm]xx[s]. The default is 60 seconds.
Try Expect/Continue Handshake?	If selected, the client will send the HTTP Expect: 100-continue header and await HTTP 100 response before sending the request body.
Chunk Encoded Request?	If true, the request entity will be sent with chunk encoding.
Maximum Request Size	Maximum size, after compression, of a request entity that can be sent with this emitter. 0 means no maximum, and a blank field will default to 256KB.
Maximum Response Size	Maximum size of a response entity that can be received by this emitter. 0 means no maximum, and a blank field will default to 256KB
Persistence	If selected, ask the server to maintain the connection.
Response Timeout value in Seconds	Seconds to wait for a response before signaling an error.
Return	Return from this agent. Select input to return an input document, status for an XML document with transaction parameters and status, or response to capture output from the server.

Edges:

The following table describes the edges that are returned by the Avro RPC HTTP Emit service.

Edge
Description
NNN
Where NNN is a 3-digit integer, HTTP status codes 100-599.
success
The Avro RPC request was successful.
fail_connect
An iFL expression could not be evaluated.
fail_info
HTTP status codes 100-199.
fail_redirection
HTTP status codes 300-399.
fail_client
HTTP status codes 400-499.
fail_server
HTTP status codes 500-599.
fail_operation
The operation could not be completed successfully.
fail_parse
An iFL expression could not be evaluated.
fail_request_too_large
The request is larger than the configured Maximum Request Size.
fail_response_too_large
The response is larger than the configured Maximum Response Size.

Edge	Description
NNN	Where NNN is a 3-digit integer, HTTP status codes 100-599.
success	The Avro RPC request was successful.
fail_connect	An iFL expression could not be evaluated.
fail_info	HTTP status codes 100-199.
fail_redirection	HTTP status codes 300-399.
fail_client	HTTP status codes 400-499.
fail_server	HTTP status codes 500-599.
fail_operation	The operation could not be completed successfully.
fail_parse	An iFL expression could not be evaluated.
fail_request_too_large	The request is larger than the configured Maximum Request Size.
fail_response_too_large	The response is larger than the configured Maximum Response Size.

The service often returns more than one edge with the more specific edge first. For example, if the request is unauthorized, the returned edges will be 401,fail_client. This can be handled by wiring the edge 401, or fail_client, or OnFailure.

Example:

The following syntax shows a sample Avro protocol taken from the Avro test suite.

{"namespace": "org.apache.avro.test",
 "protocol": "Mail",
 "types": [
     {"name": "Message", "type": "record",
      "fields": [
          {"name": "to",   "type": "string"},
          {"name": "from", "type": "string"},
          {"name": "body", "type": "string"}
      ]
     }
 ],
 "messages": {
     "send": {
         "request": [{"name": "message", "type": "Message"}],
         "response": "string"
     },
     "fireandforget": {
         "request": [{"name": "message", "type": "Message"}],
         "response": "null",
         "one-way": true
     }
 }
}

A sample request to call the send message can be similar to the following syntax (indented for display purposes only):

<av:item xmlns:av=\"http://iwaysoftware.com/avro\">
  <message>
    <to>wife</to>
    <from>husband</from>
    <body>I love you!</body>
  </message>
</av:item>

Since the send message has a single request argument called message, the anonymous record contains a single field called message.

A sample response for the send message may appear similar to the following syntax:

<av:item xmlns:av=\"http://iwaysoftware.com/avro\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">Sent message to [wife] from [husband] with body [I love you!]</av:item>

A sample response for the fireandforget message would be identical to the request for the send message because they have the same request arguments. The selected message is configured by the Message Name parameter of the service, not within the input document.

The response for the fireandforget message will be the nil item since the message is one-way.

<av:item xmlns:av="http://iwaysoftware.com/avro" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/>

iWay Software