Re: Hadoop Adaptor Usage

Hello,

I should have had you look at a different log.  In ESP there are three logs:

1. stdstreams.log - This log contains licensing information/errors.
2. esp_server.log - This log contains information/errors about the project and certain adapters
3. frameworkadapter.log - This log contains information/errors about certain adapters (including the Hadoop adapter).

In my testing I could see some errors in the "frameworkadapter.log".  However, sometimes exceptions that are thrown to "stderr" only show up in stdstreams.log.

I've been doing some testing and I finally have it working.  First thing I should mention is it doesn't look like Hadoop itself accepts the user name/password in the URL:

% hadoop fs -put test.txt hdfs://hadoopUser:Password1@servername.acme.com:9000/user/hadoop/data/test.txt
put: Permission denied: user=nstack, access=WRITE, inode="/user/hadoop/data":hadoop:supergroup:drwxr-xr-x

But there is a not-very-well-known environment variable that you can set:

% setenv HADOOP_USER_NAME hadoopUser
% hadoop fs -put test.txt hdfs://servername.acme.com:9000/user/hadoop/data/test.txt

This is essentially equivalent to setting it in your Java program:
System.setProperty("HADOOP_USER_NAME", "hadoopUser");

You can edit $ESP_HOME/adapters/framework/bin/start.sh and set the Hadoop user name there:
HADOOP_USER_NAME=hadoopUser;export HADOOP_USER_NAME

For version 2.2.0, copy these files over to %ESP_HOME%\adapters\framework\libj:
hadoop-common-2.2.0.jar
hadoop-auth-2.2.0.jar
hadoop-hdfs-2.2.0.jar
guava-11.0.2.jar
protobuf-java-2.5.0.jar
commons-cli-1.2.jar (SP04 requires but SP08 already has it)

NOTE: There is a problem with ESP 5.1 SP04.  When running the Hadoop Output Adapter inside a project (managed mode), I could not stop it.  This was using the Hadoop 2.2.0 JAR files, I did not test Hadoop 1.2.1 because it can't communicate with Hadoop 2.2.0.

This was not a problem with the upcoming release of ESP 5.1 SP08.  Nor was it a problem with SP04 when running the adapter in unmanaged mode (started and stopped manually from outside the project).

So if you wish to use ESP 5.1 SP04, run the adapter in unmanaged mode:

1) Edit $ESP_HOME/adapters/framework/bin/start.sh and set the Hadoop user name there:
HADOOP_USER_NAME=hadoop;export HADOOP_USER_NAME

2) Make a copy of the file output (and/or input) adapter:
cp -Rf $ESP_HOME/adapters/framework/instances/file_csv_output /tmp
cd /tmp/file_csv_output

3) Edit the copy's adapter_config.xml file and change it so that it can run in unmanaged mode:
a) Uncomment these lines and change the "StreamName" element to the stream in the project that the adapter should subscribe to.  Leave "ProjectName" as is:
  <ProjectName>EspProject2</ProjectName>
  <StreamName>MyStream</StreamName>
b) Change the "Dir" and "File" elements belonging to "FileOutputTransporterParameters":
  <Dir>hdfs://servername.acme.com:9000/user/hadoop/data</Dir>
  <File>neal_test.csv</File>
c) Change the "Uri" element for the project "EspProject2" to point to your ESP project:
  <Uri>esp://esp_server_name.acme.com:51011/default/hadoop_test</Uri>
d) Change the "User" and "Password" so the adapter can connect to the ESP project:
  <User>espadm</User>
  <Password encrypted="false">Password1</Password>
e) Start the adapter:
  ./start_adapter.sh
d) Stop the adapter:
  ./stop_adapter.sh

Thanks,
  Neal