I am a relatively new user to Hadoop (using version 2.4.1). I installed hadoop on my first node without a hitch, but I can't seem to get the Resource Manager to start on my second node.

I cleared up some "shared library" problems by adding this to yarn-env.sh and hadoop-env.sh:

export HADOOP_HOME="/usr/local/hadoop"

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
I also added this to hadoop-env.sh:

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native
based on the advice of this post at horton works http://hortonworks.com/community/forums/topic/hdfs-tmp-dir-issue/

That cleared up all of my error messages; when I run /sbin/start-yarn.sh I get this:

starting yarn daemons

starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-HdNode.out

localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-HdNode.out
The only problem is, JPS says that the Resource Manager isn't running.

What's going on here?

Check your version of JPS and make sure it's the same as the version of java that you are running. Sometimes you start out with an out of the box jdk install, upgrade, set alternatives for the java bin, but still have the original jps binary being referenced.

Run ps -ef |grep java and look for the resource manager threads. Maybe it's actually running. If it is, try update-alternatives --config jps to see what binary jps is pointing at and compare it with the java binary that you are using.

If the resource manager is not running, it's time to do some basic linux troubleshooting. Check log files and barring that check actual command output.

On the system I'm looking at now, the log files for resource manager are placed in the hadoop-install/logs directory in yarn-username-resourcemanager-hostname.log and yarn-user-resourcemanager-hostname.out. Your configuration may place them in /var/logs or what have you. Also, have a look at the syslog.

If the logs don't yield any good information, which can happen, my process is to generally try to figure out the command line from the startup script (usually by prefixing the command line with echo), and then trying to run the command directly to see the output as it comes out.


I have actually run into this problem before, but I can't remember the specific issue. I'm sure the same result can manifest itself from a variety of problems. Considering that you are as far as you are in the process of getting set up, I believe it's likely to be a minor configuration issue.

Are you using "poor" Apache Hadoop or some other distribution like HDP or CDH. I would heavily recommend using automated tools like these two instead of messing up with native settings. In addition to easy installation, they provide tools for monitoring and managing your cluster later

So I was never able to find the error by looking through my logs. I ended up reinstalling it with CDH5 (which was MUCH easier than installing "poor" Hadoop) Now everything runs fine!

I'm still having trouble getting things to save to the hdfs, but thats a question for another day...