Gangmax Blog

Install Hadoop on Mac

This post records how to install Apache Hadoop(single node) on Mac. This is useful when you want to run some simplge tasks on your local environment. From here. In my case the Hadoop version is “3.3.0” rather than “3.2.1” used in this article, and my JDK version is “1.8.0_201”.

  1. Install Hadoop with HomeBrew
1
brew install hadoop
  1. Configure

2.1 Open the “/usr/local/Cellar/hadoop/3.3.0/libexec/etc/hadoop/hadoop-env.sh” file and do the following modifications:

1
2
3
4
# 1. Add the following content:
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_201.jdk/Contents/Home
# 2. Update "export HADOOP_OPTS = "-Djava.net.preferIPv4Stack=true"" to:
export HADOOP_OPTS = "-Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc="

2.2 “core-site.xml”

1
2
3
4
5
6
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

2.3 “hdfs-site.xml”

1
2
3
4
5
6
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

2.4 “mapred-site.xml”

1
2
3
4
5
6
7
8
9
10
<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
  </property>
</configuration>

2.5 “yarn-site.xml”

1
2
3
4
5
6
7
8
9
10
<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
  </property>
</configuration>
  1. SSH login

From here.

Go to “go to “System Preference > Sharing”, enable “Remote Login” and “Allow access for: All Users”. In my case I only enable the “administror” and it works.

  1. Format NameNode
1
2
cd /usr/local/cellar/hadoop/3.3.0/libexec/bin
hdfs namenode -format
  1. Run Hadoop
1
2
3
4
5
6
7
8
9
10
cd /usr/local/cellar/hadoop/3.3.0/libexec/sbin
./start-all.sh
jps
# The output of jps looks like:
18646 NodeManager
19014 Jps
17847 NameNode
18039 DataNode
18457 ResourceManager
18267 SecondaryNameNode
  1. Check the status of Hadoop

Open the URL “http://localhost:9870“ in Browser.

  1. Close Hadoop
1
2
cd /usr/local/cellar/hadoop/3.3.0/libexec/sbin
./stop-all.sh

Comments