Apache Hadoop 2.0.5 alpha - start-dfs.sh (Incompatible clusterIDs)

Note: solution for 'java.io.IOException: Incompatible clusterIDs'

- ssh hostnameN
- vi $HADOOP_HOME/etc/hadoop/core-site.xml
- check the value of 'hadoop.tmp.dir'
- example

<property>
<name>hadoop.tmp.dir</name>
<value>/home/username/hadoop/tmp</value>
</property>

- rm /home/username/hadoop/tmp/dfs/data/current/VERSION (datanode)

- $HADOOP_HOME/sbin/start-dfs.sh

Apache Tajo CDH - Cluster Setup (Configuration)

1. version

- CDH: CDH-4.4.0-1.cdh4.4.0.p0.39
- Hadoop: Hadoop 2.0.0-cdh4.4.0
- Tajo: https://github.com/gruter/tajo-cdh (2014.01.10)

&

- tajo-0.2.0-incubating

2. Configuration

- cd $TAJO_HOME/conf

- cp tajo-site.xml.template tajo-site.xml
- vi tajo-site.xml
- add

<property>
<name>tajo.rootdir</name>
<value>hdfs://hostname:port/tajo</value>
</property>

<property>
<name>tajo.master.umbilical-rpc.address</name>
<value>hostname:26001</value>
</property>

<property>
<name>tajo.catalog.client-rpc.address</name>
<value>hostname:26005</value>
</property>

<property>
<name>tajo.cluster.distributed</name>
<value>true</value>
</property>

- (there was 'tajo.cluster.distributed' property before, but not for current documentation)
- $HADOOP_HOME/bin/hadoop fs -mkdir /tajo
- $HADOOP_HOME/bin/hadoop fs -chmod g+w /tajo
- vi workers
- add worker information
- example

worker1
worker2
worker3

3. Reference

- http://tajo.incubator.apache.org/configuration.html

<property>
<name>org.apache.tajo.rootdir</name>
<value>hdfs://hostname:port/tajo</value>
</property>

<property>
<name>org.apache.tajo.cluster.distributed</name>
<value>true</value>
</property>

- http://tajo.incubator.apache.org/tajo-0.2.0-doc.html

<property>
<name>tajo.rootdir</name>
<value>hdfs://hostname:port/tajo</value>
</property>

<property>
<name>tajo.master.umbilical-rpc.address</name>
<value>hostname:26001</value>
</property>

<property>
<name>tajo.catalog.client-rpc.address</name>
<value>hostname:26005</value>
</property>

Apache Tajo CDH - start-tajo.sh (org/apache/hadoop/conf/Configuration)

Note: solution for 'java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration'

1. version

- CDH: CDH-4.4.0-1.cdh4.4.0.p0.39
- Hadoop: Hadoop 2.0.0-cdh4.4.0
- Tajo: https://github.com/gruter/tajo-cdh (2014.01.10)

2. solution

- cd /opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop
- sudo mkdir -p share/hadoop
- cd share/hadoop
- ln -s /opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop/client common

Apache Tajo CDH - start-tajo.sh (org/apache/hadoop/yarn/service/CompositeService)

Note: solution for 'java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/service/CompositeService'

1. version

- CDH: CDH-4.4.0-1.cdh4.4.0.p0.39
- Hadoop: Hadoop 2.0.0-cdh4.4.0
- Tajo: https://github.com/gruter/tajo-cdh (2014.01.10)

2. solution

- cd /opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop
- sudo mkdir -p share/hadoop
- cd share/hadoop
- ln -s /opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hadoop-yarn yarn

Apache Tajo 0.2.0 incubating - tsql (Connection refused: localhost/127.0.0.1:26002)

Note: soultion for 'Connection refused: localhost/127.0.0.1:26002'

- ssh 127.0.0.1 (password less to 127.0.0.1)

Apache Hadoop 2.0.5 alpha - yarn-site.xml (Invalid shuffle port number -1 returned)

Note: solution for 'java.lang.IllegalStateException: Invalid shuffle port number -1 returned'

- vi $HADOOP_HOME/etc/hadoop/yarn-site.xml
- set the value of 'yarn.nodemanager.aux-services' as 'mapreduce.shuffle'
- not 'mapreduce_shuffle'

Ubuntu 12.04 - apt-get install (404 Not Found)

Note: solution for '404  Not Found'

- vi /etc/apt/sources.list
- replace

deb mirror://mirrors.ubuntu.com/mirrors.txt precise main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/mirrors.txt precise-updates main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/mirrors.txt precise-backports main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/mirrors.txt precise-security main restricted universe multiverse

- sudo apt-get update
- sudo apt-get install packageName

Protobuf 2.5.0 - Installation

Note: solution for 'Cannot run program "protoc"'

- cd
- wget http://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
- tar xzf protobuf-2.5.0.tar.gz
- cd protobuf-2.5.0
- ./configure
- make
- sudo make install
- sudo ldconfig

Apache Hadoop 2.2.0 - 64 bit Native library

Note: solution for 'unable to load native-hadoop library'

- cd
- wget http://apache.tt.co.kr/hadoop/common/stable/hadoop-2.2.0-src.tar.gz
- tar xvfz hadoop-2.2.0-src.tar.gz
- cd hadoop-2.2.0-src
- mvn package -Pdist,native -DskipTests -Dtar
- cd hadoop-dist/target/hadoop-2.2.0/lib/native/
- cp * $HADOOP_HOME/lib/native