Mungeol Heo

Monday, November 30, 2015

HDP - Hadoop HA

namenode HA
1. Check to make sure you have at least three hosts in your cluster and are running at least three ZooKeeper servers
2. ambari web -> services -> HDFS -> service actions -> enable namenode HA -> nameservice ID -> select hosts -> review
3. create checkpoint
  1. login to the namenode host
  2. sudo su hdfs -l -c 'hdfs dfsadmin -safemode enter'
  3. sudo su hdfs -l -c 'hdfs dfsadmin -saveNamespace'
4. configure components
5. initialize journalnodes
  1. login to the namenode host
  2. sudo su hdfs -l -c 'hdfs namenode -initializeSharedEdits'
6. start components
7. initialize metadata
  1. login to the namenode host
  2. sudo su hdfs -l -c 'hdfs zkfc -formatZK'
  3. login to the additional namenode host
  4. sudo su hdfs -l -c 'hdfs namenode -bootstrapStandby'
8. finalize HA setup
resourcemanager HA
1. Check to make sure you have at least three hosts in your cluster and are running at least three ZooKeeper servers
2. ambari web -> services -> YARN -> service actions -> enable resouremanger HA -> get started -> select hosts -> review -> configure components

HDP 2.3 installation

software requirements
1. yum, rpm, scp, curl, unzip, tar, wget, OpenSSL v1.01, build 16 or later), python v2.6
2. option
  1. chmod 755 /usr/bin/yum /bin/rpm /usr/bin/scp /usr/bin/curl /usr/bin/unzip /bin/tar /usr/bin/wget /usr/bin/ssh /usr/bin/python
check java version
1. Oracle JDK 1.8 64-bit (minimum JDK 1.8_40) (default)
2. Oracle JDK 1.7 64-bit (minimum JDK 1.7_67)
3. OpenJDK 8 64-bit
4. OpenJDK 7 64-bit
ulimit -Sn / ulimit -Hn
1. 65535
configure FQDN at /etc/hosts, /etc/sysconfig/networks and using hostname command
set up password-less SSH
1. ssh-keygen
2. Copy the SSH Public Key (id_rsa.pub) to the root account on your target hosts
  1. option
    1. vim /etc/ssh/sshd_config
    2. PermitRootLogin yes
    3. service sshd restart
    4. vim /etc/ssh/sshd_config
3. all hosts
  1. cat id_rsa.pub >> ~/.ssh/authorized_keys
4. option
  1. chmod 700 ~/.ssh
  2. chmod 600 ~/.ssh/authorized_keys
NTP yum -y install sudo (option)
1. yum -y install ntp
2. chkconfig ntpd on
3. service ntpd start
SELinux
1. vim /etc/selinux/config
2. if it is abled
  1. set SELINUX=disabled
  2. command: setenforce 0
3. if PackageKit is installed
  1. vim /etc/yum/pluginconf.d/refresh-packagekit.conf
  2. enabled=0
umask
1. umask
  1. 0022
2. if it is not 0022
  1. vim /etc/profile
  2. umask 022
THP
1. check
  1. cat /sys/kernel/mm/redhat_transparent_hugepage/defrag
  2. cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
2. command and /etc/rc.local
  1. echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
  2. echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
permission (option)
1. ambari
  1. /bin/hostname
  2. /usr/bin/sudo
2. namenode
  1. /usr/bin/which
  2. /bin/ps
  3. /bin/df
3. ambari metrics
  1. /usr/bin/gcc
  2. /usr/bin/ld
  3. /etc/centos-release
4. yarn, MR2
  1. /tmp (check)
    1. /etc/fstab (default)
    2. /proc/mounts (current)
    3. mount -o remount,exec /tmp
  2. /usr/bin/curl
5. knox
  1. /bin/netstat
6. all
  1. chmod 755 /bin/hostname /usr/bin/sudo /usr/bin/which /bin/ps /bin/df /usr/bin/gcc /usr/bin/ld /etc/centos-release /usr/bin/curl /bin/netstat
Install ambari 2.1.0 using non-default databases (HDP)

http://mungeol-heo.blogspot.kr/2015/11/install-mariadb-httpmungeol-heo.html

http://<your.ambari.server>:8080 -> admin / admin -> launch install wiizard -> cluster name -> HDP 2.3 -> install options -> confirm hosts -> choose servicesl -> assgin masters -> assign slaves and clients -> customize services -> review -> install, start and test -> summary

Install ambari 2.1.0 using non-default databases (HDP)

install mariaDB

http://mungeol-heo.blogspot.kr/2015/07/mariadb.html

mysql-connector-java
1. yum -y install mysql-connector-java
  1. use latest version if "option sql_select_limit=default" error occurs
    1. cd /usr/share
    2. mkdir java
    3. cd java
    4. wget http://cdn.mysql.com/Downloads/Connector-J/mysql-connector-java-5.1.36.zip
    5. unzip mysql-connector-java-5.1.36.zip
    6. cp mysql-connector-java-5.1.36/mysql-connector-java-5.1.36-bin.jar .
    7. ln -s mysql-connector-java-5.1.36-bin.jar mysql-connector-java.jar
2. ls /usr/share/java/mysql-connector-java.jar
3. chmod 644 /usr/share/java/mysql-connector-java.jar (option)
wget -nv http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.1.0/ambari.repo -O /etc/yum.repos.d/ambari.repo
yum -y install ambari-server
service postgresql stop
RDB setting
1. mysql -u root -p
2. CREATE USER 'ambari'@'%' IDENTIFIED BY 'bigdata';
3. GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
4. CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'bigdata';
5. GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost';
6. CREATE USER 'ambari'@'bigdata-dev01.co.kr' IDENTIFIED BY 'bigdata';
7. GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'bigdata-dev01.co.kr';
8. FLUSH PRIVILEGES;
9. mysql -u ambari -p
10. CREATE DATABASE ambari;
11. USE ambari;
12. SOURCE /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
ambari-server setup
1. Select y at Enter advanced database configuration
2. enter 3 (mysql)
3. select default values for all left
ambari-server start

HDP 2.2 installation

yum -y install java-1.7.0-openjdk (option)
ulimit -Sn / ulimit -Hnconfigure FQDN at /etc/hosts, /etc/sysconfig/networks and using hostname command
1. 65535
ssh-keygen (passwordless connection)
NTP yum -y install sudo (option)
1. yum -y install ntp
2. chkconfig ntpd on
3. service ntpd start
THP
1. check
  1. cat /sys/kernel/mm/redhat_transparent_hugepage/defrag
  2. cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
2. command and /etc/rc.local
  1. echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
  2. echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
permission (option)
1. ambari
  1. /bin/hostname
  2. /usr/bin/sudo
2. namenode
  1. /usr/bin/which
  2. /bin/ps
  3. /bin/df
3. ambari metrics
  1. /usr/bin/gcc
  2. /usr/bin/ld
  3. /etc/centos-release
4. yarn, MR2
  1. /tmp (check)
    1. /etc/fstab (default)
    2. /proc/mounts (current)
    3. mount -o remount,exec /tmp
  2. /usr/bin/curl
5. knox
  1. /bin/netstat
Install ambari 2.0.1 (HDP)

http://mungeol-heo.blogspot.kr/2015/11/install-ambari-201-hdp.html

install and run services from ambari web

Install ambari 2.0.1 (HDP)

wget -nv http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.0.1/ambari.repo -O /etc/yum.repos.d/ambari.repo
yum -y install ambari-server
ambari-server setup
ambari-server start

Friday, November 27, 2015

Machine learning tools

confusion matrix
1. https://en.wikipedia.org/wiki/Confusion_matrix
2. python function
  1. http://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html
  2. http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html
cross-validation
standard error of the mean
1. https://en.wikipedia.org/wiki/Standard_error#Standard_error_of_the_mean
2. python function
  1. http://docs.scipy.org/doc/scipy-0.16.0/reference/generated/scipy.stats.sem.html
one hot encoding
1. https://en.wikipedia.org/wiki/One-hot
2. python function
  1. http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html

How to choose a statistical test

C -> Q

ANOVA
1. https://en.wikipedia.org/wiki/Analysis_of_variance
2. python function
  1. http://statsmodels.sourceforge.net/devel/example_formulas.html#ols-regression-using-formulas
Tukey's honestly significant difference test
1. https://en.wikipedia.org/wiki/Tukey%27s_range_test
2. python function
  1. http://statsmodels.sourceforge.net/devel/generated/statsmodels.sandbox.stats.multicomp.MultiComparison.html