Install ambari 2.1.0 using non-default databases (HDP)

1. install mariaDB
1. http://mungeol-heo.blogspot.kr/2015/07/mariadb.html
2. mysql-connector-java
   
   1. yum -y install mysql-connector-java
      1. use latest version if "option sql_select_limit=default" error occurs
         1. cd /usr/share
         2. mkdir java
         3. cd java
         4. wget http://cdn.mysql.com/Downloads/Connector-J/mysql-connector-java-5.1.36.zip
         5. unzip mysql-connector-java-5.1.36.zip
         6. cp mysql-connector-java-5.1.36/mysql-connector-java-5.1.36-bin.jar .
         7. ln -s mysql-connector-java-5.1.36-bin.jar mysql-connector-java.jar
   2. ls /usr/share/java/mysql-connector-java.jar
   3. chmod 644 /usr/share/java/mysql-connector-java.jar (option)
3. wget -nv http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.1.0/ambari.repo -O /etc/yum.repos.d/ambari.repo
4. yum -y install ambari-server
   
5. service postgresql stop
6. RDB setting
   1. mysql -u root -p
   2. CREATE USER 'ambari'@'%' IDENTIFIED BY 'bigdata';
   3. GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
   4. CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'bigdata';
      
   5. GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost';
      
   6. CREATE USER 'ambari'@'bigdata-dev01.co.kr' IDENTIFIED BY 'bigdata';
      
   7. GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'bigdata-dev01.co.kr';
      
   8. FLUSH PRIVILEGES;
   9. mysql -u ambari -p
   10. CREATE DATABASE ambari;
   11. USE ambari;
   12. SOURCE /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
       
7. ambari-server setup
   1. Select y at Enter advanced database configuration
   2. enter 3 (mysql)
   3. select default values for all left
8. ambari-server start

HDP 2.2 installation

1. yum -y install java-1.7.0-openjdk (option)
2. ulimit -Sn / ulimit -Hnconfigure FQDN at /etc/hosts, /etc/sysconfig/networks and using hostname command
   1. 65535
3. ssh-keygen (passwordless connection)
4. NTP yum -y install sudo (option)
   1. yum -y install ntp
   2. chkconfig ntpd on
   3. service ntpd start
5. THP
   1. check
      1. cat /sys/kernel/mm/redhat_transparent_hugepage/defrag
      2. cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
   2. command and /etc/rc.local
      1. echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
      2. echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
6. permission (option)
   1. ambari
      1. /bin/hostname
      2. /usr/bin/sudo
   2. namenode
      1. /usr/bin/which
      2. /bin/ps
      3. /bin/df
   3. ambari metrics
      1. /usr/bin/gcc
      2. /usr/bin/ld
      3. /etc/centos-release
   4. yarn, MR2
      1. /tmp (check)
         1. /etc/fstab (default)
         2. /proc/mounts (current)
         3. mount -o remount,exec /tmp
      2. /usr/bin/curl
   5. knox
      1. /bin/netstat
7. Install ambari 2.0.1 (HDP)
1. http://mungeol-heo.blogspot.kr/2015/11/install-ambari-201-hdp.html
8. install and run services from ambari web

Install ambari 2.0.1 (HDP)

1. wget -nv http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.0.1/ambari.repo -O /etc/yum.repos.d/ambari.repo
2. yum -y install ambari-server
3. ambari-server setup
4. ambari-server start

Machine learning tools

1. confusion matrix
   1. https://en.wikipedia.org/wiki/Confusion_matrix
   2. python function
      
      1. http://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html
      2. http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html
2. cross-validation
   1. https://en.wikipedia.org/wiki/Cross-validation_(statistics)
   2. python function
      1. http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.cross_val_score.html
   3. k-fold
      1. https://en.wikipedia.org/wiki/Cross-validation_(statistics)#k-fold_cross-validation
      2. python function
         1. http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.KFold.html
   4. leave-one-out
      
      1. https://en.wikipedia.org/wiki/Cross-validation_(statistics)#Leave-one-out_cross-validation
      2. python function
         1. http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.LeaveOneOut.html
3. standard error of the mean
   1. https://en.wikipedia.org/wiki/Standard_error#Standard_error_of_the_mean
   2. python function
      1. http://docs.scipy.org/doc/scipy-0.16.0/reference/generated/scipy.stats.sem.html
4. one hot encoding
   1. https://en.wikipedia.org/wiki/One-hot
   2. python function
      1. http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html

How to choose a statistical test

C -> Q

1. ANOVA
   1. https://en.wikipedia.org/wiki/Analysis_of_variance
   2. python function
      1. http://statsmodels.sourceforge.net/devel/example_formulas.html#ols-regression-using-formulas
2. Tukey's honestly significant difference test
   1. https://en.wikipedia.org/wiki/Tukey%27s_range_test
   2. python function
      1. http://statsmodels.sourceforge.net/devel/generated/statsmodels.sandbox.stats.multicomp.MultiComparison.html

Graphing decisions flowchart

R

1. 2015.09.03
   1. read file from remote server (windows)
      1. install openssh
         1. download the latest version from http://www.mls-software.com/opensshd.html
         2. the blank should not be included in the path where you install it
            1. e.g. D:\OpenSSH
      2. create keys
         1. run 'cmd'
         2. d:
         3. cd OpenSSH
         4. cd bin
         5. ssh-keygen -t rsa
            1. it will tell you where the keys are created
            2. e.g. your identification has been saved in /home/admin/.ssh/id_rsa
            3. the real path is something like, C:\Users\admin\.ssh
         6. scp /home/admin/.ssh/id_rsa.pub <user name>@<host name>:
         7. ssh <user name>@<host name>
         8. if there is no .ssh directory
            1. mkdir .ssh
            2. chmod 700 .ssh
         9. cat id_rsa.pub >> .ssh/authorized_keys
         10. chmod 644 .ssh/authorized_keys
         11. exit
         12. ssh <user name>@<host name>
             1. it should not ask you to enter the password this time
         13. warning: unprotected private key file! / permissions 0xxx for 'path/to/id_rsa' are too open (OpenSSH)
             1. chmod 600 path/to/id_rsa
             2. try again
      3. run R/Rstudio
         1. install.packages("RCurl", dependencies = TRUE)
         2. require(RCurl)
         3. x = scp("host name", "/path/to/the/file", key=c("C:/Users/admin/.ssh/id_rsa.pub", "C:/Users/admin/.ssh/id_rsa"), user="user name", binary=F)
         4. or
         5. d <- read.table(pipe( 'ssh -l <user name> <host name> "cat /path/to/the/file"' ))