Friday, December 30, 2016

Hadoop HA (HDP)

  • namenode HA
    • enable
      • prerequisite (HDP 2.4)
        • login to the hive metastore node
        • hive --config /etc/hive/conf.server --service metatool -listFSRoot
        • remember the hive metastore FS root
          • hdfs://name-node-host-name/apps/hive/warehouse
      • Check to make sure you have at least three hosts in your cluster and are running at least three ZooKeeper servers
      • Check to make sure that the HDFS and ZooKeeper services are not in Maintenance Mode
      • ambari web -> services -> HDFS -> service actions -> enable namenode HA -> nameservice ID -> select hosts -> review
      • create checkpoint
        • login to the namenode host
        • sudo su hdfs -l -c 'hdfs dfsadmin -safemode enter'
        • sudo su hdfs -l -c 'hdfs dfsadmin -saveNamespace'
      • configure components
      • initialize journalnodes
        • login to the namenode host
        • sudo su hdfs -l -c 'hdfs namenode -initializeSharedEdits'
      • start components
      • initialize metadata
        • login to the namenode host
        • sudo su hdfs -l -c 'hdfs zkfc -formatZK'
        • login to the additional namenode host
        • sudo su hdfs -l -c 'hdfs namenode -bootstrapStandby'
      • finalize HA setup
      • change the Hive Metastore FS root  (HDP 2.4)
        • login to the hive metastore node
        • hive --config /etc/hive/conf.server --service metatool -updateLocation <new-location> <old-location>
        • hive --config /etc/hive/conf.server --service metatool -updateLocation hdfs://cluster-name/apps/hive/warehouse hdfs://name-node-host-name/apps/hive/warehouse
    • disable
      • On the Active NameNode host, execute the following commands to save the namespace. You must be the HDFS service user to do this
        • sudo su -l <HDFS_USER> -c 'hdfs dfsadmin -safemode enter' sudo su -l <HDFS_USER> -c 'hdfs dfsadmin -saveNamespace'
      • Browse to Ambari Web > Services, then choose Stop All in the Services navigation panel. You must wait until all the services are completely stopped
      • Log into the Ambari server host and set the following environment variables to prepare for the rollback procedure
        • export NAMESERVICE_ID=hadoopdev
        • export AMBARI_USER=admin
        • export AMBARI_PW=admin
        • export AMBARI_PORT=8080
        • export AMBARI_PROTO=http
        • export CLUSTER_NAME=dev
        • export NAMENODE_HOSTNAME=lqahadoopname01.net
        • export ADDITIONAL_NAMENODE_HOSTNAME=lqahadoopname02.net
        • export SECONDARY_NAMENODE_HOSTNAME=lqahadoopname02.net
        • export JOURNALNODE1_HOSTNAME=lqahadoopdata01.net
        • export JOURNALNODE2_HOSTNAME=lqahadoopdata02.net
        • export JOURNALNODE3_HOSTNAME=lqahadoopdata03.net
        • curl -u $AMBARI_USER:$AMBARI_PW -H "X-Requested-By: ambari" -i $AMBARI_PROTO://localhost:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/host_components?HostRoles/component_name=ZKFC
        • curl -u $AMBARI_USER:$AMBARI_PW -H "X-Requested-By: ambari" -i -X DELETE $AMBARI_PROTO://localhost:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/hosts/$NAMENODE_HOSTNAME/host_components/ZKFC 
        • curl -u $AMBARI_USER:$AMBARI_PW -H "X-Requested-By: ambari" -i -X DELETE $AMBARI_PROTO://localhost:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/hosts/$ADDITIONAL_NAMENODE_HOSTNAME/host_components/ZKFC
        • curl -u $AMBARI_USER:$AMBARI_PW -H "X-Requested-By: ambari" -i $AMBARI_PROTO://localhost:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/host_components?HostRoles/component_name=ZKFC
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $ABARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.nameservices
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.client.failover.proxy.provider.$NAMESERVICE_ID
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.ha.namenodes.$NAMESERVICE_ID
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.ha.fencing.methods
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.ha.automatic-failover.enabled
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.namenode.http-address.$NAMESERVICE_ID.nn1
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.namenode.http-address.$NAMESERVICE_ID.nn2
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.namenode.https-address.$NAMESERVICE_ID.nn1
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.namenode.https-address.$NAMESERVICE_ID.nn2
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.namenode.rpc-address.$NAMESERVICE_ID.nn1
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.namenode.rpc-address.$NAMESERVICE_ID.nn2
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.namenode.shared.edits.dir
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.journalnode.edits.dir
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.journalnode.http-address
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.journalnode.kerberos.internal.spnego.principal
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.journalnode.kerberos.principal
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME hdfs-site dfs.journalnode.keytab.file
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT delete localhost $CLUSTER_NAME core-site ha.zookeeper.quorum
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT set localhost $CLUSTER_NAME core-site fs.defaultFS hdfs://$NAMENODE_HOSTNAME
        • /var/lib/ambari-server/resources/scripts/configs.sh -u $AMBARI_USER -p $AMBARI_PW -port $AMBARI_PORT get localhost $CLUSTER_NAME core-site
        • curl -u $AMBARI_USER:$AMBARI_PW -H "X-Requested-By: ambari" -i -X GET $AMBARI_PROTO://localhost:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/host_components?HostRoles/component_name=SECONDARY_NAMENODE
        • curl -u $AMBARI_USER:$AMBARI_PW -H "X-Requested-By: ambari" -i -X POST -d '{"host_components" : [{"HostRoles":{"component_name":"SECONDARY_NAMENODE"}}]}' $AMBARI_PROTO://localhost:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/hosts?Hosts/host_name=$SECONDARY_NAMENODE_HOSTNAME
        • curl -u $AMBARI_USER:$AMBARI_PW -H "X-Requested-By: ambari" -i -X '{"RequestInfo":{"context":"Enable Secondary NameNode"},"Body":{"HostRoles":{"state":"INSTALLED"}}}'$AMBARI_PROTO://localhost:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/hosts/$SECONDARY_NAMENODE_HOSTNAME/host_components/SECONDARY_NAMENODE
        • curl -u $AMBARI_USER:$AMBARI_PW -H "X-Requested-By: ambari" -i -X GET $AMBARI_PROTO://localhost:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/host_components?HostRoles/component_name=JOURNALNODE
        • curl -u $AMBARI_USER:$AMBARI_PW -H "X-Requested-By: ambari" -i -X DELETE $AMBARI_PROTO://localhost:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/hosts/$JOURNALNODE1_HOSTNAME/host_components/JOURNALNODE
        • curl -u $AMBARI_USER:$AMBARI_PW -H "X-Requested-By: ambari" -i -X DELETE $AMBARI_PROTO://localhost:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/hosts/$JOURNALNODE2_HOSTNAME/host_components/JOURNALNODE
        • curl -u $AMBARI_USER:$AMBARI_PW -H "X-Requested-By: ambari" -i -X DELETE $AMBARI_PROTO://localhost:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/hosts/$JOURNALNODE3_HOSTNAME/host_components/JOURNALNODE
      • Go to Ambari Web UI > Services, then select HDFS
        • Check the Summary panel and make sure that the first three lines look like this:
          • NameNode
          • SNameNode
          • DataNodes
          • You should not see any line for JournalNodes
  • resourcemanager HA
    • enable HA
      • Check to make sure you have at least three hosts in your cluster and are running at least three ZooKeeper servers
      • ambari web -> services -> YARN -> service actions -> enable resouremanger HA -> get started -> select hosts -> review -> configure components
    • disable HA
      • In Ambari Web, stop YARN and ZooKeeper services
      • On the Ambari Server host, use the Ambari API to retrieve the YARN configurations into a JSON file
        • /var/lib/ambari-server/resources/scripts/configs.sh get ambari.server cluster.name yarn-site yarn-site.json
      • Modify the following priorities in the yarn-site.json file

        Property
        Value
        yarn.resourcemanager.ha.enabled
        Change the value to false.
        yarn.resourcemanager.ha.rm-ids
        Delete this property.
        yarn.resourcemanager.hostname.rm1
        Delete this property.
        yarn.resourcemanager.hostname.rm2
        Delete this property.
        yarn.resourcemanager.webapp.address.rm1
        Delete this property.
        yarn.resourcemanager.webapp.address.rm2
        Delete this property.
        yarn.resourcemanager.webapp.https.address.rm1
        Delete this property.
        yarn.resourcemanager.webapp.https.address.rm2
        Delete this property.
        yarn.resourcemanager.cluster-id
        Delete this property.
        yarn.resourcemanager.ha.automatic-failover.zk-base-path
        Delete this property.
      • Verify the following properties in the yarn-site.json file are set to the ResourceManager hostname you will be keeping:

        Property
        Value
        yarn.resourcemanager.hostname
        <ResourceManager hostname>
        yarn.resourcemanager.admin.address
        <ResourceManager hostname>
        yarn.resourcemanager.webapp.address
        <ResourceManager hostname>
        yarn.resourcemanager.resource-tracker.address
        <ResourceManager hostname>
        yarn.resourcemanager.scheduler.address
        <ResourceManager hostname>
        yarn.resourcemanager.webapp.https.address
        <ResourceManager hostname>
        yarn.timeline-service.webapp.address
        <ResourceManager hostname>
        yarn.timeline-service.webapp.https.address
        <ResourceManager hostname>
        yarn.timeline-service.address
        <ResourceManager hostname>
        yarn.log.server.url
        <ResourceManager hostname>
      • Search the yarn-site.json file and remove any references to the ResourceManager hostname that you will be removing
      • Search the yarn-site.json file and remove any properties that might still be set for ResourceManager IDs. For example, rm1 and rm2
      • Save the yarn-site.json file and set that configuration against the Ambari Server
        • /var/lib/ambari-server/resources/scripts/configs.sh set ambari.server cluster.name yarn-site yarn-site.json
      • Using the Ambari API, delete the ResourceManager host component for the host that you are deleting
      • In Ambari Web, start the ZooKeeper service
      • On a host that has the ZooKeeper client installed, use the ZooKeeper client to change znode permissions
        • /usr/hdp/current/zookeeper-client/bin/zkCli.sh
        • setAcl /rmstore/ZKRMStateRoot world:anyone:rwcda
      • In Ambari Web, restart ZooKeeper service and start YARN service
Posted by at

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.

Subscribe to: Post Comments (Atom)