Using spark with hive at Windows

• Download the 64-bit winutils.exe (106KB)
  • Direct download link https://github.com/steveloughran/winutils/raw/master/hadoop-2.6.0/bin/winutils.exe
  • NOTE: there is a different winutils.exe file for the 32-bit Windows and it will not work on the 64-bit OS
• Copy the downloaded file winutils.exe into a folder like C:\hadoop\bin (or C:\spark\hadoop\bin)
• Set the environment variable HADOOP_HOME to point to the above directory but without \bin. For example:
  • if you copied the winutils.exe to C:\hadoop\bin, set HADOOP_HOME=C:\hadoop
  • if you copied the winutils.exe to C:\spark\hadoop\bin, set HADOOP_HOME=C:\spark\hadoop
• Double-check that the environment variable HADOOP_HOME is set properly by opening the Command Prompt and running echo %HADOOP_HOME%
• Change the write permission of '\tmp\hive'
  • %HADOOP_HOME%\bin\winutils.exe chmod 777 \tmp\hive
  • NOTE: use 'cd' command to change the driver where you run the spark app
• For spark 2, if you see an error like "Relative path in absolute URI: file:E:/eclipse-neon/workspace/Test/spark-warehouse", then set the config like below
  
  • SparkSession.buider().config("spark.sql.warehouse.dir", "E:/eclipse-neon/workspace/Test/spark-warehouse")
• Reference
  • https://blogs.msdn.microsoft.com/arsen/2016/02/09/resolving-spark-1-6-0-java-lang-nullpointerexception-not-found-value-sqlcontext-error-when-running-spark-shell-on-windows-10-64-bit/