Help 4 Error

• test.csv: Error detected while parsing row starting at position: 556531513. Error: Bad character (ASCII 0) encountered (bigquery)
  • Find the character causing the problem.
    • less +556531513P test.csv
  • There will be a character like Ctrl-@ which is ^@.
  • Avoid or remove it before producing the CSV file, or from the CSV file.
    • E.g. gsutil cp gs://bucket/test.csv - | tr -d '\000' | gsutil cp - gs://bucket/test2.csv
• error: object Y is not a member of package X (intellij)
  • If the error is because of a local jar file, one of the options to solve it is installing the local jar file to the local maven repository.
  • http://mungeol-heo.blogspot.kr/2017/08/install-local-jar-to-local-maven.html
• Unable to lock the administration directory (/var/lib/dpkg/), is another process using it? (linux)
  • ps -ef | grep apt
  • If there is apt-get or aptitude process, then kill it by using kill command
  • dpkg --configure -a
• Script does not give expected result while using "sudo su - $user /path/to/script" (linux)
  • use "sudo su $user -l -c 'sh /path/to/script'" instead.
• There are stopped jobs (linux)
  • fg
• Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes (maven)
  • Add the following configuration to the maven-shade-plugin.

    <configuration>     <filters>         <filter>             <artifact>*:*</artifact>             <excludes>                 <exclude>META-INF/*.SF</exclude>                 <exclude>META-INF/*.DSA</exclude>                 <exclude>META-INF/*.RSA</exclude>             </excludes>         </filter>     </filters>     <!-- Additional configuration. --> </configuration>

• Some dependencies are not included in the JAR (maven)
  • remove the following options from maven-shade-plugin
    • <minimizeJar>true</minimizeJar>
• Unexpected token 'non English character' (minidev)
  • If the value consists of number followed by non English character, then it will give unexpected token error.
    • E.g.
      • {creative_id:04 (23842564830590116),0아:1야}
      • Unexpected token 야 at position 39
  • Put the value in brackets
    • E.g. 
      • {creative_id:04 (23842564830590116),0아:(1야)}
      • {creative_id:04 (23842564830590116),0아:1(야)}
• Strange host name while using openstack (openstack)
  • E.g. VM shows the host name as "test.novalocal", even if the host name in /etc/hosts is "test.com" or "hostname test.com" command is used to specify the host name.
  • Check the dhcp_domain in nova.conf.
  • If it is "dhcp_domain=novalocal", then change it or report it to the related team, like infra.
• java.lang.NoClassDefFoundError: net/minidev/asm/F (minidev)
  • need "accessors-smart-1.2.jar"
• cannot resolve symbol toDS/toDF (intellij)
  • import spark.implicits._
• Using platform encoding (UTF-8 actually) to copy filtered resources (intellij)
  • Add the following content to the pom.xml file
  • <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties>
• refusing to merge unrelated histories (git)
  • Use '--allow-unrelated-histories'
  • E.g.
    • Go to your project folder
    • git pull origin branch_name --allow-unrelated-histories
• java.io.IOException: Could not locate executable null\bin\winutils.exe (windows)
  • Download the 64-bit winutils.exe (106KB)
    • Direct download link https://github.com/steveloughran/winutils/raw/master/hadoop-2.6.0/bin/winutils.exe
    • NOTE: there is a different winutils.exe file for the 32-bit Windows and it will not work on the 64-bit OS
  • Copy the downloaded file winutils.exe into a folder like C:\hadoop\bin (or C:\spark\hadoop\bin)
  • Set the environment variable HADOOP_HOME to point to the above directory but without \bin. For example:
    • if you copied the winutils.exe to C:\hadoop\bin, set HADOOP_HOME=C:\hadoop
    • if you copied the winutils.exe to C:\spark\hadoop\bin, set HADOOP_HOME=C:\spark\hadoop
  • Double-check that the environment variable HADOOP_HOME is set properly by opening the Command Prompt and running echo %HADOOP_HOME%
  • Restart the eclipse or intellij and try again
• The root scratch dir: /tmp/hive on HDFS should be writable (windows)
  
  • %HADOOP_HOME%\bin\winutils.exe chmod 777 \tmp\hive
  • NOTE: use 'cd' command to change the driver where you run the spark app
• Relative path in absolute URI: file:E:/eclipse-neon/workspace/Test/spark-warehouse (windows)
  • SparkSession.buider().config("spark.sql.warehouse.dir", "E:/eclipse-neon/workspace/Test/spark-warehouse")
• java.lang.NoSuchMethodError: scala.runtime.ObjectRef.create(Ljava/lang/Object;)Lscala/runtime/ObjectRef (scala)
  • Check the scala version
  • E.g., app and compiler
• You ($userName) are not allowed to access to (crontab) because of pam configuration. (crontab)
  • It is because the password of the zeppelin is expired.
  • Reset the password will solve the problem.
• "'ascii' codec can't encode characters in position XXX: ordinal not in range(128)" (python)
  • add "from desktop.lib.i18n import smart_str"
  • use smart_str instead of str
• no acceptable C compiler found in $PATH (general)
  • CentOS
    • yum groupinstall "Development tools"
  • Ubuntu
    • apt-get install build-essential
• gmp.h: No such file or directory (general)
  • apt-get install libgmp3-dev
• mysql_config not found (HUE)
  • apt-get install libmysqlclient-dev
• openssl/e_os2.h: No such file or directory (general)
  • apt-get install libssl-dev
• Unable to correct problems, you have held broken packages (apt-get)
  • aptitude install <packagename>
• [Errno 1] _ssl.c:504: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol (easy_install)
  • pip install pyopenssl
• [Errno 101] Network is unreachable (easy_install)
  • export http_proxy=http://220.95.218.21:3128
  • export https_proxy=http://220.95.218.21:3128
• [Errno 101] Network is unreachable (pip)
  • pip --upgrade pip
• ffi.h: No such file or directory (general)
  • centOS: yum -y install libffi-devel
  • ubuntu: apt-get install libffi-dev
• openssl/opensslv.h: No such file or directory (general)
  
  • yum -y install openssl-devel
• character set collation (SQL server)
  • recreate related database or change the collation 
    • USE master;
      GO
      ALTER DATABASE MyOptionsTest
      COLLATE French_CI_AS ;
      GO openssl
      
      --Verify the collation setting.
      SELECT name, collation_name
      FROM sys.databases
      WHERE name = N'MyOptionsTest';
      GO
  • or, use collate like below
    • SELECT PID COLLATE Korean_100_CI_AS_WS as pid ...
    • INNER JOIN #PAY_FROM_REGIST B
      ON A.Person_ID = B.Person_ID COLLATE Korean_Wansung_CI_AS
      AND A.Game_CD = B.Game_CD COLLATE Korean_Wansung_CI_AS
• Hostname FQDN repeats the suffix twice (general)
  • e.g.
    • hostname datanade08.net
    • hostname
    • • datanade08.net
    • hostname -f
    • • datanade08.net.net
  • check /etc/hostname and /etc/hosts files, and the spelling of the argument which is given to hostname command
    • e.g. the argument which is datanade08.net should be datanode08.net.
• Failed to fetch (apt-get)
  • cd /etc/apt
  • check sources.list and files inside sources.list.d
• date is not synchronized even if NTP is running (ntp)
  • vim /etc/ntp.conf
  • add 
    • server 222.122.134.177
    • server 222.122.169.177
  • service ntp restart
• X must be owned by uid 0 and have the setuid bit set (chmod)
  • chmod 4755 X
• couldn't connect: Connection refused (apt-key)
  • http://keyserver.ubuntu.com/
  • search '0x<your key>'
  • click url from result
  • copy key block to a file. e.g. key.txt
  • apt-key add key.txt
  • apt-get update
• dpkg: warning: files list file for package 'libdbd-mysql-perl' missing (apt-get)
  • for package in $(apt-get upgrade 2>&1 | grep "warning: files list file for package '" | grep -Po "[^'\n ]+'" | grep -Po "[^']+"); do apt-get install --reinstall "$package"; done
• Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', error(<error number>, '<error message>'))' (pip)
  • pip --proxy IP:port install <name>
• Proxy URLs must have explicit schemes (pip)
  • pip --proxy IP:port install <name>
• gcc: error trying to exec 'cc1plus': execvp: No such file or directory (general)
  
  • yum -y install gcc-c++
• fatal error: sasl/sasl.h: No such file or directory (general)
  
  • yum -y install openldap-devel
  • apt-get -y install libsasl2-dev python-dev libldap2-dev libssl-dev
• thrift.transport.TTransport.TTransportException: Could not start SASL: Error in sasl_client_start (-4) SASL(-4): no mechanism available: No worthy mechs found (general)
  • yum -y install cyrus-sasl-plain