[repost ] hadoop集群升级手札

original:http://blog.csdn.net/rzhzhz/article/details/7701700

之前集群的配置为hadoop-0.20.3,hbase-0.90.4,zookeeper-3.3.4,hive-0.8.1。hadoop还算稳定,基本没什么bug,而hive基于hbse查询时真是问题百出,hbase各种bug,比如丢数据,丢表,regionserver频繁宕机,各种打补丁,改错误搞得我脑袋都要爆了。于是决定给hbase来一个彻底的升级替换。

一.   先是把hbase升级为0.94.0,升级(就是安装包简单的替换)还算顺利,但启动的时候却报错

  1. 2012-06-26 15:59:19,051 WARN org.apache.hadoop.hbase.util.FSUtils: Unable to create cluster ID file in hdfs://server:9000/hbase, retrying in 10000msec: org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.protocol.ClientProtocol.create(java.lang.String, org.apache.hadoop.fs.permission.FsPermission, java.lang.String, boolean, boolean, short, long)
  2.         at java.lang.Class.getMethod(Class.java:1605)
  3.         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:517)
  4.         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1383)
  5.         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1379)
  6.         at java.security.AccessController.doPrivileged(Native Method)
  7.         at javax.security.auth.Subject.doAs(Subject.java:396)
  8.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
  9.         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1377)
  10.         at org.apache.hadoop.ipc.Client.call(Client.java:1066)
  11.         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
  12.         at $Proxy10.create(Unknown Source)
  13.         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  14.         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  15.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  16.         at java.lang.reflect.Method.invoke(Method.java:597)
  17.         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
  18.         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
  19.         at $Proxy10.create(Unknown Source)
  20.         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3245)
  21.         at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:713)
  22.         at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:182)
  23.         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
  24.         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
  25.         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
  26.         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:435)
  27.         at org.apache.hadoop.hbase.util.FSUtils.setClusterId(FSUtils.java:463)
  28.         at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:357)
  29.         at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
  30.         at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:112)
  31.         at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:480)
  32.         at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:343)
  33.         at java.lang.Thread.run(Thread.java:662)

查看源码可知hadoop-0.20.3中ClientProtocol.create方法如下

  1. public void create(String src, FsPermission masked,String clientName,
  2.                              boolean overwrite, short replication,long blockSize) throws IOException;

与hbase0.94.0中调用的create方法不兼容。

二.  所以只能把hadoop给升级了,查看了release版本,决定把0.20.3升级为1.0.3版本

hadoop升级不像hbase,涉及到数据版本可能不兼容(这在1.0版本之前尤为明显),所以相对hbase来说升级比较繁琐。

除了简单替换了hadoop安装包的文件之外,还得给HDFS的数据和元数据升级。

然而HDFS的数据和元数据的升级有一定的风险,可能会造成数据丢失等,所以在生产线上集群升级之前最好在测试集群测试通过才进行。下面说说升级步骤:

 

1.确保文件系统健康(fsck工具全面检查)

2.清空HDFS和Mapreduce的临时数据

3.确保上一次升级已经定妥。

4.关闭MapReduce,终止tasktracker上运行的任何孤儿任务。

5.关闭HDFS,并备份元数据

6.替换安装文件(集群所有机器)

7.使用-upgrade选项启动HDFS升级   (start-dfs.sh -upgrade)

8.等待,直到升级完成(hadoop dfsadmin -upgradeProgress status查看升级状态)

9.检查HDFS是否运行正常(fsck工具)

10.启动MapReduce

11.回滚(start-dfs.sh -roollback)或定妥升级(hadoop dfsadmin -finalizeUpgrade)

三.  升级zookeeper到3.4.3版本(过程略)

四.  启动hbase,此时会有如下提示

[plain] view plaincopy

  1. SLF4J: Class path contains multiple SLF4J bindings.
  2. SLF4J: Found binding in [jar:file:/usr/local/hbase/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  3. SLF4J: Found binding in [jar:file:/usr/local/hadoop/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  4. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

因为hadoop lib下的slf4j-log4j12-1.5.8.jar于hbase lib的slf4j-log4j12-1.4.3.jar冲突,删除Hbase lib下的slf4j-log4j12-1.4.3.jar即可。

此时hbase能正常启动。升级完毕。

 

五.  替换hive中相关的jar包,及配置文件,启动hive的时候,又出现了问题

  1. 2012-06-28 16:27:55,303 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.core.resources” but it cannot be resolved.
  2. 2012-06-28 16:27:55,303 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.core.resources” but it cannot be resolved.
  3. 2012-06-28 16:27:55,307 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.core.runtime” but it cannot be resolved.
  4. 2012-06-28 16:27:55,307 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.core.runtime” but it cannot be resolved.
  5. 2012-06-28 16:27:55,307 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.text” but it cannot be resolved.
  6. 2012-06-28 16:27:55,307 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.text” but it cannot be resolved.
  7. 2012-06-28 16:27:58,605 WARN  parse.SemanticAnalyzer (SemanticAnalyzer.java:genBodyPlan(5821)) - Common Gby keys:null
  8. 2012-06-28 16:27:58,850 WARN  hbase.HBaseConfiguration (HBaseConfiguration.java:<init>(48)) - instantiating HBaseConfiguration() is deprecated. Please use HBaseConfiguration#create() to construct a plain Configuration
  9. 2012-06-28 16:27:59,006 WARN  hbase.HBaseConfiguration (HBaseConfiguration.java:<init>(48)) - instantiating HBaseConfiguration() is deprecated. Please use HBaseConfiguration#create() to construct a plain Configuration
  10. 2012-06-28 16:27:59,539 ERROR CliDriver (SessionState.java:printError(380)) - Failed with exception java.io.IOException:java.lang.NullPointerException
  11. java.io.IOException: java.lang.NullPointerException
  12.         at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:341)
  13.         at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:154)
  14.         at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1383)
  15.         at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:266)
  16.         at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
  17.         at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
  18.         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
  19.         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
  20.         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  21.         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  22.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  23.         at java.lang.reflect.Method.invoke(Method.java:597)
  24.         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
  25. Caused by: java.lang.NullPointerException
  26.         at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:173)
  27.         at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:135)
  28.         at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:209)
  29.         at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:1)
  30.         at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:326)
  31.         … 12 more

Bundle “org.eclipse.jdt.core” requires系列的问题先不说,可以忽略,对hive的正常使用没影响。

首先以为是hive的版本不兼容hbase,于是悻悻呼把hive升级到0.9.0版本。

问题还是没解决,却出现了新的问题,如下

 

  1. 2012-06-28 11:16:47,822 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.core.resources” but it cannot be resolved.
  2. 2012-06-28 11:16:47,822 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.core.resources” but it cannot be resolved.
  3. 2012-06-28 11:16:47,826 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.core.runtime” but it cannot be resolved.
  4. 2012-06-28 11:16:47,826 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.core.runtime” but it cannot be resolved.
  5. 2012-06-28 11:16:47,826 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.text” but it cannot be resolved.
  6. 2012-06-28 11:16:47,826 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.text” but it cannot be resolved.
  7. 2012-06-28 11:16:52,709 WARN  client.ZooKeeperSaslClient (ZooKeeperSaslClient.java:<init>(123)) - SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
  8. 2012-06-28 11:16:52,720 WARN  zookeeper.ClientCnxn (ClientCnxn.java:run(1057)) - Session 0×0 for server null, unexpected error, closing socket connection and attempting reconnect
  9. java.net.ConnectException: Connection refused
  10.     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
  11.     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
  12.     at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
  13.     at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
  14. 2012-06-28 11:16:52,837 WARN  zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:retryOrThrow(195)) - Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master

说是连不上zookeeper,奇怪了配置文件啥都没变。最后在hive-site.xml文件中加上zookeeper的相关配置,问题解决。

[plain] view plaincopy

  1. <property>
  2.       <name>hbase.zookeeper.quorum</name>
  3.       <value>Debian,hadoop25,hadoop26</value>
  4.     </property>
  5.     <property>
  6.       <name>hbase.zookeeper.property.clientPort</name>
  7.       <value>2181</value>
  8.     </property>

这个问题解决,但是执行mapreduce的时候出现了新问题,task执行的时候报错

[plain] view plaincopy

  1. MapAttempt TASK_TYPE=”MAP” TASKID=”task_201206281426_0002_m_000000″ TASK_ATTEMPT_ID=”attempt_201206281426_0002_m_000000_0″ TASK_STATUS=”FAILED” FINISH_TIME=”1340876831790″ HOSTNAME=”Debian” ERROR=”Error: java\.lang\.ClassNotFoundException: com\.google\.protobuf\.Message
  2.     at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)
  3.     at java\.security\.AccessController\.doPrivileged(Native Method)
  4.     at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:190)
  5.     at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:306)
  6.     at sun\.misc\.Launcher$AppClassLoader\.loadClass(Launcher\.java:301)
  7.     at java\.lang\.ClassLoader\.loadClass(ClassLoader\.java:247)
  8.     at org\.apache\.hadoop\.hbase\.io\.HbaseObjectWritable\.<clinit>(HbaseObjectWritable\.java:263)
  9.     at org\.apache\.hadoop\.hbase\.ipc\.Invocation\.write(Invocation\.java:138)
  10.     at org\.apache\.hadoop\.hbase\.ipc\.HBaseClient$Connection\.sendParam(HBaseClient\.java:537)
  11.     at org\.apache\.hadoop\.hbase\.ipc\.HBaseClient\.call(HBaseClient\.java:898)
  12.     at org\.apache\.hadoop\.hbase\.ipc\.WritableRpcEngine$Invoker\.invoke(WritableRpcEngine\.java:150)
  13.     at $Proxy3\.getProtocolVersion(Unknown Source)
  14.     at org\.apache\.hadoop\.hbase\.ipc\.WritableRpcEngine\.getProxy(WritableRpcEngine\.java:183)
  15.     at org\.apache\.hadoop\.hbase\.ipc\.HBaseRPC\.getProxy(HBaseRPC\.java:303)
  16.     at org\.apache\.hadoop\.hbase\.ipc\.HBaseRPC\.getProxy(HBaseRPC\.java:280)
  17.     at org\.apache\.hadoop\.hbase\.ipc\.HBaseRPC\.getProxy(HBaseRPC\.java:332)
  18.     at org\.apache\.hadoop\.hbase\.ipc\.HBaseRPC\.waitForProxy(HBaseRPC\.java:236)
  19.     at org\.apache\.hadoop\.hbase\.client\.HConnectionManager$HConnectionImplementation\.getHRegionConnection(HConnectionManager\.java:1284)
  20.     at org\.apache\.hadoop\.hbase\.client\.HConnectionManager$HConnectionImplementation\.getHRegionConnection(HConnectionManager\.java:1240)
  21.     at org\.apache\.hadoop\.hbase\.client\.HConnectionManager$HConnectionImplementation\.getHRegionConnection(HConnectionManager\.java:1227)
  22.     at org\.apache\.hadoop\.hbase\.client\.HConnectionManager$HConnectionImplementation\.locateRegionInMeta(HConnectionManager\.java:936)
  23.     at org\.apache\.hadoop\.hbase\.client\.HConnectionManager$HConnectionImplementation\.locateRegion(HConnectionManager\.java:832)
  24.     at org\.apache\.hadoop\.hbase\.client\.HConnectionManager$HConnectionImplementation\.locateRegion(HConnectionManager\.java:801)
  25.     at org\.apache\.hadoop\.hbase\.client\.HConnectionManager$HConnectionImplementation\.locateRegionInMeta(HConnectionManager\.java:933)
  26.     at org\.apache\.hadoop\.hbase\.client\.HConnectionManager$HConnectionImplementation\.locateRegion(HConnectionManager\.java:836)
  27.     at org\.apache\.hadoop\.hbase\.client\.HConnectionManager$HConnectionImplementation\.locateRegion(HConnectionManager\.java:801)
  28.     at org\.apache\.hadoop\.hbase\.client\.HTable\.finishSetup(HTable\.java:234)
  29.     at org\.apache\.hadoop\.hbase\.client\.HTable\.<init>(HTable\.java:174)
  30.     at org\.apache\.hadoop\.hive\.hbase\.HiveHBaseTableInputFormat\.getRecordReader(HiveHBaseTableInputFormat\.java:92)
  31.     at org\.apache\.hadoop\.hive\.ql\.io\.HiveInputFormat\.getRecordReader(HiveInputFormat\.java:240)
  32.     at org\.apache\.hadoop\.hive\.ql\.io\.CombineHiveInputFormat\.getRecordReader(CombineHiveInputFormat\.java:522)
  33.     at org\.apache\.hadoop\.mapred\.MapTask$TrackedRecordReader\.<init>(MapTask\.java:197)
  34.     at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:418)
  35.     at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:372)
  36.     at org\.apache\.hadoop\.mapred\.Child$4\.run(Child\.java:255)
  37.     at java\.security\.AccessController\.doPrivileged(Native Method)
  38.     at javax\.security\.auth\.Subject\.doAs(Subject\.java:396)
  39.     at org\.apache\.hadoop\.security\.UserGroupInformation\.doAs(UserGroupInformation\.java:1121)
  40.     at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:249)
  41. ” .

因为缺少protobuf-java-2.4.0a.jar,故在hive-site.xml中配置该jar即可,如下

[plain] view plaincopy

  1. <property>
  2.   <name>hive.aux.jars.path</name>
  3.   <value>file:///usr/local/hive/lib/hive-hbase-handler-0.9.0.jar,file:///usr/local/hive/lib/hbase-0.92.1.jar,file:///usr/local/hive/lib/zookeeper-3.4.3.jar,file:///usr/local/hive/lib/protobuf-java-2.4.0a.jar</value>
  4. </property>

解决完该问题,问题又重新指向了原来hive-0.8.1版本遇到的问题

 

[plain] view plaincopy

  1. 2012-06-28 16:27:55,303 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.core.resources” but it cannot be resolved.
  2. 2012-06-28 16:27:55,303 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.core.resources” but it cannot be resolved.
  3. 2012-06-28 16:27:55,307 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.core.runtime” but it cannot be resolved.
  4. 2012-06-28 16:27:55,307 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.core.runtime” but it cannot be resolved.
  5. 2012-06-28 16:27:55,307 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.text” but it cannot be resolved.
  6. 2012-06-28 16:27:55,307 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle “org.eclipse.jdt.core” requires “org.eclipse.text” but it cannot be resolved.
  7. 2012-06-28 16:27:58,605 WARN  parse.SemanticAnalyzer (SemanticAnalyzer.java:genBodyPlan(5821)) - Common Gby keys:null
  8. 2012-06-28 16:27:58,850 WARN  hbase.HBaseConfiguration (HBaseConfiguration.java:<init>(48)) - instantiating HBaseConfiguration() is deprecated. Please use HBaseConfiguration#create() to construct a plain Configuration
  9. 2012-06-28 16:27:59,006 WARN  hbase.HBaseConfiguration (HBaseConfiguration.java:<init>(48)) - instantiating HBaseConfiguration() is deprecated. Please use HBaseConfiguration#create() to construct a plain Configuration
  10. 2012-06-28 16:27:59,539 ERROR CliDriver (SessionState.java:printError(380)) - Failed with exception java.io.IOException:java.lang.NullPointerException
  11. java.io.IOException: java.lang.NullPointerException
  12.         at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:341)
  13.         at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:154)
  14.         at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1383)
  15.         at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:266)
  16.         at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
  17.         at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
  18.         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
  19.         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
  20.         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  21.         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  22.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  23.         at java.lang.reflect.Method.invoke(Method.java:597)
  24.         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
  25. Caused by: java.lang.NullPointerException
  26.         at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:173)
  27.         at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:135)
  28.         at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:209)
  29.         at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:1)
  30.         at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:326)
  31.         … 12 more

从错误内容更可以看出因为hbase高版本的代码出现了更改导致hive在调用hbase的api时出现了NPE的错误,看样子Hbase0.9.4还没有得到hive的完美支持。于是呼貌似只能把hbase降级为hive0.9.0原生支持的hbase-0.92版本,虽说很不甘心,但结果往往是残酷的。

在给Hbase降级的时候出现了一个致命的问题,整个之前hbase的数据都丢失了,所有的regionserver没有负载任何region,看样子是hbase的meta出现了问题,也有可能是我瞎折腾导致的,没办法项目急着用,而且我的数据恢复也不是很费劲,所以只能先重新导入数据了,备份了hbase的meta,有时间得好好分析下是什么导致出现了此问题。

 

因为我是采用的flume采集的数据,在涉及到连接hadoop的地方出现错误

  1. java.io.IOException: failure to login
  2.         at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:490)
  3.         at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:452)
  4.         at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
  5.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  6.         at java.lang.reflect.Method.invoke(Method.java:597)
  7.         at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
  8.         at org.apache.hadoop.hbase.security.User.call(User.java:586)
  9.         at org.apache.hadoop.hbase.security.User.callStatic(User.java:576)
  10.         at org.apache.hadoop.hbase.security.User.access$400(User.java:50)
  11.         at org.apache.hadoop.hbase.security.User$SecureHadoopUser.<init>(User.java:393)
  12.         at org.apache.hadoop.hbase.security.User$SecureHadoopUser.<init>(User.java:388)
  13.         at org.apache.hadoop.hbase.security.User.getCurrent(User.java:139)
  14.         at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280)
  15.         at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332)
  16.         at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236)
  17.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1278)
  18.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1235)
  19.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1222)
  20.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:918)
  21.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:814)
  22.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:788)
  23.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1024)
  24.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818)
  25.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782)
  26.         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249)
  27.         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213)
  28.         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
  29.         at org.mostar.flume.hbase.MetaHBaseSink.open(MetaHBaseSink.java:138)
  30.         at com.cloudera.flume.core.EventSinkDecorator.open(EventSinkDecorator.java:75)
  31.         at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:88)
  32. Caused by: javax.security.auth.login.LoginException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.KerberosName
  33.         at org.apache.hadoop.security.User.<init>(User.java:44)
  34.         at org.apache.hadoop.security.User.<init>(User.java:39)
  35.         at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:130)
  36.         at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  37.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  38.         at java.lang.reflect.Method.invoke(Method.java:597)
  39.         at javax.security.auth.login.LoginContext.invoke(LoginContext.java:769)
  40.         at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
  41.         at javax.security.auth.login.LoginContext$5.run(LoginContext.java:706)
  42.         at java.security.AccessController.doPrivileged(Native Method)
  43.         at javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:703)
  44.         at javax.security.auth.login.LoginContext.login(LoginContext.java:576)
  45.         at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:471)
  46.         at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:452)
  47.         at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
  48.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  49.         at java.lang.reflect.Method.invoke(Method.java:597)
  50.         at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
  51.         at org.apache.hadoop.hbase.security.User.call(User.java:586)
  52.         at org.apache.hadoop.hbase.security.User.callStatic(User.java:576)
  53.         at org.apache.hadoop.hbase.security.User.access$400(User.java:50)
  54.         at org.apache.hadoop.hbase.security.User$SecureHadoopUser.<init>(User.java:393)
  55.         at org.apache.hadoop.hbase.security.User$SecureHadoopUser.<init>(User.java:388)
  56.         at org.apache.hadoop.hbase.security.User.getCurrent(User.java:139)
  57.         at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280)
  58.         at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332)
  59.         at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236)
  60.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1278)
  61.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1235)
  62.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1222)
  63.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:918)
  64.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:814)
  65.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:788)
  66.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1024)
  67.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818)
  68.         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782)
  69.         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249)
  70.         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213)
  71.         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
  72.         at org.mostar.flume.hbase.MetaHBaseSink.open(MetaHBaseSink.java:138)
  73.         at com.cloudera.flume.core.EventSinkDecorator.open(EventSinkDecorator.java:75)
  74.         at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:88)
  75.         at javax.security.auth.login.LoginContext.invoke(LoginContext.java:872)
  76.         at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
  77.         at javax.security.auth.login.LoginContext$5.run(LoginContext.java:706)
  78.         at java.security.AccessController.doPrivileged(Native Method)
  79.         at javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:703)
  80.         at javax.security.auth.login.LoginContext.login(LoginContext.java:576)
  81.         at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:471)
  82.         … 29 more

此时是因为hadoop-core-1.0.3.jar不包含org.apache.hadoop.security.KerberosName类,把commons-configuration-1.6.jar复制到当前应用的lib下即可。

然后hadoop集群升级最终完成。由hadoop-0.20.3,hbase-0.90.4,zookeeper-3.3.4,hive-0.8.1升级为hadoop-1.0.3,hbase-0.92.1,zookeeper-3.4.3,hive-0.9.0。期待Hbase能表现得出色点。