返回首页
当前位置: 主页 > 编程语言 > C#教程 >

Flume-ng的HdfsSink出现Lease mismatch错误

时间:2017-06-04 22:41来源:电脑教程学习网 www.etwiki.cn 编辑:admin

多台Flume-ng Agent做HA,实际环境中出现Lease mismatch错误,具体报错如下:

11 Mar 2014 12:21:02,971 WARN [SinkRunner-PollingRunner-DefaultSinkProcessor] (
org.apache.flume.sink.hdfs.HDFSEventSink.process:418) - HDFS IO error
java.io.IOException: IOException flush:org.apache.hadoop.ipc.RemoteException(org
.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): Lease mismatch on /l
ogdata/2014/03/11/91pc/FlumeData.1394467200001.tmp owned by DFSClient_NONMAPREDU
CE_1235353284_49 but is accessed by DFSClient_NONMAPREDUCE_1765113176_43
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSName
system.java:2459)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSName
system.java:2437)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.fsync(FSNamesyste
m.java:3106)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.fsync(NameNo
deRpcServer.java:823)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTra
nslatorPB.fsync(ClientNamenodeProtocolServerSideTranslatorPB.java:714)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$Cl
ientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:4501
0)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.cal
l(ProtobufRpcEngine.java:453)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1701)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1697)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1408)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1695)

at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.ja
va:1643)
at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:15
25)
at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1510
)
at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:
116)
at org.apache.flume.sink.hdfs.HDFSDataStream.sync(HDFSDataStream.java:11
7)
at org.apache.flume.sink.hdfs.BucketWriter$5.call(BucketWriter.java:356)

at org.apache.flume.sink.hdfs.BucketWriter$5.call(BucketWriter.java:353)

at org.apache.flume.sink.hdfs.BucketWriter$8$1.run(BucketWriter.java:536
)
at org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.ja
va:160)
at org.apache.flume.sink.hdfs.BucketWriter.access$1000(BucketWriter.java
:56)
at org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:533)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:724)

 


这个是由于配置了多个Flume agent做HA,每个agent都有多个HDFS客户端。多个HDFS客户端同时向一个HDFS文件写数据的时候造成Lease租约问题,Lease可以认为是一个文件写锁。解决方法是配置每个Agent的HDFS文件前缀后者后缀
 

------分隔线----------------------------
标签(Tag):c#
------分隔线----------------------------
推荐内容
猜你感兴趣