代码之家  ›  专栏  ›  技术社区  ›  Julien Séveno-Piltant

Python hdfscli-服务器IPC版本9无法与客户端版本4通信

urllib hdfs hadoop python-3.x python
  •  0
  • Julien Séveno-Piltant  · 技术社区  · 4 年前

    我试图从Python连接到本地运行的Hadoop集群,但一直收到此错误: requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine('\x00\x00\x00\x7f~\x08ÿÿÿÿ\x0f\x10\x02\x18\t")org.apache.hadoop.ipc.RPC$VersionMismatch*>Server IPC version 9 cannot communicate with client version

    这是我的代码: `从hdfs导入配置

    类HDFS客户端: _实例=无 

    def __init__(self, env):
    self._client = Config().get_client(env)

def write_data(self, data, hdfs_filename):
    """
    Write data to an HDFS file.
    """
    pass

def write_data_stream(self, iterator, hdfs_filename):
    """
    Write data to an HDFS file.
    Data is retrieved iteratively from @iterator.
    """
    self._client.write(hdfs_filename, iterator)

@staticmethod
def get_instance(env="dev"):
    if HDFSClient._instance:
        return HDFSClient._instance
    HDFSClient._instance = HDFSClient(env)
    return HDFSClient._instance`

    我的配置文件也很简单: 

    [global]
default.alias = dev

[dev.alias]
url = http://localhost:9000

[prod.alias]
url = http://localhost:9000

    以下是完整的错误: 

    Traceback (most recent call last):
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\urllib3\connectionpool.py", line 436, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
    # Permission is hereby granted, free of charge, to any person obtaining a copy
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\http\client.py", line 1347, in getresponse
    response.begin()
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\http\client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\http\client.py", line 289, in _read_status
    raise BadStatusLine(line)
http.client.BadStatusLine: 

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\urllib3\util\retry.py", line 531, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\urllib3\packages\six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
    # Permission is hereby granted, free of charge, to any person obtaining a copy
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\http\client.py", line 1347, in getresponse
    response.begin()
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\http\client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\http\client.py", line 289, in _read_status
    raise BadStatusLine(line)
urllib3.exceptions.ProtocolError: ('Connection aborted.', BadStatusLine('\x00\x00\x00\x7f~\x08ÿÿÿÿ\x0f\x10\x02\x18\t")org.apache.hadoop.ipc.RPC$VersionMismatch*>Server IPC version 9 cannot communicate with client version 470\x0e:\x00@\x01Hõ*'))

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\flask\app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\flask\app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\flask\app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\flask\_compat.py", line 39, in reraise
    raise value
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\flask\app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\flask\app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\flask\app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\flask\_compat.py", line 39, in reraise
    raise value
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\flask\app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\flask\app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "C:\Users\Julien\Documents\Travail\PanoptesTechnologies\block-analysis-backend\api\src\with_cors.py", line 12, in decorated_function
    return f(*args, **kwargs)
  File "C:\Users\Julien\Documents\Travail\PanoptesTechnologies\block-analysis-backend\api\utils\log.py", line 37, in decorated
    return f(*args, **kwargs)
  File "C:\Users\Julien\Documents\Travail\PanoptesTechnologies\block-analysis-backend\api\app.py", line 154, in heuristic
   #=======================================================================================================================
File "C:\Users\Julien\Documents\Travail\PanoptesTechnologies\block-analysis-backend\api\utils\hadoop.py", line 21, in write_data_stream
    self._client.write(hdfs_filename, iterator)
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\hdfs\client.py", line 459, in write
    res = self._create(
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\hdfs\client.py", line 125, in api_handler
    raise err
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\hdfs\client.py", line 102, in api_handler
    res = client._request(
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\hdfs\client.py", line 209, in _request
    return self._session.request(
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\requests\sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\Julien\AppData\Local\Programs\Python\Python38\Lib\site-packages\requests\adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine('\x00\x00\x00\x7f~\x08ÿÿÿÿ\x0f\x10\x02\x18\t")org.apache.hadoop.ipc.RPC$VersionMismatch*>Server IPC version 9 cannot communicate with client version 470\x0e:\x00@\x01Hõ*'))

    Python版本:3.8 hdfscli版本:2.6.0 Hadoop版本:3.2.1

    我已经检查了hdsfcli所需的软件包版本。

    有什么我可以尝试解决的吗? 谢谢你的帮助。

    最佳赛格,

    0 回复  |  直到 4 年前
        1
  •  0
  •   Matheus Ferreira da Rocha    3 年前

    嗯,我也有过同样的挣扎。 上下文: 此包使用webhdfs API。在其他版本上,它运行在50070端口上,但对于Hadoop 3.2.1和3.3.1(我的版本),它运行的是9870端口。

    只需更改: 

    [dev.alias]
url = http://localhost:9000

[prod.alias]
url = http://localhost:9000

    致: 

    [dev.alias]
url = http://localhost:9870

[prod.alias]
url = http://localhost:9870

    你是金色的。

    推荐文章
    Ξένη Γήινος  ·  Python lxml.html语法错误:使用lxml find时XPATH的谓词无效
    7 月前
    Cam  ·  Pandas列表日期到日期时间
    7 月前
    RASEL MAHMUD  ·  为什么以及如何在is_even()函数内的IF条件中递归X变量在满足0后递增?[副本]
    7 月前
    jjkennedy  ·  Pandas文本文件导入:当每个文件中存在多个表时,自动选择1个表
    7 月前
    LMC  ·  Numpy数组布尔索引以获取包含元素
    8 月前
    vr8ce  ·  非成对标记中特定字符的正则表达式
    8 月前
    Kernel  ·  如果指定了crs参数,shapefile的geopandas.read_file将出错
    8 月前
    ShaAnder  ·  为什么sqllachemy返回的是类而不是字符串
    8 月前
    sixtytrees  ·  detectron2软件包未安装(没有名为“torch”的模块),但我安装了torch
    8 月前
    Pernoctador  ·  Python映射可以复制吗?我需要参考地图
    9 月前
    关于   移动版
    代码之家 - 一站式码农服务社区
    沪ICP备11025650号