代码之家  ›  专栏  ›  技术社区  ›  lampShadesDrifter

H2O H2OServerError:培训模型时HTTP 500服务器错误

h2o
  •  0
  • lampShadesDrifter  · 技术社区  · 7 年前

    尝试在h2o(版本3.20.0.5)中训练DRF分类器时,出现错误“H2OServerError:HTTP500服务器错误”,没有进一步解释。

    ---------------------------------------------------------------------------
    H2OServerError                            Traceback (most recent call last)
    <ipython-input-44-f52d1cb4b77a> in <module>()
          4     training_frame=train_u, validation_frame=val_u,
          5     weights_column='weight',
    ----> 6     max_runtime_secs=max_train_time_hrs*60*60)
          7 
          8 
    
    /home/mapr/python-virtual-envs/ml1c/venv/lib/python2.7/site-packages/h2o/estimators/estimator_base.pyc in train(self, x, y, training_frame, offset_column, fold_column, weights_column, validation_frame, max_runtime_secs, ignored_columns, model_id, verbose)
        224         rest_ver = parms.pop("_rest_version") if "_rest_version" in parms else 3
        225 
    --> 226         model_builder_json = h2o.api("POST /%d/ModelBuilders/%s" % (rest_ver, self.algo), data=parms)
        227         model = H2OJob(model_builder_json, job_type=(self.algo + " Model Build"))
        228 
    
    /home/mapr/python-virtual-envs/ml1c/venv/lib/python2.7/site-packages/h2o/h2o.pyc in api(endpoint, data, json, filename, save_to)
        101     # type checks are performed in H2OConnection class
        102     _check_connection()
    --> 103     return h2oconn.request(endpoint, data=data, json=json, filename=filename, save_to=save_to)
        104 
        105 
    
    /home/mapr/python-virtual-envs/ml1c/venv/lib/python2.7/site-packages/h2o/backend/connection.pyc in request(self, endpoint, data, json, filename, save_to)
        400                                     auth=self._auth, verify=self._verify_ssl_cert, proxies=self._proxies)
        401             self._log_end_transaction(start_time, resp)
    --> 402             return self._process_response(resp, save_to)
        403 
        404         except (requests.exceptions.ConnectionError, requests.exceptions.HTTPError) as e:
    
    /home/mapr/python-virtual-envs/ml1c/venv/lib/python2.7/site-packages/h2o/backend/connection.pyc in _process_response(response, save_to)
        728         # Note that it is possible to receive valid H2OErrorV3 object in this case, however it merely means the server
        729         # did not provide the correct status code.
    --> 730         raise H2OServerError("HTTP %d %s:\n%r" % (status_code, response.reason, data))
        731 
        732 
    
    H2OServerError: HTTP 500 Server Error:
    Server error java.lang.NullPointerException:
      Error: Caught exception: java.lang.NullPointerException
      Request: None
    

    max_train_time_hrs = 8 
    drf_proc.train(
        x=train_features, y=train_response,
        training_frame=train_u, validation_frame=val_u,
        weights_column='weight',
        max_runtime_secs=max_train_time_hrs*60*60)
    

    运行 h2o.init() 命令看起来像

    Checking whether there is an H2O instance running at http://172.18.4.62:54321. connected.
    Warning: Your H2O cluster version is too old (7 months and 24 days)! Please download and install the latest version from http://h2o.ai/download/
    H2O cluster uptime: 06 secs
    H2O cluster timezone:   Pacific/Honolulu
    H2O data parsing timezone:  UTC
    H2O cluster version:    3.20.0.5
    H2O cluster version age:    7 months and 24 days !!!
    H2O cluster name:   H2O_88021
    H2O cluster total nodes:    4
    H2O cluster free memory:    15.34 Gb
    H2O cluster total cores:    8
    H2O cluster allowed cores:  8
    H2O cluster status: accepting new members, healthy
    H2O connection url: http://172.18.4.62:54321
    H2O connection proxy:   None
    H2O internal security:  False
    H2O API Extensions: AutoML, XGBoost, Algos, Core V3, Core V4
    Python version: 2.7.12 fin
    

    关于这里可能发生的事情或者可以研究的调试步骤,有什么想法吗?

    0 回复  |  直到 7 年前
        1
  •  2
  •   Darren Cook    7 年前

    15GB的内存可能不足以满足您期望持续8小时的训练过程。(旁白:我建议使用 early stopping max_runtime_secs .)

    作为一个调试步骤,我建议您在流界面中观察(将浏览器指向端口54321-查看您的浏览器中的连接URL) h2o.init()

    (有时一个“500”错误仅仅意味着它变得不稳定,内存不足是一个常见的触发因素。)

    如果您立即得到错误,那么问题就不大可能出现(除非您有一个庞大的数据集)。

    在这种情况下,我会尽量缩小某个特定列或数据行是否可能导致问题的范围。例如。

    • train_features
    • 实验2:第二部分的列 列车功能
    • 实验3:实验中的前半行 train_u
    • 实验4:实验中的第二部分行 火车站
    • 实验5/6(如果仍然没有运气):同样的 valid_u

    如果实验对中的一个崩溃了,而另一个没有,那么对崩溃的那一半重复实验。