代码之家  ›  专栏  ›  技术社区  ›  Tshilidzi Mudau Simon Müller

“JavaPackage”对象不可调用-MLeap

  •  1
  • Tshilidzi Mudau Simon Müller  · 技术社区  · 8 年前

    当我尝试使用 MLeap 使用以下代码:

    import mleap.pyspark
    from mleap.pyspark.spark_support import SimpleSparkSerializer
    
    # Import standard PySpark Transformers and packages
    from pyspark.ml.feature import VectorAssembler, StandardScaler, OneHotEncoder, StringIndexer
    from pyspark.ml import Pipeline, PipelineModel
    from pyspark.sql import Row
    
    # Create a test data frame
    l = [('Alice', 1), ('Bob', 2)]
    rdd = sc.parallelize(l)
    Person = Row('name', 'age')
    person = rdd.map(lambda r: Person(*r))
    df2 = spark.createDataFrame(person)
    df2.collect()
    
    # Build a very simple pipeline using two transformers
    string_indexer = StringIndexer(inputCol='name', outputCol='name_string_index')
    
    feature_assembler = VectorAssembler(inputCols=[string_indexer.getOutputCol()], outputCol="features")
    
    feature_pipeline = [string_indexer, feature_assembler]
    
    featurePipeline = Pipeline(stages=feature_pipeline)
    
    fittedPipeline = featurePipeline.fit(df2)
    
    
    # serialize the model:
    fittedPipeline.serializeToBundle("jar:file:/tmp/pyspark.example.zip", fittedPipeline.transform(df2))
    

    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    <ipython-input-2-98a49e4cd023> in <module>()
    ----> 1 fittedPipeline.serializeToBundle("jar:file:/tmp/pyspark.example.zip", fittedPipeline.transform(df2))
    
    /opt/anaconda2/envs/py345/lib/python3.4/site-packages/mleap/pyspark/spark_support.py in serializeToBundle(self, path, dataset)
         22 
         23 def serializeToBundle(self, path, dataset=None):
    ---> 24     serializer = SimpleSparkSerializer()
         25     serializer.serializeToBundle(self, path, dataset=dataset)
         26 
    
    /opt/anaconda2/envs/py345/lib/python3.4/site-packages/mleap/pyspark/spark_support.py in __init__(self)
         37     def __init__(self):
         38         super(SimpleSparkSerializer, self).__init__()
    ---> 39         self._java_obj = _jvm().ml.combust.mleap.spark.SimpleSparkSerializer()
         40 
         41     def serializeToBundle(self, transformer, path, dataset):
    
    TypeError: 'JavaPackage' object is not callable
    

    请帮忙?

    1 回复  |  直到 8 年前
        1
  •  3
  •   Tshilidzi Mudau Simon Müller    8 年前

    我通过下载并指向spark submit脚本中丢失的jar文件来解决这个问题。在我的例子中,我安装了MLeap 0.8.1并使用了Scalar11上构建的Spark2,所以我从 MvnRepository :

    • metrics-core-2.2.0标准
    • mleap-base U 2.11-0.8.1版
    • mleap-核心2.11-0.8.1
    • mleap-spark U 2.11-0.8.1公司

    然后我还使用 --jar --repository 标志):

    export PYSPARK_SUBMIT_ARGS='--master yarn --deploy-mode client --driver-memory 40g --num-executors 15 --executor-memory 30g --executor-cores 5 --packages ml.combust.mleap:mleap-runtime_2.11:0.8.1 --repositories http://YOUR MAVEN REPO/ --jars arpack_combined_all-0.1.jar,mleap-base_2.11-0.8.1.jar,mleap-core_2.11-0.8.1.jar,mleap-runtime_2.11-0.8.1.jar,mleap-spark_2.11-0.8.1.jar,mleap-spark-base_2.11-0.8.1.jar,mleap-tensor_2.11-0.8.1.jar pyspark-shell'
    jupyter notebook --no-browser --ip=$(hostname -f)
    

    Source

    推荐文章