使用笔记本时将jar添加到pyspark

我正在尝试mongodb hadoop与spark的集成,但无法弄清楚如何让IPython笔记本可以访问这些jar。

这里是我想做的:

# set up parameters for reading from MongoDB via Hadoop input format
config = {"mongo.input.uri": "mongodb://localhost:27017/db.collection"}
inputFormatClassName = "com.mongodb.hadoop.MongoInputFormat"

# these values worked but others might as well
keyClassName = "org.apache.hadoop.io.Text"
valueClassName = "org.apache.hadoop.io.MapWritable"

# Do some reading from mongo
items = sc.newAPIHadoopRDD(inputFormatClassName, keyClassName, valueClassName, None, None, config)

当我使用以下命令在pyspark中启动该代码时,该代码工作正常:

spark-1.4.1/bin/pspark--jars“mongo-hadoop-core-1.4.0.jar,mongo-java-driver-3.0.2.jar”

哪里 mongo-hadoop-core-1.4.0.jar 和 mongo-java-driver-2.10.1.jar 允许使用java中的mongodb。然而,当我这样做时:

IPYTHON_OPTS=“笔记本”spark-1.4.1/bin/pypark--jars'mongo-hadoop-core-1.4.0.jar,mongo-java-driver-3.0.2.jar'

罐子不再可用,我收到以下错误:

java.lang.ClassNotFoundException:com.mongodb.hadoop.MongoInputFormat

有人知道如何在IPython笔记本电脑中为火花提供罐子吗?我很确定这不是mongo特有的,所以也许有人已经在使用笔记本时成功地将jar添加到类路径中了?