如何使用sc.addPyFile()在pyspark中导入matplotlib python库?

我将发布我所做的事情。首先,我是和你一起工作的 virtualenv virtualenv path .

然后我启动了它 source path/bin/activate .

我安装了我需要的软件包 pip3 install packageName

之后,我用python创建了一个脚本,该脚本创建了路径中随virtualenv一起安装的库的zip存档 ./path/lib/python3.7/site-packages/

此脚本的代码如下所示(仅限压缩) numpy ):

import zipfile
import os

#function to archive a single package
def ziplib(general_path, libName):

   libpath = os.path.dirname(general_path + libName)      # this should point to your packages directory 
   zippath = libName  + '.zip'      # some random filename in writable directory
   zf = zipfile.PyZipFile(zippath, mode='w')
   try:
       zf.debug = 3             # making it verbose, good for debugging 
       zf.writepy(libpath)
       return zippath           # return path to generated zip archive
   finally:
       zf.close()


general_path = './path//lib/python3.7/site-packages/'
matplotlib_name = 'matplotlib'
seaborn_name = 'seaborn'
numpy_name = 'numpy'
zip_path = ziplib(general_path, numpy_name)      # generate zip archive containing your lib                            
print(zip_path)

之后,必须在pyspark文件中引用归档文件 myPyspark.py . 您可以通过调用方法来实现这一点 addPyFile() sparkContext 班之后,您可以像往常一样导入代码。就我而言,我做了以下工作:

from pyspark import SparkContext
sc = SparkContext.getOrCreate()
sc.addPyFile("matplot.zip") #generate with testZip.py
sc.addPyFile("numpy.zip") #generate with testZip.py
import matplotlib
import numpy

--py-files . 例如:

sudo spark-submit --py-files matplot.zip --py-files numpy.zip myPyspark.py

您可以压缩matplotlib目录并将其传递给 addPyFile() export PYTHONPATH="venv//lib/python3.7/site-packages/:$PYTHONPATH"