代码之家 › 专栏 › 技术社区 › Popeye

通过spark submit使用本地发布的库

sbt apache-spark scala

0

Popeye · 技术社区 · 5 年前

我在scala中创建了一个库,并在本地部署该库。我已经记录了构建过程。下面是sbt


ThisBuild / name := "sqlparser"

ThisBuild / version := "0.1"

ThisBuild / organization := "xx.xx.xxxxxx.xxxxxx"

scalaVersion := "2.12.13"

idePackagePrefix := Some("xx.xx.xxxxxxx.xxxxxx.sqlparser")


libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.8" % Test

publishTo := some(Resolver.file("file",  new File(Path.userHome.absolutePath+"/.ivy2/local/"))(Resolver.ivyStylePatterns))

publishMavenStyle := false

我以前发布的命令

sbt publish

效验如神

现在有一个基于spark(scala)的项目,包括前面描述的包。Sbt如下所示

name := "brandinfoproducer"

version := "0.1"

scalaVersion := "2.12.13"

idePackagePrefix := Some("xx.xx.xxxxxxx.xxxxxx.brandinfoproducer")

libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.8" % Test

libraryDependencies += "com.google.guava" % "guava" % "30.1-jre"

libraryDependencies += "org.apache.spark" %% "spark-core" % "3.1.0"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.1.0"

resolvers += Resolver.file("file",  new File(Path.userHome.absolutePath+"/.ivy2/local/"))(Resolver.ivyStylePatterns)

libraryDependencies += "xx.xx.xxxxxxx.xxxxxx" %% "sqlparser" % "0.1"

编译和打包命令

sbt clean compile package

完美地编译和构建。

然后我使用下面的命令启动spark作业

我执行以下命令

/home/user_home/hdp26_c4000_stg/spark2/bin/spark-submit \
  --name "XXX_XXXXXXXX_XXXXXXXXXX_XXX" \
  --driver-java-options "-Dspring.profiles.active=stg -Dsource=MEDIA -Dspark.executor.memory=500m -Dspark.driver.memory=1g -Djava.security.krb5.conf=${KRB5_CONFIG}" \
  --master local[*] \
  /home/user_home/brandinfoproducer/target/scala-2.12/brandinfoproducer_2.12-0.1.jar

我得到以下错误

Exception in thread "main" java.lang.NoClassDefFoundError: xx/xx/xxxxxx/xxxxxx/sqlparser/SQLParser$
        at xx.xx.xxxxxxx.xxxxxx.brandinfoproducer.BrandInfoProducer$.main(BrandInfoProducer.scala:9)
        at xx.xx.xxxxxxx.xxxxxx.brandinfoproducer.BrandInfoProducer.main(BrandInfoProducer.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:750)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: xx.xx.xxxxxxx.xxxxxx.sqlparser.SQLParser$
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 11 more

问题非常清楚。但不是解决方案。我正在努力找到答案;无法找到合适的解决方案。

我会很感激你的解决方案。此外,作为scala和sbt的新用户,我的构建中可能存在一些明显的奇怪之处。sbt;请告诉我。

0 回复 | 直到 5 年前