代码之家  ›  专栏  ›  技术社区  ›  Bikas Katwal

SpringSpark集成-java。木卫一。NotSerializableException:org。springframework。上下文注释。注释ConfigApplicationContext

spring-boot apache-spark spring
  •  0
  • Bikas Katwal  · 技术社区  · 8 年前

    我正在为我的spark应用程序使用spring引导,所有依赖项都通过spring管理,我正在使用Autowire添加依赖项。提交给执行器的我的函数类和自定义类实现了可序列化。

    但当我运行它并将任务提交给执行者时,它抛出异常:一个spring类是不可序列化的- AnnotationConfigApplicationContext 

    Caused by: java.io.NotSerializableException: org.springframework.context.annotation.AnnotationConfigApplicationContext

    堆栈跟踪: 

    java.lang.IllegalStateException: Failed to execute CommandLineRunner
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:800) [spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:781) [spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:335) [spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255) [spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243) [spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
    at com.bikas.MyStarter(MyStarter:67) [classes/:?]
Caused by: org.apache.spark.SparkException: Task not serializable
    at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:345) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:335) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:159) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.SparkContext.clean(SparkContext.scala:2299) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:928) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:927) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:363) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:927) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.api.java.JavaRDDLike$class.foreachPartition(JavaRDDLike.scala:219) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.api.java.AbstractJavaRDDLike.foreachPartition(JavaRDDLike.scala:45) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at com.bikas.MyStarter.run(MyStarter:81) ~[classes/:?]
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:797) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
    ... 5 more
Caused by: java.io.NotSerializableException: org.springframework.context.annotation.AnnotationConfigApplicationContext
Serialization stack:
    - object not serializable (class: org.springframework.context.annotation.AnnotationConfigApplicationContext, value: org.springframework.context.annotation.AnnotationConfigApplicationContext@33a55bd8: startup date [Sun Aug 12 13:59:34 IST 2018]; root of context hierarchy)
    - field (class: com.bikas.services.MyServiceImpl, name: applicationContext, type: interface org.springframework.context.ApplicationContext)
    - object (class com.bikas.services.MyServiceImpl, com.bikas.services.MyServiceImpl@1260c85e)
    - field (class: com.bikas.services.MyProcessor, name: myServiceImpl, type: interface com.bikas.services.MyService)
    - object (class com.bikas.services.MyProcessor, com.bikas.services.MyProcessor@2b551e7b)
    - field (class: org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1, name: f$12, type: interface org.apache.spark.api.java.function.VoidFunction)
    - object (class org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1, <function1>)
    at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:342) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:335) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:159) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.SparkContext.clean(SparkContext.scala:2299) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:928) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:927) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:363) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:927) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.api.java.JavaRDDLike$class.foreachPartition(JavaRDDLike.scala:219) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at org.apache.spark.api.java.AbstractJavaRDDLike.foreachPartition(JavaRDDLike.scala:45) ~[spark-core_2.11-2.3.1.jar:2.3.1]
    at com.bikas.MyStarter.run(MyStarter.java:81) ~[classes/:?]
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:797) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]

    有什么帮助吗?还是其他人面临这个问题?

    1 回复  |  直到 8 年前
        1
  •  1
  •   DW.    8 年前

    包含在执行器上运行的函数的函数和任何类都不应该依赖于Spring引导。看看是否可以通过检查找到引用,或者序列化堆栈跟踪可能会帮助您。将在执行器上运行的函数声明为专用类中的静态方法是避免此问题的好方法。

        2
  •  0
  •   Hakob Hakobyan    4 年前

    将编写我找到的解决方法。 您可以在要调用的组件上使用spring组件包装器。 f、 e。 

    
@Component
public class ComponentIWantToUseWrapper {

    private static ComponentIWantToUse componentIWantToUse;

    public ComponentIWantToUseWrapper(ComponentIWantToUse component) {
        ComponentIWantToUseWrapper.componentIWantToUse = component;
    }

    public static ComponentIWantToUse getComponent() {
        return component;
    }
}

    因此,组件初始化期间,spring将更改包装器的静态变量。您现在可以像这样在spark lambdas中使用您的组件 ComponentIWantToUseWrapper.getComponent().someMethod() . 希望能有帮助。

    推荐文章
    PatPanda  ·  使用springboot验证请求中的列表是否为[null]
    1 年前
    Bojidar Kaloyanov  ·  为什么即使有有效的JWT,Spring Boot也会对安全端点的POST请求返回403 Forbidden?
    1 年前
    mikeb  ·  Maven和Spring引导-我需要强制logback 1.5.13,但不能
    1 年前
    Nuñito Calzada  ·  AWS SDK 1.12.780 Java版的凭据
    1 年前
    Python_user  ·  build/lib不是由gradle build生成的
    1 年前
    Chirag Arora  ·  如何在构造函数中使用注入的依赖关系?
    1 年前
    sree  ·  Spring Mongo仓库:通过传递id列表来获取对象列表
    1 年前
    David  ·  如何修复javax/xml/bind/DatatypeConverter的java.lang.NoClassDefFoundError错误
    2 年前
    RobertC  ·  Spring中默认实现的Autowire接口
    2 年前
    molLbach  ·  向Spring后端发送axios post请求时出现400错误
    2 年前
    关于   移动版
    代码之家 - 一站式码农服务社区
    沪ICP备11025650号