代码之家  ›  专栏  ›  技术社区  ›  Josh Laird

从coderregistry推断编码程序失败:无法为com.google.api.services.bigquery.model.tablerow提供编码程序

  •  0
  • Josh Laird  · 技术社区  · 7 年前

    我有一个在本地运行时运行良好的数据流作业,使用 DataFlowRunner 但当我尝试使用gcp的composer/airflow运行它时,它会给我一个错误:

    Exception in thread "main" java.lang.IllegalStateException: Unable to return a default Coder for ConvertToYouTubeMetadata/ParDo(convertToTableRow$1)/ParMultiDo(convertToTableRow$1).output [PCollection]. Correct one of the following root causes:
        No Coder has been manually specified;  you may do so using .setCoder().
        Inferring a Coder from the CoderRegistry failed: Unable to provide a Coder for com.google.api.services.bigquery.model.TableRow.
        Building a Coder using a registered CoderProvider failed.
        See suppressed exceptions for detailed failures.
        Using the default output Coder from the producing PTransform failed: PTransform.getOutputCoder called.
        at org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
        at org.apache.beam.sdk.values.PCollection.getCoder(PCollection.java:259)
        at org.apache.beam.sdk.values.PCollection.finishSpecifying(PCollection.java:107)
        at org.apache.beam.sdk.runners.TransformHierarchy.finishSpecifyingInput(TransformHierarchy.java:190)
        at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:536)
        at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:491)
        at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:299)
        at MainKt.runMetadataPipeline(main.kt:66)
        at MainKt.main(main.kt:34)
    

    作曲家在本地运行时的执行方式有何不同?

    我只是在用

    BigQueryIO.writeTableRows()

    1 回复  |  直到 7 年前
        1
  •  1
  •   Josh Laird    7 年前

    ShadowJar DataFlowJavaOperator

    推荐文章