代码之家  ›  专栏  ›  技术社区  ›  Explorer

Avro未填充数组类型的方括号

  •  1
  • Explorer  · 技术社区  · 7 年前

    {
    "name": "schema_name",
    "type": "record",
    "fields" : [
    {
    "name": "schema",
    "type": "string"
    },
    {
      "name": "data",
      "type": {
        "type": "array",
        "items":
          {
            "name": "data",
            "type": "record",
            "fields": [
              {
                "name": "phone_number",
                "type": "string"
              }
            ]
          }
      }
    },
    {
      "name":"flag",
      "type":"string"
    }
    ]
    }
    

    我用它从文本文件生成Avro消息:

    def main(args: Array[String]): Unit = {
      val avroSchemaStr = Source.fromFile("avro_schema.txt").mkString
      val avroSchema = new Schema.Parser().parse(avroSchemaStr)
    
      Source.fromFile("phone_numbers.txt").getLines.foreach { msg =>
        println(fixedWidthToAvro(msg, avroSchema))
      }
    }
    
    def fixedWidthToAvro(record: String, avroSchema: Schema): GenericRecord = {
      val childSchema = new GenericData.Record(avroSchema).getSchema.getField("data").schema.getElementType
      val parentRrecord = new GenericData.Record(avroSchema)
      val childRecord = new GenericData.Record(childSchema)
    
      childRecord.put("phone_number", "1234567890")
      parentRrecord.put("schema", "schema_name")
      parentRrecord.put("data", childRecord)
      parentRrecord.put("flag", "I")
    
      println(parentRrecord)
      parentRrecord
    }
    

    一切正常,对于给定的消息,我得到以下输出:

    {"schema": "schema_name", "data": {"phone_number": "1234567890"}, "flag": "I"}
    

    然而,正如我宣布的 data 字段类型为 array

    {"schema": "schema_name", "data": [{"phone_number": "1234567890"}], "flag": "I"}
    

    我想要那个 数据

    1 回复  |  直到 7 年前
        1
  •  1
  •   Yuval Itzchakov    7 年前

    data . 一个是数组,另一个是数组中的元素名,我想这就是让你困惑的地方。

    当你经过的时候 schema.getElementType Record Array[Record] 保存所有这些记录。

    您需要的是一个阵列,它将容纳您的所有记录:

    val avroSchema = new Schema.Parser().parse(schema)
    val childSchema = new GenericData.Record(avroSchema).getSchema.getField("data").schema
    
    val parentRecord = new GenericData.Record(avroSchema)
    val childRecords = new GenericData.Array[GenericData.Record](1024, childSchema)
    
    val childRecord = new GenericData.Record(childSchema.getElementType)
    
    childRecord.put("phone_number", "33333")
    childRecords.add(childRecord)
    
    parentRecord.put("schema", "schema_name")
    parentRecord.put("data", childRecords)
    parentRecord.put("flag", "I")
    
    println(parentRecord)
    

    产量:

    {"schema": "schema_name", "data": [{"phone_number": "33333"}], "flag": "I"}