代码之家 › 专栏 › 技术社区 › kkesley

从kinesis firehose解析json

amazon-kinesis-firehose go json

kkesley · 技术社区 · 7 年前

嗨,我试着用动势消防水龙带和S3。我试着读那些s3文件。我用GO来读。

下面是该文件的示例(请注意,原始输入相互附加,为了格式化,我用换行符将它们拆分):

{"ticker_symbol":"PLM","sector":"FINANCIAL","change":-0.16,"price":19.99}
{"ticker_symbol":"AZL","sector":"HEALTHCARE","change":-0.78,"price":16.51}
{"ticker_symbol":"IOP","sector":"TECHNOLOGY","change":-1.98,"price":121.88}
{"ticker_symbol":"VVY","sector":"HEALTHCARE","change":-0.56,"price":47.62}
{"ticker_symbol":"BFH","sector":"RETAIL","change":0.74,"price":16.61}
{"ticker_symbol":"WAS","sector":"RETAIL","change":-0.6,"price":16.72}

}{ 并再次附加它们。但这是相当黑客。

------更新------

目前我已经实现了替换所有具有 },{ 然后加上 [ 开始和结束时 ]

然而,我仍然在寻找替代品,因为这个解决方案将限制任何 }{ 在json对象的内容中

1 回复 | 直到 7 年前

Himanshu 7 年前

创建一个简单的结构来解组成批出现的json。因此,每批json都被解组到一个json对象中。然后创建一个结构切片,将解析后的json附加到切片中。这将把结果json全部附加到struct的切片中。

package main

import (
    "encoding/json"
    "fmt"
)

type Ticker struct {
    TickerSymbol string  `json:"ticker_symbol"`
    Sector       string  `json:"sector"`
    Change       float64 `json:"change"`
    Price        float64 `json:"price"`
}

var jsonBytes = []byte(`{"ticker_symbol":"PLM","sector":"FINANCIAL","change":-0.16,"price":19.99}`)

func main() {
    var singleResult Ticker
    var result []Ticker
    if err := json.Unmarshal(jsonBytes, &singleResult); err != nil {
        fmt.Println(err)
    }

    if len(result) == 0 {
        result = append(result, singleResult)
    }
    fmt.Printf("%+v", result)
}

编辑时间:

如果数据是成批出现的,其中包含互相附加的json对象,那么可以使用regex表达式来替换 } 具有 }, 然后修剪最右边 ,

package main

import (
    "fmt"
    "regexp"
    "strings"
)

type Ticker struct {
    TickerSymbol string  `json:"ticker_symbol"`
    Sector       string  `json:"sector"`
    Change       float64 `json:"change"`
    Price        float64 `json:"price"`
}

var str = `{"ticker_symbol":"PLM","sector":"FINANCIAL","change":-0.16,"price":19.99}
{"ticker_symbol":"AZL","sector":"HEALTHCARE","change":-0.78,"price":16.51}
{"ticker_symbol":"IOP","sector":"TECHNOLOGY","change":-1.98,"price":121.88}
{"ticker_symbol":"VVY","sector":"HEALTHCARE","change":-0.56,"price":47.62}
{"ticker_symbol":"BFH","sector":"RETAIL","change":0.74,"price":16.61}
{"ticker_symbol":"WAS","sector":"RETAIL","change":-0.6,"price":16.72}`

func main() {

    r := regexp.MustCompile("}")
    output := strings.TrimRight(r.ReplaceAllString(str, "},"), ",")
    output = fmt.Sprintf("[%s]", output)
    fmt.Println(output)
}

使用 r := regexp.MustCompile("}") }{ 这会妨碍替换字符串。那就换个吧 } 具有 },

我使用MustCompile的原因还有:

使用正则表达式创建常量时,可以使用 MustCompile编译的变体。一个简单的编译对你不起作用常量,因为它有2个返回值。

打开json解析的完整工作代码 Go playground