代码之家 › 专栏 › 技术社区 › Bogdan Stoica

logstash从数组创建json

logstash elasticsearch json

Bogdan Stoica · 技术社区 · 6 年前

当前日志类型如下:

{"event": "Task succeeded", "id": "13e495e3-1a0a-4612-bd11-34293ada3133", "level": "info", "logger": "celery.worker", "name": "kraken.tasks.RPCTask", "rpc_endpoint": "routing", "rpc_method": "route_sms", "rpc_params": [32425657567, 606104377393580], "runtime": 0.48229960491880774, "service": "core", "timestamp": "2018-10-24T05:30:21.557987Z"}
{"event": "Received task", "id": "13e495e3-1a0a-4612-bd11-34293ada3133", "level": "info", "logger": "celery.worker", "name": "kraken.tasks.RPCTask", "rpc_endpoint": "routing", "rpc_method": "route_sms", "rpc_params": [657547567567, 606104377393580], "timestamp": "2018-10-24T05:30:21.074257Z"}
{"event": "Task accepted", "id": "4a0e3e50-3876-4850-a637-889ea36dedd3", "level": "info", "logger": "celery.worker", "name": "kraken.tasks.SilentRPCTask", "pid": 140530763197808, "rpc_endpoint": "analytics.fact", "rpc_method": "add", "rpc_params": {"city_id": null, "correlation_id": "08ee597c-8ee4-48f2-87b7-89ef4c844069", "country_id": null, "dst_is_platform": true, "dst_ln": 56546456, "dst_mid": null, "dst_operator_id": null, "dst_package_id": null, "dst_user_id": null, "duration": null, "ignore_errors": true, "medium": null, "platform_event": "sms-in", "service": "wallet", "service_event": null, "service_path": null, "sms_in_size": 1, "sms_out_size": null, "src_is_platform": false, "src_ln": 4353434543, "src_mid": null, "src_operator_id": 35345, "src_package_id": null, "src_user_id": 30639, "ts_end": null, "ts_start": null}, "timestamp": "2018-10-24T05:30:18.675206Z"}

如您所见,它是json格式的,除了一些条目。这里的问题是,我有rpc_params,大多数时候是json格式的,但是当rpc_params看起来像这样时,会有一些条目: “rpc_params”:[[]] 或 “rpc_params”:[] 或 “rpc_params”:[[12312312312312312312123123123]]

我应该在logstash中使用什么过滤器?日志与filebeat一起发送到logstash,然后发送到elasticsearch到特定索引。我当前的logstash过滤器如下所示:

filter {
    if "celleryapp" in [tags] {

        mutate {
            gsub => ["rpc_params", "[\[\]]", ""]
            gsub => ["rpc_params", "[\[\[\]\]]", ""]
        }

        mutate {
            split => { "rpc_params" => "," }
        }

        json {
                source => "message"
        }
    }
}

不管有没有变异过滤器,我总是在logstash中得到这个错误:

[2018-10-24T05:34:57,878][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"celleryapp-2018.10.24", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x47ab2458>], :response=>{"index"=>{"_index"=>"celleryapp-2018.10.24", "_type"=>"doc", "_id"=>"XfuRpGYBiD1kYJIQm9JM", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [rpc_params] tried to parse field [null] as object, but found a concrete value"}}}}
[2018-10-24T05:34:57,879][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"celleryapp-2018.10.24", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x44883a69>], :response=>{"index"=>{"_index"=>"celleryapp-2018.10.24", "_type"=>"doc", "_id"=>"XvuRpGYBiD1kYJIQm9JM", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [rpc_params] tried to parse field [null] as object, but found a concrete value"}}}}

问题是完全一样 rpc_参数 是一个杰森格式,然后是阵列是的。

使用mutate过滤器,我试图删除[][[]]。我可能需要一个过滤器,只有在 rpc_params不是json格式

0 回复 | 直到 6 年前