代码之家 › 专栏 › 技术社区 › Muhammad Idrees

使用logstash将csv文件导入elasticsearch时出错

logstash elasticsearch

Muhammad Idrees · 技术社区 · 7 年前

我想使用logstash将一个csv文件导入Logsearch。我的代码如下

    input {
file {
path =>"C:\Users\welcome\Dropbox\IT_Department\BigDataProjects\StudentWithdraElastic\student_withdraw.csv"
start_position => "beginning"
sincedb_path => "j:\null"
}
}
filter {
csv {
separator => ","
columns => ["DEPT_NAME" ,"CERT_NAME",   "SPEC_NAME" ,"STUDENT_NO",  "STUD_NAME",    "GENDER",   "ADVISORS_NAME",    "ACADEMIC_YEAR",    "REQUEST_NO",
        "withdraw_reason_category", "STATUS",   "WITHDRAW_DATE",    "LECTURER", "COURSE_NO",    "COURSE_NAME",  "SECTION_NO",   "TOTAL_REG"]
    mutate {convert => ["SECTION_NO", "integer"] }
    mutate {convert => ["TOTAL_REG", "integer"] }

}
output { 
 elasticsearch {
  hosts => "https://hadoop.hct.org"
  index => "withDrawIndex"
  document_type => "studentWithdrawDocument"
  }
  stdout {}
  }

我用下面的命令运行它

C:\Elastic\logstash-6.2.2>bin\logstash -f C:\Users\welcome\Dropbox\IT_Department\BigDataProjects\StudentWithdraElastic\logstash_withdraw.config

它给出以下错误

Sending Logstash's logs to C:/Elastic/logstash-6.2.2/logs which is now configured via log4j2.properties
[2018-03-29T09:26:09,570][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"C:/Elastic/logstash-6.2.2/modules/fb_apache/configuration"}
[2018-03-29T09:26:09,595][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"C:/Elastic/logstash-6.2.2/modules/netflow/configuration"}
[2018-03-29T09:26:09,964][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-03-29T09:26:10,565][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.2"}
[2018-03-29T09:26:11,027][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-03-29T09:26:11,400][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 13, column 12 (byte 486) after filter {\ncsv {\nseparator => \",\"\ncolumns => [\"DEPT_NAME\"\t,\"CERT_NAME\",\t\"SPEC_NAME\"\t,\"STUDENT_NO\",\t\"STUD_NAME\",\t\"GENDER\",\t\"ADVISORS_NAME\",\t\"ACADEMIC_YEAR\",\t\"REQUEST_NO\",\n\t\t\"withdraw_reason_category\",\t\"STATUS\",\t\"WITHDRAW_DATE\",\t\"LECTURER\",\t\"COURSE_NO\",\t\"COURSE_NAME\",\t\"SECTION_NO\",\t\"TOTAL_REG\"]\n    mutate ", :backtrace=>["C:/Elastic/logstash-6.2.2/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "C:/Elastic/logstash-6.2.2/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "C:/Elastic/logstash-6.2.2/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "C:/Elastic/logstash-6.2.2/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "C:/Elastic/logstash-6.2.2/logstash-core/lib/logstash/pipeline.rb:51:in `initialize'", "C:/Elastic/logstash-6.2.2/logstash-core/lib/logstash/pipeline.rb:169:in `initialize'", "C:/Elastic/logstash-6.2.2/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "C:/Elastic/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'", "C:/Elastic/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "C:/Elastic/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:312:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in `each'", "C:/Elastic/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:299:in `converge_state'", "C:/Elastic/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "C:/Elastic/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "C:/Elastic/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "C:/Elastic/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:90:in `execute'", "C:/Elastic/logstash-6.2.2/logstash-core/lib/logstash/runner.rb:348:in `block in execute'", "C:/Elastic/logstash-6.2.2/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

我寻找解决方案,但没有找到任何有帮助的。

1 回复 | 直到 7 年前

Val 7 年前

您的结尾缺少一个右大括号 csv 过滤器:

filter {
  csv {
    separator => ","
    columns => ["DEPT_NAME" ,"CERT_NAME",   "SPEC_NAME" ,"STUDENT_NO",  "STUD_NAME",    "GENDER",   "ADVISORS_NAME",    "ACADEMIC_YEAR",    "REQUEST_NO", "withdraw_reason_category", "STATUS",   "WITHDRAW_DATE",    "LECTURER", "COURSE_NO",    "COURSE_NAME",  "SECTION_NO",   "TOTAL_REG"]
  }                                                <-- this is missing
  mutate {convert => ["SECTION_NO", "integer"] }
  mutate {convert => ["TOTAL_REG", "integer"] }
}