代码之家 › 专栏 › 技术社区 › Mariya James

使用tdagent将aws elasticssearch与fluentd连接

td-agent fluentd elasticsearch

0

Mariya James · 技术社区 · 5 年前

我已经使用tdagent成功地在本地机器上连接了elasticsearch,但是在staging环境中我需要连接aws elasticsearch,如果我使用ruby安装了fluentd,那么就有一个插件

gem 'fluent-plugin-aws-elasticsearch-service'

但是当使用fluentd和tdagent时,我们如何配置aws elasticsearch?

<match catalog>
  @type elasticsearch
  host https://elastic_devel_aws.com
  # port 9200
  logstash_format true 
  include_timestamp true 
  index_name _logs_test
  flush_interval 10s
  reconnect_on_error true
  reload_on_failure true
  reload_connections false
  request_timeout 120s
  <buffer>
    @type file
   flush_interval 10s
   retry_type periodic
   retry_forever true
   retry_wait 10s
   chunk_limit_size 16Mb
   queue_limit_length 4096
   total_limit_size 60Gb
   path /var/lib/td-agent/buffers/output_elasticsearch-1
 </buffer>
</match>

当你试着用这样的东西

  Could not communicate to Elasticsearch, resetting connection and trying again. getaddrinfo: Name or service not known (SocketError)

0 回复 | 直到 5 年前

1

Al-waleed Shihadeh 5 年前

要使用td-agent安装gem,您需要执行以下命令

/usr/sbin/td-agent-gem install fluent-plugin-aws-elasticsearch-service'

<match *.**>
  @type elasticsearch

  host "#{ENV['ELASTICSEARCH_HOST']}"
  port "#{ENV['ELASTICSEARCH_PORT']}"
  scheme "#{ENV['ELASTICSEARCH_SCHEME'] || 'http'}"
  type_name log

  include_tag_key true
  tag_key @logtag

  logstash_format true
  logstash_prefix fos
  logstash_dateformat %Y%m%d

  reconnect_on_error true

  <buffer>
    @type file
    path /fluentd/log/fos/elastic-buffer
    flush_thread_count 8
    flush_interval 1s
    chunk_limit_size 32M
    queue_limit_length 4
    flush_mode interval
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>

您还需要确保staging环境允许访问弹性搜索实例,您可以通过执行以下命令进行检查

telnet elastic_devel_aws.com 443