logstash Getting started with logstash Outputting to ElasticSearch: multiple indices and mappings

Example

Sometimes, you need to output to more than one index in ElasticSearch, or have a custom mapping you want to apply to new indices as they roll in.

There are two ways to apply a custom mapping. One way, is to upload an ElasticSearch template. See the ElasticSearch documentation for that. The other way is to specify a mapping in the elasticsearch {} output itself. That is what is shown here.

output {
  if [type] == 'metrics' {
    # The 'metrics' index rotates weekly.
    # The 'metrics-mapping.json' file defines the custom mappings.
    elasticsearch {
      hosts              => [ 'localhost' ]
      index              => "metrics-%{xxxx.ww}"
      manage_template    => true
      template           => "/etc/logstash/metrics-mapping.json"
      template_overwrite => true
    }
  }
}

This will output metrics events to metrics- indexes on ElasticSearch, which will rotate weekly using the ISO week. The template used for new indexes is defined as part of this configuration. Defining a template has the advantage of forcing the types of fields to a uniform type. This is useful in larger configurations where multiple types may attempt to define a field as a slightly different data-type.

This method is useful in staging and QA environments, as the ElasticSearch templates are defined by the LogStash code and don't have to be configured separately as part of the ElasticSearch cluster setup.