ELK FileBeat - pipeline not ingesting

Question

I have multiple indices in the same filebeat.yml the pipeline is not being ingested eventhough pipeline test does not throw any error and is able to parse the document id successfully

#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: aws-cloudwatch
  enabled: true
  access_key_id: "xxxxxxxxxxxxx"
  secret_access_key: "xxxxxxxxxxxx"
  region_name: "xxxxxxxxx"
  log_group_arn: "arn:aws:logs:xxxxxxxx:apache_logs"
  log_group_name: "xxxxxxxxx/apache"
  fields:
    name: "prod-cloudwatch_apache_logs"
  ignore_older: 720h
  start_position: end
- type: aws-cloudwatch
  enabled: true
  access_key_id: "xxxxxxxxxxxxx"
  secret_access_key: "xxxxxxxxxxxx"
  region_name: "xxxxxxxxx"
  log_group_arn: "arn:aws:logs:xxxxxxxx:apache_logs"
  log_group_name: "xxxxxxxxx/apache"
  fields:
    name: "uat-cloudwatch_apache_logs"
  ignore_older: 720h
  start_position: end

output.elasticsearch:
  hosts: ["https://xxxxxxx.aws.elastic-cloud.com:xxxxxx"]
  protocol: "https"
  username: "${ESUSER}"
  password: "${ESPASS}"
  output.elasticsearch.allow_older_versions: true
  indices:
   - index: "cloudwatch_uat_apache_logs-%{+yyyy.MM}"
     when.contains:
      fields.name: "uat-cloudwatch_apache_logs"
   - index: "cloudwatch_prod_apache_logs%{+yyyy.MM}"
     when.contains:
      fields.name: "prod-cloudwatch_apache_logs"
  pipelines:
    - pipeline: "apache_pipeline"
      when:
       has_fields: ['name.prod-cloudwatch_apache_logs']

My pipeline does not run ( fields are not visible on kibana )

Fix configuration issue

Answer 1

There 2 small problems in your configuration.

output.elasticsearch.allow_older_versions: true

should be just

allow_older_versions: true

because it is already under the output.elasticsearch block.

The second issue (probably not the main problem) is a small one. There is a missing dash.

  - index: "cloudwatch_prod_apache_logs%{+yyyy.MM}"

  - index: "cloudwatch_prod_apache_logs-%{+yyyy.MM}"