ELK Stack: Logstash shows that it's receiving log entries from Filebeat, but Elasticsearch is not creating my index

Question

I am new to the ELK stack and I wanted to try and test it out to see if I wanted to use it. I have elasticsearch, kibana, and logstash installed on one virtual machine and I have filebeat and nginx installed on another virtual machine.

I have a custom log format for my nginx access.log that looks like this:

<IP> - - [21/Dec/2023:00:46:10 +0000] "GET /favicon.ico HTTP/1.1" 404 134 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15" "-" "<IP>" sn="test.com" rt=0.000 ua="-" us="-" ut="-" ul="-" cs=-

#log format

log_format  main_ext  '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'"$host" sn="$server_name" '
'rt=$request_time '
'ua="$upstream_addr" us="$upstream_status" '
'ut="$upstream_response_time" ul="$upstream_response_length" '
'cs=$upstream_cache_status' ;

I have everything configured and have the Kibana dashboard up and running with data being sent to the dashboard. The only problem I am having is that the correct indexes are not showing up in elasticsearch or my kibana dashboard. The only index that is showing up is the default filebeat-* and I am not able to see my nginx-access-logs or nginx-error-logs indices.

Here is my logstash config file path /etc/logstash/conf.d/beats.conf and here is what it looks like:

input {
  beats {
    port => 5044
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGLINE}" }
    }
    date {
      match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }

  if [type] == "nginxaccess" {
    grok {
      match => { "message" => '%{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} %{NUMBER:bytes} "%{URI:referrer}" "%{DATA:agent}" "%{IPORHOST:x_forwarded_for}" sn="%{DATA:sn}" rt=%{NUMBER:request_time} ua="%{DATA:upstream_addr}" us="%{DATA:upstream_status}" ut="%{DATA:upstream_response_time}" ul="%{DATA:upstream_response_length}" cs=%{DATA:upstream_cache_status}' }
    }
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z", "ISO8601" ]
    }
  }

  if [type] == "nginxerror" {
    grok {
      match => { "message" => '%{TIMESTAMP_ISO8601:timestamp} \[%{WORD:log_level}\] %{NUMBER:pid}#%{NUMBER:tid}: %{GREEDYDATA:message}' }
    }
    date {
      match => [ "timestamp", "yyyy/MM/dd HH:mm:ss" ]
    }
  }
}



output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

And here is what my etc/filebeat/filebeat.yml file looks like:

filebeat.inputs:

#Each - is an input. Most options can be set at the input level, so
#you can use different inputs for various configurations.
#Below are the input-specific configurations.

#filestream is an input for collecting log messages from files.
- type: filestream
  #Unique ID among all inputs, an ID is required.
  id: my-filestream-id
  #Change to true to enable this input configuration.
  enabled: true
  #Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
  fields:
    type: syslog

- type: filestream
  id: nginx-access-logs
  enabled: true
  paths:
    - /var/log/nginx/access.log*
  fields:
    type: nginxaccess  # Set the log type to nginx_access
    beat: nginxaccess

- type: filestream
  id: nginx-error-logs
  enabled: true
  paths:
    - /var/log/nginx/error.log*
  fields:
    type: nginxerror  # Set the log type to nginx_error
    beat: nginxerror

filebeat.config.modules:
  #Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  #Set to true to enable config reloading
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

setup.kibana:

output.logstash:
  #The Logstash hosts
  hosts: ["<IP>:5044"]

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Now after configuring those files I restarted both logstash and filebeat and ran the commands:

`sudo filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=[":9200"]'

sudo filebeat setup -E output.logstash.enabled=false -E output.elasticsearch.hosts=[':9200'] -E setup.kibana.host=:5601`

Then I went and checked my indexes and my nginx-access-logs and nginx-error-logs were still not showing in my indices. The only ones that show are the default filebeat-*

curl -X GET "http://localhost:9200/_cat/indices?v" health status index uuid pri rep docs.count docs.deleted store.size pri.store.size dataset.size yellow open filebeat-2023.12.20 _nckH4WZSmuaI1umnjVR-w 1 1 91772 0 36.1mb 36.1mb 36.1mb yellow open filebeat-2023.12.14 yA8Sl4lXSYG_vB8d67Cqfg 1 1 47876 0 19.6mb 19.6mb 19.6mb yellow open filebeat-2023.12.15 OwUwZMdBR3myvZkuvUgF2A 1 1 75513 0 28.5mb 28.5mb 28.5mb yellow open .ds-filebeat-8.11.2-2023.12.08-000001 PPRThZq3RIK490NVmW605A 1 1 0 0 249b 249b 249b yellow open filebeat-2023.12.16 4EOUSNCKRlOzAy1zdih6hg 1 1 79795 0 29.4mb 29.4mb 29.4mb yellow open filebeat-2023.12.17 JL7TRkUgTzeGbT-M0bBD5g 1 1 64067 0 24.4mb 24.4mb 24.4mb yellow open filebeat-2023.12.10 m3aWcEayTnu3r_iTxd_5aA 1 1 77669 0 27.9mb 27.9mb 27.9mb yellow open filebeat-2023.12.21 mvbk8wNiQ9-rT9Vs-W-Vqg 1 1 62321 0 27.9mb 27.9mb 27.9mb yellow open filebeat-2023.12.11 bXq4al_xQ62eMjAnEKR5Xw 1 1 81750 0 29mb 29mb 29mb yellow open filebeat-2023.12.12 V2ojtGRTR4ixSGT_tgkhHg 1 1 70454 0 27mb 27mb 27mb yellow open filebeat-2023.12.13 eRuR2uf2QdqF00VagDnjpw 1 1 72317 0 27.4mb 27.4mb 27.4mb yellow open filebeat-2023.12.18 Q_IEBhszSOSK9305LsvXOg 1 1 82494 0 30.5mb 30.5mb 30.5mb yellow open filebeat-2023.12.19 KsJZ2um5Q8e7v2ckP9MlGA 1 1 77330 0 29.6mb 29.6mb 29.6mb yellow open filebeat-2023.12.08 C8ih6TUMRdm2AsSO5idwkw 1 1 13953 0 5.8mb 5.8mb 5.8mb yellow open filebeat-2023.12.09 2Taw_nROSCiBeYJFk-kyXA 1 1 58190 0 21.2mb 21.2mb 21.2mb

Can someone please help me figure out what I'm doing wrong or what is going on. I am lost at this point!