Can't connect to ElastiCache Redis

Question

I have a small Rails 7.0 application that uses Redis as a storage from which data is fetched to be processed later in the background. The application communicates with it through ActiveJob APIs which uses Sidekiq as the adapter.

Everything went great in my local environment. But the moment I deployed it to AWS, it stoped working.

When I get in the remote environment and run sidekiq, I get the following error:

2024-01-23 17:13:52.760152 I [678:5260] Sidekiq -- Booted Rails 7.0.8 application in staging environment
2024-01-23 17:13:52.760235 I [678:5260] Sidekiq -- Running in ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) +YJIT [x86_64-linux-musl]
2024-01-23 17:13:52.760246 I [678:5260] Sidekiq -- See LICENSE and the LGPL-3.0 for licensing details.
2024-01-23 17:13:52.760250 I [678:5260] Sidekiq -- Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org
2024-01-23 17:13:52.760275 I [678:5260] Sidekiq -- Sidekiq 7.2.1 connecting to Redis with options {:size=>10, :pool_name=>"internal", :url=>"redis://abc-def-ghi.serverless.apne1.cache.amazonaws.com:6379/0"}
Waited 1.0 seconds
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client/ruby_connection/buffered_io.rb:140:in `block in fill_buffer'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client/ruby_connection/buffered_io.rb:122:in `loop'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client/ruby_connection/buffered_io.rb:122:in `fill_buffer'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client/ruby_connection/buffered_io.rb:114:in `ensure_remaining'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client/ruby_connection/buffered_io.rb:85:in `getbyte'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client/ruby_connection/resp3.rb:113:in `parse'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client/ruby_connection/resp3.rb:50:in `load'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client/ruby_connection.rb:94:in `read'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client/connection_mixin.rb:51:in `block in call_pipelined'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client/connection_mixin.rb:49:in `times'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client/connection_mixin.rb:49:in `call_pipelined'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client.rb:749:in `block in connect'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client/middlewares.rb:16:in `call'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client.rb:748:in `connect'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client.rb:710:in `raw_connection'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client.rb:675:in `ensure_connected'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client.rb:255:in `call'
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client/decorator.rb:26:in `call'
/usr/local/bundle/gems/sidekiq-7.2.1/lib/sidekiq/config.rb:144:in `block in redis_info'
/usr/local/bundle/gems/sidekiq-7.2.1/lib/sidekiq/config.rb:163:in `block in redis'
/usr/local/bundle/gems/connection_pool-2.4.1/lib/connection_pool.rb:110:in `block (2 levels) in with'
/usr/local/bundle/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `handle_interrupt'
/usr/local/bundle/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `block in with'
/usr/local/bundle/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `handle_interrupt'
/usr/local/bundle/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `with'
/usr/local/bundle/gems/sidekiq-7.2.1/lib/sidekiq/config.rb:160:in `redis'
/usr/local/bundle/gems/sidekiq-7.2.1/lib/sidekiq/config.rb:143:in `redis_info'
/usr/local/bundle/gems/sidekiq-7.2.1/lib/sidekiq/cli.rb:75:in `run'
/usr/local/bundle/gems/sidekiq-7.2.1/bin/sidekiq:31:in `<top (required)>'
/usr/local/bundle/bin/sidekiq:25:in `load'
/usr/local/bundle/bin/sidekiq:25:in `<main>'

And when I launch Rails console and run a command to place a job, I get the following error:

irb(main):001:0> ExampleJob.perform_later(user_id: 1)
2024-01-23 17:17:15.180008 I [685:4720 redis_connection.rb:15] Sidekiq -- Sidekiq 7.2.1 connecting to Redis with options {:size=>10, :pool_name=>"internal", :url=>"redis://abc-def-ghi.serverless.apne1.cache.amazonaws.com:6379/0"}
2024-01-23 17:17:17.194664 I [685:4720 subscriber.rb:149] Rails -- Enqueued ExampleJob (Job ID: 804eca79-8e6a-4b99-8368-310eb28ac0f2) to Sidekiq(default) -- {:event_name=>"enqueue.active_job", :adapter=>"Sidekiq", :queue=>"default", :job_class=>"ExampleJob", :job_id=>"804eca79-8e6a-4b99-8368-310eb28ac0f2", :provider_job_id=>nil, :arguments=>"[\n  {\n    \"user_id\": 1\n  }\n]"}
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client/ruby_connection/buffered_io.rb:140:in `block in fill_buffer': Waited 1.0 seconds (RedisClient::CannotConnectError)
/usr/local/bundle/gems/redis-client-0.19.1/lib/redis_client/ruby_connection/buffered_io.rb:140:in `block in fill_buffer': Waited 1.0 seconds (RedisClient::ReadTimeoutError)

I am very confused. Can anybody tell what the culprit might be?

A few more things to mention would be:

• bundle exec sidekiq has been executed on the application side
• The Redis is ElastiCache "Serverless"
• The Redis belongs to the same VPC as the application
• The Redis' security group at this moment is very loose
  • Inbound -- HTTP(port 80), HTTPS (port 443), TCP (port 6379), all of which have 0.0.0.0/0 as a source
  • Outbound-- All protocol, all ports with 0.0.0.0/0
• The Redis has no access control by user group