How do I simulate a bad connection to AWS S3 using toxiproxy?

Question

What I'm trying to accomplish:

• I have a ruby on rails app which uses carrierwave to store data using the fog-aws adapter
• I'm trying to simulate poor communication with AWS S3

What I've done:

I have a sample ruby script which tries to connect to AWS and enumerate objects:

require 'aws-sdk-s3'
params = {region: 'us-west-2', access_key_id: 'key', secret_access_key: 'secret'}
s3 = Aws::S3::Client.new(params)
puts s3.list_objects({bucket: 'bucketname', prefix: '', max_keys: 1})

This works - so the credentials are fine.

I now want to toxify this using toxiproxy (a chaos engineering tool that'll randomly break connections, etc)

• Added an http proxy to the params: {region: 'us-west-2', access_key_id: 'key', secret_access_key: 'secret', http_proxy: 'http://localhost:7890'}
• Created a toxiproxy definition using anything I could think of:

toxiproxy-cli create --listen localhost:7890  -u bucketname.s3.amazonaws.com:443 test-aws
toxiproxy-cli create --listen localhost:7890  -u s3-r-w.us-west-2.amazonaws.com:443 test-aws

I then execute the above mentioned script and all I see is: lib/ruby/2.6.0/net/protocol.rb:225:in 'rbuf_fill': end of file reached (Seahorse::Client::NetworkingError)

I can't figure out the magic parameters required to make everything connect.

So questions:

• Can what I'm trying to accomplish work using toxiproxy?
• If not, what is recommended?

Answer 1

I figured it out. It looks like toxiproxy by itself is not enough to intercept & forward the connection -- I needed to also use tinyproxy.

This is what I did:

1. Setup tinyproxy as an http proxy on port 8888
2. Setup toxiproxy on port 7890 to forward to port 8888
3. Configured the AWS client to connect to an http_proxy on port 7890 -- this allows toxiproxy to mess with the connection, but will allow tinyproxy to consume the HTTP CONNECT and tunnel to AWS.

Example code:

require 'aws-sdk-s3'
s3 = Aws::S3::Client.new(region: 'us-west-2', access_key_id: 'key', secret_access_key: 'secret', http_proxy: 'http://localhost:7890')
puts s3.list_objects({bucket: 'bucketname', prefix: '', max_keys: 1})

Example tinyproxy configuration:

Port 8888
Listen 127.0.0.1
Timeout 600
Allow 127.0.0.1

Example toxiproxy configuration:

toxiproxy-cli create --listen 127.0.0.1:7890 -u 127.0.0.1:8888 test-aws
toxiproxy-cli toxic add --type reset_peer -a timeout=25 test-aws