We had a need to upload a large volume of data to AWS S3, obviously we want this to upload as fast as possible, we choose to use the AWS S3 Sync tool; however to get the most performance out of it, you need to perform some tweaks to the configuration.
Although you can make changes within environment variables, the way we did it was to change the configuration with our ~/.aws/config file instead. These changes are immediate for the next time you run the command, i.e. you don’t need to reauthenticate your session for them to take effect.
The full list of configuration options can be found in: https://docs.aws.amazon.com/cli/latest/topic/s3-config.html to apply them edit the relevant “profile” for the relevant AWS Account within your ~/.aws/config file, for example:
[profile s3bucketaccount]
sso_session = mysso
sso_account_id = 1234567891
sso_role_name = AWSAdministratorAccess
region = us-west-2
s3 =
max_concurrent_requests = 50You’ll notice the last two lines s3 = and the max_concurrent_requests, you can add more options here, as per the link above to further tweak the settings. You can try different values and observe the result in any improvements in performance.
We found that max_concurrent_requests made a fair bit of difference to upload speed, in our test of 10GB of data to be uploaded, the default setting of 10 seconds took 49 seconds to upload, while at setting of 50 reduced this to only about 12 seconds.
