There are graphical tools to test and use the Dell ECS S3 interface, but everyone wants a CLI right? The below should also work for AWS S3 (with some modification) but this documentation is intended for specific use with the Dell ECS S3 endpoint to verify behaviour and perform some performance testing. The below assumes you already have an Object User created on the Dell ECS along with an S3 Bucket in a Namespace.

Say for example your Dell ECS endpoint is called “ecs1.domain.com”, and your namespace (where your bucket lives) is called “testing”, then you’d have a host_base of: “testing.ecs1.domain.com”, you’d thjen need to specify the bucket_location in this case let’s say its called “testbucket”.

Beyond that all you then need is the access_key which is actually the Object User in Dell ECS parlance, let’s call that “test-user” in this example. Then finally the secret_key which you set when you create the Object User in this example: “secretkeyhere1234”

Install s3cmd

Install s3cmd using the following command:

sudo apt install s3cmd

Configure s3cmd.cfg

You can configure using the “s3cmd –configure” command, but in our case we just need a simple configuration file so create on as follows in your home directory, s3cmd will by default use the path: ~/.s3cfg unless you specify otherwise.

touch .s3cfg
chmod 600 .s3cfg

Edit the file and add the following details, obviously replace the relevant properties as required.

.s3cfg

[default]
# Object Storage Region
host_base = testing.ecs1.domain.com
host_bucket = %(bucket).testing.ecs1.domain.com
bucket_location = testbucket
use_https = True
# Login credentials
access_key = test-user
secret_key = secretkeyhere1234
check_ssl_certificate = False

Tip: You can also use environment variables for the access_key and secret_key if you don’t want these hard coded into a configuration file.

List Contents of S3 Bucket

To query the contents of an S3 Bucket you can run the below, if you wish to see within a sub-directory, just append the sub-directory name onto the end.

s3cmd ls s3://testbucket

You’ll see the contents of the directory for example, excuse all the ISO images and rubbish!

You can also specify on any s3cmd command a specific .s3cmd configuration file. By default it will use .s3cfg (if present). If you need a number of different S3 configuration files, you can create another called .s3cfg-number2 with:

touch .s3cfg
chmod 600 .s3cfg

Add the relevant configuration file contents, then to use it use the “-c” argument.

s3cmd ls -c ~/.s3cfg-number2 s3://testbucket2

The above “-c” switch can be used on s3cmd for List, Get, Put etc. for brevity only the “default” is shown in the rest of this document.

Put a File into S3 Bucket

Put a file called myfile.txt into the bucket as follows:

s3cmd put myfile.txt s3://testbucket

Get a File from S3 Bucket

Get a file called myfile.txt from the bucket as follows:

s3cmd get s3://testbucket/myfile.txt

Simple Performance Test

To determine the performance you can use some simple tests for “get” and “put” requests.

The speed you’ll get will be dependent on the speed of the network, the Dell ECS infrastructure and also your network speed from your testing machine. In this example we have 21 Dell ECS nodes spread over 3 sites with a pair of Kemp ECS Connection Managers at each site, with 10Gbit edge port connectivity and a 40Gbit backhaul. Test test machine is also connected at 10Gbit and is using flash storage for the storage of the file for upload or download from the Dell ECS.

Firstly we’ll generate a 10GB file full of random stuff to ensure nothing can de-dupe or compress (very well) during transit.

dd if=/dev/urandom of=random10GB.bin bs=64M count=160 iflag=fullblock

Put Performance Test

The following can give you an idea of the performance of your Dell ECS environment during a single threaded upload (put), notice the “time” command on the front which will monitor the duration of the upload, knowing the time, the size of the file allows you to calculate the upload speed.

time s3cmd put random10GB.bin s3://testbucket

And we get a result such as:

upload: 'random10GB.bin' -> 's3://testbucket/random10GB.bin'  [part 680 of 683, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    39.89 MB/s  done
upload: 'random10GB.bin' -> 's3://testbucket/random10GB.bin'  [part 681 of 683, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    42.17 MB/s  done
upload: 'random10GB.bin' -> 's3://testbucket/random10GB.bin'  [part 682 of 683, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    38.04 MB/s  done
upload: 'random10GB.bin' -> 's3://testbucket/random10GB.bin'  [part 683 of 683, 10MB] [1 of 1]
 10485760 of 10485760   100% in    0s    34.09 MB/s  done
 
real    5m2.594s
user    1m22.670s
sys     0m22.797s

To calculate:

5 Minutes 2.5 Seconds = 302.5 Seconds

10240MB / 302.5 = 33.8MB/s (Megabytes per second) i.e. 270Mbit/s (Megabits per second) upload speed

Get Performance Test

The following can give you an idea of the performance of your Dell ECS environment during a single threaded download (get). You can wrap this in a “time” if you wish, but the information provided by the s3cmd may be enough for you.

$ s3cmd get s3://testbucket/random10GB.bin

You can get a result like:

download: 's3://testbucket/random10GB.bin' -> './random10GB.bin'  [1 of 1]
 10737418240 of 10737418240   100% in   94s   108.85 MB/s  done

Thus giving:

94 Seconds

10240MB / 94 = 108.9MB/s (Megabytes per second) i.e. 871Mbit/s (Megabits per second) download speed.

Additional Information

• https://support.morrodata.com/support/solutions/articles/14000133197-how-can-i-test-aws-s3-file-upload-speed-
• https://docs.safespring.com/storage/s3cmd/
• https://www.scaleway.com/en/docs/tutorials/s3cmd/
• https://www.dell.com/support/kbdoc/en-uk/000021492/ecs-how-to-access-buckets-when-using-the-aws-cli

Image Attribution