{"id":4018,"date":"2023-12-20T16:11:31","date_gmt":"2023-12-20T16:11:31","guid":{"rendered":"https:\/\/geekmungus.co.uk\/?p=4018"},"modified":"2023-12-20T16:12:52","modified_gmt":"2023-12-20T16:12:52","slug":"testing-s3-access-and-performance-with-s3cmd-for-dell-ecs","status":"publish","type":"post","link":"https:\/\/geekmungus.co.uk\/?p=4018","title":{"rendered":"Testing S3 Access (and Performance) with s3cmd (for Dell ECS)"},"content":{"rendered":"\n

There are graphical tools to test and use the Dell ECS S3 interface, but everyone wants a CLI right? The below should also work for AWS S3 (with some modification) but this documentation is intended for specific use with the Dell ECS S3 endpoint to verify behaviour and perform some performance testing. The below assumes you already have an Object User created on the Dell ECS along with an S3 Bucket in a Namespace.<\/p>\n\n\n\n

Say for example your Dell ECS endpoint is called “ecs1.domain.com”<\/strong>, and your namespace <\/em>(where your bucket lives) is called “testing”<\/strong>, then you’d have a host_base<\/em> of: “testing.ecs1.domain.com”<\/strong>, you’d thjen need to specify the bucket_location<\/em> in this case let’s say its called “testbucket”<\/strong>.<\/p>\n\n\n\n

Beyond that all you then need is the access_key<\/em> which is actually the Object User <\/em>in Dell ECS parlance, let’s call that “test-user”<\/strong> in this example. Then finally the secret_key<\/em> which you set when you create the Object User in this example: “secretkeyhere1234”<\/strong><\/p>\n\n\n\n

Install s3cmd<\/h1>\n\n\n\n

Install s3cmd using the following command:<\/p>\n\n\n\n

sudo apt install s3cmd<\/code><\/code><\/pre>\n\n\n\n
Configure s3cmd.cfg<\/h1>\n\n\n\n
You can configure using the “s3cmd –configure” command, but in our case we just need a simple configuration file so create on as follows in your home directory, s3cmd will by default use the path: ~\/.s3cfg<\/strong> unless you specify otherwise.<\/p>\n\n\n\n
touch .s3cfg\r\nchmod 600 .s3cfg<\/code><\/pre>\n\n\n\n
Edit the file and add the following details, obviously replace the relevant properties as required.<\/p>\n\n\n\n
.s3cfg<\/strong><\/p>\n\n\n\n
[default]\r\n# Object Storage Region\r\nhost_base = testing.ecs1.domain.com\r\nhost_bucket = %(bucket).testing.ecs1.domain.com\r\nbucket_location = testbucket\r\nuse_https = True\r\n# Login credentials\r\naccess_key = test-user\r\nsecret_key = secretkeyhere1234\r\ncheck_ssl_certificate = False<\/code><\/pre>\n\n\n\n
Tip: You can also use environment variables for the access_key and secret_key if you don’t want these hard coded into a configuration file.<\/p>\n\n\n\n
List Contents of S3 Bucket<\/h1>\n\n\n\n
To query the contents of an S3 Bucket you can run the below, if you wish to see within a sub-directory, just append the sub-directory name onto the end.<\/p>\n\n\n\n
s3cmd ls s3:\/\/testbucket<\/code><\/pre>\n\n\n\n
You’ll see the contents of the directory for example, excuse all the ISO images and rubbish!<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
You can also specify on any s3cmd command a specific .s3cmd configuration file. By default it will use .s3cfg (if present). If you need a number of different S3 configuration files, you can create another called .s3cfg-number2 with:<\/p>\n\n\n\n
touch .s3cfg\r\nchmod 600 .s3cfg<\/code><\/pre>\n\n\n\n
Add the relevant configuration file contents, then to use it use the “-c” argument.<\/p>\n\n\n\n
s3cmd ls -c ~\/.s3cfg-number2 s3:\/\/testbucket2<\/code><\/pre>\n\n\n\n
The above “-c” switch can be used on s3cmd for List, Get, Put etc. for brevity only the “default” is shown in the rest of this document.<\/p>\n\n\n\n
Put a File into S3 Bucket<\/h1>\n\n\n\n
Put a file called\u00a0myfile.txt<\/strong>\u00a0into the bucket as follows:<\/p>\n\n\n\n
s3cmd put myfile.txt s3:\/\/testbucket<\/code><\/pre>\n\n\n\n
Get a File from S3 Bucket<\/h1>\n\n\n\n
Get a file called\u00a0myfile.txt<\/strong>\u00a0from the bucket as follows:<\/p>\n\n\n\n
s3cmd get s3:\/\/testbucket\/myfile.txt<\/code><\/pre>\n\n\n\n
Simple Performance Test<\/h1>\n\n\n\n
To determine the performance you can use some simple tests for “get” and “put” requests.<\/p>\n\n\n\n
The speed you’ll get will be dependent on the speed of the network, the Dell ECS infrastructure and also your network speed from your testing machine. In this example we have 21 Dell ECS nodes spread over 3 sites with a pair of Kemp ECS Connection Managers at each site, with 10Gbit edge port connectivity and a 40Gbit backhaul. Test test machine is also connected at 10Gbit and is using flash storage for the storage of the file for upload or download from the Dell ECS.<\/p>\n\n\n\n
Firstly we’ll generate a 10GB file full of random stuff to ensure nothing can de-dupe or compress (very well) during transit.<\/p>\n\n\n\n
dd if=\/dev\/urandom of=random10GB.bin bs=64M count=160 iflag=fullblock<\/code><\/pre>\n\n\n\n
Put Performance Test<\/h2>\n\n\n\n
The following can give you an idea of the performance of your Dell ECS environment during a single threaded upload (put), notice the “time” command on the front which will monitor the duration of the upload, knowing the time, the size of the file allows you to calculate the upload speed.<\/p>\n\n\n\n
time s3cmd put random10GB.bin s3:\/\/testbucket<\/code><\/pre>\n\n\n\n
And we get a result such as:<\/p>\n\n\n\n
upload: 'random10GB.bin' -> 's3:\/\/testbucket\/random10GB.bin'  [part 680 of 683, 15MB] [1 of 1]\r\n 15728640 of 15728640   100% in    0s    39.89 MB\/s  done\r\nupload: 'random10GB.bin' -> 's3:\/\/testbucket\/random10GB.bin'  [part 681 of 683, 15MB] [1 of 1]\r\n 15728640 of 15728640   100% in    0s    42.17 MB\/s  done\r\nupload: 'random10GB.bin' -> 's3:\/\/testbucket\/random10GB.bin'  [part 682 of 683, 15MB] [1 of 1]\r\n 15728640 of 15728640   100% in    0s    38.04 MB\/s  done\r\nupload: 'random10GB.bin' -> 's3:\/\/testbucket\/random10GB.bin'  [part 683 of 683, 10MB] [1 of 1]\r\n 10485760 of 10485760   100% in    0s    34.09 MB\/s  done\r\n \r\nreal    5m2.594s\r\nuser    1m22.670s\r\nsys     0m22.797s<\/code><\/pre>\n\n\n\n
To calculate:<\/p>\n\n\n\n
5 Minutes 2.5 Seconds = 302.5 Seconds\r\n\r\n10240MB \/ 302.5 = 33.8MB\/s (Megabytes per second) i.e. 270Mbit\/s (Megabits per second) upload speed<\/code><\/pre>\n\n\n\n
Get Performance Test<\/h2>\n\n\n\n
The following can give you an idea of the performance of your Dell ECS environment during a single threaded download (get). You can wrap this in a “time” if you wish, but the information provided by the s3cmd may be enough for you.<\/p>\n\n\n\n
$ s3cmd get s3:\/\/testbucket\/random10GB.bin<\/code><\/pre>\n\n\n\n
You can get a result like:<\/p>\n\n\n\n
download: 's3:\/\/testbucket\/random10GB.bin' -> '.\/random10GB.bin'  [1 of 1]\r\n 10737418240 of 10737418240   100% in   94s   108.85 MB\/s  done<\/code><\/pre>\n\n\n\n
Thus giving:<\/p>\n\n\n\n
94 Seconds\r\n\r\n10240MB \/ 94 = 108.9MB\/s (Megabytes per second) i.e. 871Mbit\/s (Megabits per second) download speed.<\/code><\/pre>\n\n\n\n
Additional Information<\/h1>\n\n\n\n