BETA - Using Gateway MT with the AWS CLI

On January 25th, 2021 we deployed a beta feature on the Europe-West-1 Satellite. This feature is called Gateway MT; it's a Tardigrade hosted S3 gateway. The biggest difference between this new gateway and our existing gateway is that the new gateway is multi-tenant, and Storj Labs will be hosting a few of them. This means you’ll be able to integrate with the Storj network via HTTP, and you won’t have to run anything extra on your end.

One thing to keep in mind is that the Gateway MT is still in BETA, so DO NOT run anything mission-critical on it.

Get started using the Gateway MT on Europe West 1:

  1. Navigate to the Access Grants page and then click on 'Create Access Grant'. A modal window will pop up and you can enter a name for this access grant.

Naming an access grant

2. Assign the permissions you want this access grant to have

Permissions

3. Enter the Encryption Passphrase you used for your other access grants or if this is the first access grant you create we strongly encourage you to use a mnemonic phase as your encryption passphase. (The GUI can generate one on the client-side for you)

4. Click on the gateway Credentials (beta) drop-down menu and then click on the 'Generate Gateway Credentials' button.

When you click on this button you are opting into server-side encryption.

5. Copy your Access Key, Secret Key, and Endpoint to a safe location.

To continue make sure you have the AWS CLI installed on your machine

6. Configure your AWS CLI with the gateway MT credentials from the previous step by running the AWS configure command in your terminal

~ % aws configure
AWS Access Key ID [****************e53q]: <<yourAccessKey>>
AWS Secret Access Key [****************bbxq]: <<yourSecretKey>>
Default region name [us-east-1]:
Default output format [None]:
~ %

7. Set the multipart threshold to 1 TB. This step is nessessary because the Gateway MT does not currently support multipart upload. The AWS default is 5 MB so to avoid errors you need to set the threshold to an arbitrarily high value.

~ % aws configure set default.s3.multipart_threshold 1TB

8. Run an AWS command such as create bucket! When running commands using the gateway and AWS comobination you must specify the endpoint you are hitting with the '--endpoint-url flag'. The AWS CLI doesn't support setting the endpoint in the config and it needs to be specified via a flag.

~ % aws --endpoint-url "https://gateway.tardigradeshare.io" s3 mb s3://demo

If you use an AWS CLI v1 then you actually can install a plugin to allow to specify the endpoint:

Install awscli-plugin-endpoint:

~ % pip3 install awscli-plugin-endpoint

Configure plugin in your ~/.aws/config file:

~ % aws configure set plugins.endpoint awscli_plugin_endpoint

..and configure the default profile:

~ % aws configure set default.s3.endpoint_url https://gateway.tardigradeshare.io
~ % aws configure set default.s3.multipart_threshold 1TB

The resulting file would looks like:

[plugins]
endpoint = awscli_plugin_endpoint
[default]
s3 =
multipart_threshold = 1TB
endpoint_url = https://gateway.tardigradeshare.io

Now you can use your AWS CLI without specifying of the endpoint:

~ % aws s3 ls
2021-01-08 19:41:13 demo

To configure s3api endpoint you can use this command:

~ % aws configure set default.s3api.endpoint_url https://gateway.tardigradeshare.io

You also can use a different profile for Storj:

~ % aws configure set profile.storj.s3.endpoint_url https://gateway.tardigradeshare.io
~ % aws configure set profile.storj.s3.multipart_threshold 1TB

To use AWS CLI with a separate profile storj:

~ % aws s3 --profile storj ls
2021-01-08 19:41:13 demo

Installing pluginawscli-plugin-endpoint will also install the AWS CLI v1!