BETA - Using Gateway MT with the AWS CLI
On January 25th, 2021 we deployed a beta feature on the Europe-West-1 Satellite. This feature is called Gateway MT; it's a Tardigrade hosted S3 gateway. The biggest difference between this new gateway and our existing gateway is that the new gateway is multi-tenant, and Storj Labs will be hosting a few of them. This means you’ll be able to integrate with the Storj network via HTTP, and you won’t have to run anything extra on your end.
One thing to keep in mind is that the Gateway MT is still in BETA, so DO NOT run anything mission-critical on it.
Get started using the Gateway MT on Europe West 1:
Navigate to the Access Grants page and then click on 'Create Access Grant'. A modal window will pop up and you can enter a name for this access grant.
2. Assign the permissions you want this access grant to have
3. Enter the Encryption Passphrase you used for your other access grants or if this is the first access grant you create we strongly encourage you to use a mnemonic phase as your encryption passphase. (The GUI can generate one on the client-side for you)
4. Click on the gateway Credentials (beta) drop-down menu and then click on the 'Generate Gateway Credentials' button.
When you click on this button you are opting into server-side encryption.
5. Copy your Access Key, Secret Key, and Endpoint to a safe location.
To continue make sure you have the AWS CLI installed on your machine
6. Configure your AWS CLI with the gateway MT credentials from the previous step by running the AWS configure command in your terminal
~ % aws configureAWS Access Key ID [****************e53q]: <<yourAccessKey>>AWS Secret Access Key [****************bbxq]: <<yourSecretKey>>Default region name [us-east-1]:Default output format [None]:~ %
7. Set the multipart threshold to 1 TB. This step is nessessary because the Gateway MT does not currently support multipart upload. The AWS default is 5 MB so to avoid errors you need to set the threshold to an arbitrarily high value.
~ % aws configure set default.s3.multipart_threshold 1TB
8. Run an AWS command such as create bucket! When running commands using the gateway and AWS comobination you must specify the endpoint you are hitting with the '--endpoint-url flag'. The AWS CLI doesn't support setting the endpoint in the config and it needs to be specified via a flag.
~ % aws --endpoint-url "https://gateway.tardigradeshare.io" s3 mb s3://demo
If you use an AWS CLI v1 then you actually can install a plugin to allow to specify the endpoint:
Install awscli-plugin-endpoint:
~ % pip3 install awscli-plugin-endpoint
Configure plugin in your ~/.aws/config file:
~ % aws configure set plugins.endpoint awscli_plugin_endpoint
..and configure the default profile:
~ % aws configure set default.s3.endpoint_url https://gateway.tardigradeshare.io~ % aws configure set default.s3.multipart_threshold 1TB
The resulting file would looks like:
[plugins]endpoint = awscli_plugin_endpoint[default]s3 =multipart_threshold = 1TBendpoint_url = https://gateway.tardigradeshare.io
Now you can use your AWS CLI without specifying of the endpoint:
~ % aws s3 ls2021-01-08 19:41:13 demo
To configure s3api endpoint you can use this command:
~ % aws configure set default.s3api.endpoint_url https://gateway.tardigradeshare.io
You also can use a different profile for Storj:
~ % aws configure set profile.storj.s3.endpoint_url https://gateway.tardigradeshare.io~ % aws configure set profile.storj.s3.multipart_threshold 1TB
To use AWS CLI with a separate profile storj:
~ % aws s3 --profile storj ls2021-01-08 19:41:13 demo
Installing pluginawscli-plugin-endpoint will also install the AWS CLI v1!
