Tardigrade S3 Gateway

An application that mimics the AWS S3 API. Migrate your data from AWS to Tardigrade with minimal effort.
A download can become a chargeable event for 2 times the actual file size if the gateway is running on another cloud provider. We recommend interfacing with the network directly through the Uplink Library.
For a complete list of the supported architectures and API calls for the S3 Gateway, see Concepts: S3 Compatibility.
Minimum Requirements
✅  1 CPU
✅  2GB of RAM
Depending on the load and throughput, more resources may be required.
Dependencies
✅  Tardigrade Satellite Account​
✅  Tardigrade Satellite Project​
✅  Tardigrade Satellite API Key​
Set Up
1. Download, unzip, and install the binary for your OS:
Windows
Linux
macOS
Docker
Windows
Curl Download (PowerShell)
curl https://github.com/storj/gateway/releases/latest/download/gateway_windows_amd64.exe.zip -o gateway_windows_amd64.exe.zip; Expand-Archive gateway_windows_amd64.exe.zip -Destination . -Force
Direct Download
​Windows Gateway Binary​
Linux
AMD64
Curl Download
curl -L https://github.com/storj/gateway/releases/latest/download/gateway_linux_amd64.zip -O && unzip gateway_linux_amd64.zip
chmod 755 gateway
sudo mv gateway /usr/local/bin/gateway
Direct Download
​Linux AMD64 Gateway Binary​
ARM
Curl Download
curl -L https://github.com/storj/gateway/releases/latest/download/gateway_linux_arm.zip -O && unzip gateway_linux_arm.zip
chmod 755 gateway
sudo mv gateway /usr/local/bin/gateway
Direct Download
​Linux ARM Gateway Binary​
macOS
Curl Download
curl -L https://github.com/storj/gateway/releases/latest/download/gateway_darwin_amd64.zip -O && unzip gateway_darwin_amd64.zip
chmod 755 gateway
sudo mv gateway /usr/local/bin/gateway
Direct Download
​macOS Gateway Binary​
Docker
docker pull storjlabs/gateway
2. Setup your S3 gateway by running the following command and following the instructions provided by the wizard:
Windows
Linux
macOS
Docker
Windows
PowerShell
Navigate to the directory your gateway.exe file is located in (for example: Desktop)
cd $env:USERPROFILE/Desktop/
./gateway.exe setup
Linux
gateway setup
macOS
gateway setup
Docker
docker run -it --rm --mount type=bind,source=/path/to/gateway-config-dir/,destination=/root/.local/share/storj/gateway/ --name gateway storjlabs/gateway setup
3. Enter the numeric choice or satellite address corresponding to the satellite you've created your account on. 
The satellite address should be entered as <nodeid>@<address>:<port> for example: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@europe-west-1.tardigrade.io
4. Choose an access name (this step may not yet be implemented in the version of S3 Gateway you are using - if you don't see this prompt, skip to step 5 below).
If you would like to choose your own access name, please be sure to only use lowercase letters. Including any uppercase letters will result in your access name not getting recognized when creating buckets.
5.  Enter the API key you generated:
Didn't save your API key? Simply create another one in the satellite web interface.
6. Create and confirm an encryption passphrase, which is used to encrypt your files before they are uploaded:
Please note that Storj Labs does not know or store your encryption passphrase, so if you lose it, you will not be able to recover your files. 
7.  Your S3 gateway is configured and ready to use!
Configure using an Access Grant
To obtain an access grant with Uplink CLI, enter:
Windows
Linux
macOS
Windows
cd $env:USERPROFILE/Desktop/
./uplink share sj://site --readonly --not-after +2h
Linux
uplink share sj://site --readonly --not-after +2h
macOS
uplink share sj://site --readonly --not-after +2h
Please note that the--readonly and -not-after flags are optional.
Now we got our access grant and can configure the gateway as follows:
Windows
Linux
macOS
Docker
Windows
cd $env:USERPROFILE/Desktop/
./gateway setup --access 14aV.... --non-interactive
Linux
gateway setup --access 14aV.... --non-interactive
macOS
gateway setup --access 14aV.... --non-interactive
Docker
docker run -it --rm --mount type=bind,source=/path/to/gateway-config-dir/,destination=/root/.local/share/storj/gateway/ --name gateway storjlabs/gateway setup --access 14aV.... --non-interactive
This command will set up the default access in the gateway config file.
You can see the path to the default config file config.yaml with this command:
Windows
Linux
macOS
Windows
cd $env:USERPROFILE/Desktop/
./gateway help
Linux
gateway help
macOS
gateway help
Add multiple Access Grants to the Gateway config file
You can add several access grants to the config.yaml. using this format:
access: 14aV.... # default Access
accesses.site: 26NBm..... # the Access with name "site"
Run the Gateway
The gateway functions as a daemon. Start it and leave it running.
Windows
Linux
macOS
Docker
Windows
./gateway.exe run
Linux
gateway run
macOS
gateway run
Docker
docker run -it --rm -p 127.0.0.1:7777:7777 --mount type=bind,source=/path/to/gateway-config-dir/,destination=/root/.local/share/storj/gateway/ --name gateway storjlabs/gateway run
If you want to connect to your S3 Gateway via the network, then you should replace the -p 127.0.0.1:7777:7777 port mapping with -p 7777:7777
The gateway should output your S3-compatible endpoint, access key, and secret key.
Example terminal output
Running the Gateway with Access Grant
You can run a gateway with specifying the access grant (or its name) with the option --access, for example:
Windows
Linux
macOS
Docker
Windows
./gateway run --access 14aV....
or with the name of the access grant from your config (see Add multiple Access grants)
./gateway run --access site
Linux
gateway run --access 14aV....
or with name of the access grant from your config (see Add multiple Access grants)
gateway run --access site
macOS
gateway run --access 14aV....
or with name of the access grant from your config (see Add multiple Access grants)
gateway run --access site
Docker
docker run -it --rm -p 127.0.0.1:7777:7777 --mount type=bind,source=/path/to/gateway-config-dir/,destination=/root/.local/share/storj/gateway/ --name gateway storjlabs/gateway run --access 14aV....
or with name of the access grant from your config (see Add multiple Access grants)
docker run -it --rm -p 127.0.0.1:7777:7777 --mount type=bind,source=/path/to/gateway-config-dir/,destination=/root/.local/share/storj/gateway/ --name gateway storjlabs/gateway run --access site
Running the Gateway to host a static website
You can also run a gateway to handle a bucket as a static website. Make sure to limit the access to the exposed buckets.
Windows
Linux
macOS
Docker
Windows
./gateway run --access 14aV.... --website
or with the name of the access grant from your config (see Add multiple Access grants)
./gateway run --access site --website
Linux
gateway run --access 14aV.... --website
or with name of the access grant from your config (see Add multiple Access grants)
gateway run --access site --website
macOS
gateway run --access 14aV.... --website
or with name of the access grant from your config (see Add multiple Access grants)
gateway run --access site --website
Docker
docker run -it --rm -p 127.0.0.1:7777:7777 --mount type=bind,source=/path/to/gateway-config-dir/,destination=/root/.local/share/storj/gateway/ --name gateway storjlabs/gateway run --access 14aV.... --website
or with name of the access grant from your config (see Add multiple Access grants)
docker run -it --rm -p 127.0.0.1:7777:7777 --mount type=bind,source=/path/to/gateway-config-dir/,destination=/root/.local/share/storj/gateway/ --name gateway storjlabs/gateway run --access site --website
Now you can navigate to http://localhost:7777/site/ to see the bucket site as XML or to http://localhost:7777/site/index.html to see a static page, uploaded to the bucket site.
You can publish this page to the internet, but in this case, you should run your gateway with the option --server.address local_IP:local_Port (replacelocal_IPwith the local IP of your PC andlocal_Port with the port you want to expose). 
If you uselocalhost or 127.0.0.1 as your local_IP, you will not be able to publish it directly (via port forwarding for example), instead, you will have to use a reverse proxy here.
Running the Gateway to host a static website with cache
You can use the Minio caching technology in conjunction with the hosting of a static website.
The following example uses /mnt/drive1, /mnt/drive2 ,/mnt/cache1 ... /mnt/cache3 for caching, while excluding all objects under bucket mybucket and all objects with '.pdf' extensions on a S3 Gateway setup. Objects are cached if they have been accessed three times or more. Cache max usage is restricted to 80% of disk capacity in this example. Garbage collection is triggered when the high watermark is reached (i.e. at 72% of cache disk usage) and will clear the least recently accessed entries until the disk usage drops to the low watermark - i.e. cache disk usage drops to 56% (70% of 80% quota).
Export the environment variables before running the Gateway:
Windows
Linux
macOS
Docker
Windows
Cache disks are not supported, because caching requires the atime function to be enabled.
$env:MINIO_CACHE="on"
$env:MINIO_CACHE_EXCLUDE="*.pdf,mybucket/*"
$env:MINIO_CACHE_QUOTA=80
$env:MINIO_CACHE_AFTER=3
$env:MINIO_CACHE_WATERMARK_LOW=70
$env:MINIO_CACHE_WATERMARK_HIGH=90
Linux
export MINIO_CACHE="on"
export MINIO_CACHE_DRIVES="/mnt/drive1,/mnt/drive2,/mnt/cache{1...3}"
export MINIO_CACHE_EXCLUDE="*.pdf,mybucket/*"
export MINIO_CACHE_QUOTA=80
export MINIO_CACHE_AFTER=3
export MINIO_CACHE_WATERMARK_LOW=70
export MINIO_CACHE_WATERMARK_HIGH=90
macOS
export MINIO_CACHE="on"
export MINIO_CACHE_DRIVES="/mnt/drive1,/mnt/drive2,/mnt/cache{1...3}"
export MINIO_CACHE_EXCLUDE="*.pdf,mybucket/*"
export MINIO_CACHE_QUOTA=80
export MINIO_CACHE_AFTER=3
export MINIO_CACHE_WATERMARK_LOW=70
export MINIO_CACHE_WATERMARK_HIGH=90
Docker
You can create a file with environment variables, for example - minio_vars with such content:
MINIO_CACHE="on"
MINIO_CACHE_DRIVES="/mnt/drive1,/mnt/drive2,/mnt/cache{1...3}"
MINIO_CACHE_EXCLUDE="*.pdf,mybucket/*"
MINIO_CACHE_QUOTA=80
MINIO_CACHE_AFTER=3
MINIO_CACHE_WATERMARK_LOW=70
MINIO_CACHE_WATERMARK_HIGH=90
For Windows, the option -e MINIO_CACHE_DRIVES is useless due to the lack of an atime function, and can be removed with mounts those drives to the docker container.
Then add parameters --env-file ./minio_vars --mount type=bind,src=/mnt/drive1,dst=/mnt/drive1 --mount type=bind,src=/mnt/drive2,dst=/mnt/drive2 --mount type=bind,src=/mnt/cache1,dst=/mnt/cache1 --mount type=bind,src=/mnt/cache2,dst=/mnt/cache2 --mount type=bind,src=/mnt/cache3,dst=/mnt/cache3 to the docker run section, for example:
docker run -it --rm -p 127.0.0.1:7777:7777 --env-file ./minio_vars --mount type=bind,src=/mnt/drive1,dst=/mnt/drive1 --mount type=bind,src=/mnt/drive2,dst=/mnt/drive2 --mount type=bind,src=/mnt/cache1,dst=/mnt/cache1 --mount type=bind,src=/mnt/cache2,dst=/mnt/cache2 --mount type=bind,src=/mnt/cache3,dst=/mnt/cache3 --mount type=bind,source=/path/to/gateway-config-dir/,destination=/root/.local/share/storj/gateway/ --name gateway storjlabs/gateway run --access site --website
Setting MINIO_BROWSER=off env variable would disable the Minio browser. This would make sense if running the gateway as a static website in production.
Then run the gateway with static site enabled.
Reconfiguring the AWS CLI to interface with the Tardigrade Network
To show that this gateway is working, first, make sure you have the AWS S3 CLI installed.
Once you do, in a new terminal session, configure it with your Gateway's credentials:
$ aws configure
---
AWS Access Key ID: [Enter your Gateway's Access Key]
AWS Secret Access Key: [Enter your Gateway's Secret Key]
Default region name: [null]
Default output format: [null]
Then, test out some AWS S3 CLI commands!
Create a bucket
AWS CLI
AWS CLI
aws s3 --endpoint=http://localhost:7777/ mb s3://bucket-name
Upload an object
AWS CLI
AWS CLI
aws s3 --endpoint=http://localhost:7777/ cp ~/Desktop/your-large-file.mp4 s3://bucket-name
List objects in a bucket
AWS CLI
AWS CLI
aws s3 --endpoint=http://localhost:7777/ ls s3://bucket-name/
Download an object
AWS CLI
AWS CLI
aws s3 --endpoint=http://localhost:7777/ cp s3://bucket-name/your-large-file.mp4 ~/Desktop/your-large-file.mp4
Generate a URL for an object
AWS CLI
AWS CLI
aws s3 --endpoint=http://localhost:7777/ presign s3://bucket-name/your-large-file.mp4
(This URL will allow live video streaming through your browser or VLC)
Delete an object
AWS CLI
AWS CLI
aws s3 --endpoint=http://localhost:7777/ rm s3://bucket-name/your-large-file.mp4
All Commands
​cp - Copies a local file or S3 object to another location locally or in S3
​ls - List S3 objects and common prefixes under a prefix or all S3 buckets
​mb - Creates an S3 bucket
​mv - Moves a local file or S3 object to another location locally or in S3.
​presign - Generate a pre-signed URL for an S3 object. This allows anyone who receives the pre-signed URL to retrieve the S3 object with an HTTP GET request.
​rb - Deletes an empty S3 bucket
​rm - Deletes an S3 object
​sync - Syncs directories and S3 prefixes. Recursively copies new and updated files from the source directory to the destination. Only creates folders in the destination if they contain one or more files
And that's it! You've learned how to use our S3-compatible Gateway. Ideally, you'll see how easy it is to swap out AWS for the Uplink, going forward.
​
Next - SDK & Reference
Storj Client Libraries
Last updated 3 months ago