Pooja Bhavani's Blog

Project-10: Mounting AWS S3 Bucket On Amazon EC2 Linux Using S3FS

Pooja Bhavani's photo
Pooja Bhavani
·

4 min read

Project Description

  • The project involves Mounting of AWS S3 Bucket On Amazon EC2 Linux Using S3FS.

  • This is a AWS Mini Project that will teach you AWS, S3, EC2, S3FS.

What is S3FS?

S3FS is a FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem. This means that you can access your S3 bucket files as if they were stored locally on your computer.

Why would you want to do this?

There are a few reasons why you might want to mount an S3 bucket as a local filesystem. For example, you might want to:

  • Access your S3 bucket files more quickly.

  • Use local filesystem tools to manage your S3 bucket files.

  • Share your S3 bucket files with other users on your local network.

How do I mount an S3 bucket as a local filesystem?

To mount an S3 bucket as a local filesystem, you will need to follow these steps:

  1. Create an EC2 server.

  2. Create an IAM user with programmatic access.

  3. Create a credential file for S3FS.

  4. Run the sudo s3fs command to mount the S3 bucket.

  5. Verify that the S3 bucket is mounted by running the mount command.

Here are the steps in detail:

  1. Create an EC2 server.

In the AWS console, create an EC2 instance. You can use any Linux AMI that you want.

  1. Create an IAM user with programmatic access.

In the IAM console, create an IAM user with programmatic access. Give the user the following permissions:

  • AmazonS3FullAccess

  • AmazonS3ReadOnlyAccess

  1. Create a credential file for S3FS.

Create a file called .passwd-s3fs in your home directory. The file should contain the following line:

AWS_ACCESS_KEY_ID:AWS_SECRET_KEY_ID

Replace AWS_ACCESS_KEY_ID and AWS_SECRET_KEY_ID with your IAM user's access key ID and secret access key.

  1. Run the sudo s3fs command to mount the S3 bucket.

Run the following command to mount the S3 bucket as a local filesystem:

sudo s3fs bucketname /path/to/mount -o passwd_file=.passwd-s3fs,nonempty,rw,allow_other

Replace bucketname with the name of your S3 bucket and /path/to/mount with the path to the directory where you want to mount the S3 bucket.

  1. Verify that the S3 bucket is mounted by running the mount command.

Run the following command to verify that the S3 bucket is mounted:

mount

You should see the S3 bucket listed in the output of the mount command.

That's it! You have now successfully mounted an S3 bucket as a local filesystem.

Here are some additional tips:

  • You can use the fstab file to make the S3 bucket mount persistent after a reboot.

  • You can use the -o url option to specify the endpoint for your S3 bucket.

  • You can use the -o use_path_request_style option to enable path-style requests for your S3 bucket.

Task -01

Step 1: Create a New IAM User

Begin by creating a new IAM user in the AWS console. Go to the IAM service, click on "Users," and then "Add user." Enter the name of the new user and proceed to the next step.

Step 2: Attach Policies to the User

During user creation, select "Attach Policies directly" and click on "Create Policy." Create a new policy with the settings mentioned below, focusing on S3-related actions:

  • Service: S3

  • Actions: ListAllMyBuckets, ListBucket, ListBucketVersions, GetObject, GetObjectVersion, PutObject

  • Resources: Specific

    • Bucket: Any

    • Object: Any

Step 3: Create IAM User and Attach Policy

After creating the policy, give it a suitable name and proceed to create the IAM user, attaching the newly created policy to the user.

Step 4: Get Access Keys

Once the user is created, go to "Security Credentials," and under "Access Keys," click on "Create Keys." Choose "Command Line Interface (CLI)" and get the Access Key and Secret Key.

Step 5: Create EC2 Instance

Create a new t2.micro instance on AWS EC2.

Step 6: Install AWS CLI

On the EC2 instance, install the AWS CLI using the appropriate package manager.

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Step 7: Install S3FS

After installing AWS CLI, proceed to install S3FS on the EC2 instance.

sudo apt install s3fs -y

Step 8: Create a Folder and Add Files

Create a folder named "bucket" at a location /home/ubuntu on the EC2 instance. Add 2–3 files to this folder.

mkdir bucket


touch test1.txt test2.txt test3.txt

Step 9: Create S3 Bucket

In the AWS console, create an S3 bucket with a suitable name.

Step 10: Configure AWS CLI

On the EC2 instance, configure the AWS CLI by running the command aws configure and providing the Access Key and Secret Key obtained earlier.

Step 11: Sync Files to S3 Bucket

Run the below command to sync the files from the given location on the EC2 instance to the S3 bucket.

aws s3 sync /home/ubuntu/bucket s3://devopschallenge-day89-s3bucket

Step 12: Verify the Sync

Refresh the objects inside the S3 bucket to confirm that all the files from the EC2 instance are successfully uploaded to the S3 bucket.

S3-bucketS3ec2