Hello Seekers! In this blog, I will take you through the procedure of compressing files into a zip from Amazon S3 without memory or disk usage. Interesting ? Hop in then.

Amazon S3

To put it simply, Amazon S3 is a virtual drive. It is an object storage system that you can use to store, retrieve and manage unlimited amounts of data from anywhere on the internet. S3 is one of the most popular cloud storage systems and huge amounts of data is transferred over it everyday. 

Imagine we have to download a very large number of files from S3. If we can compress these files into a zip file it will reduce the download time and data usage. While Amazon offers various methods for handling data, there is no method that retrieves multiple files as a single zip file. But, there is a workaround with AWS Lambda to achieve our goal.

AWS Lambda function

AWS Lambda is a serverless compute service. It allows the users to run code for virtually any type of backend service. AWS Lambda service can function without having to provision or manage servers.

File compression with S3 and Lambda

Fundamentally, the idea is that we stream the files from AWS S3 straight into a zip file, which then streams to S3 as we add files to it. We can use the AWS Lambda function for the job. Moreover, lambda does not use memory or disk space. Hence, this is the most effective way.

Lambda code to zip files from S3

The below code receives data and saves as a zip file at destination location

{

     "bucket": "aws-bucket-name",

     "destination_key": "location-to-s3/data.zip",

     "files": [

         {

             "uri": "s3-file-key-or-url",

             "filename": "file-name-of-file-inside-zip",

             "type": "options: [file, url]"

         }

     ]

 }