Fetch more than 1000 objects from S3 using C#


Hello!
Have you worked with S3? If you have, you know that we can’t get more than 1000 objects in a response from S3. So, the next question is, what do we do if we have more objects in S3?
Go ahead and read to know!

I have used ListObjectsAsync() here. See the following code.
‘bucket’ here refers to S3 bucket, and ‘key’ is the folder path inside the bucket(excluding the bucket name)

public static async Task<List<S3Object>> FetchS3ObjectsAsync(string bucket, string key)
{
    ListObjectsResponse response = null;
    List<S3Object> s3Objects = new List<S3Object>();
    using (AmazonS3Client client = new AmazonS3Client(<AwsAccessKey>, <AwsSecretAccessKey>, Amazon.RegionEndpoint.USEast1))
    {
        ListObjectsRequest request = new ListObjectsRequest
        {
            BucketName = bucket,
            Prefix = key
        };

        do
        {
            try
            {
                response = await client.ListObjectsAsync(request);

                foreach (S3Object obj in response.S3Objects)
                {
                    s3Objects.Add(obj);
                }
                if (response.IsTruncated)
                {
                    request.Marker = response.NextMarker;
                }
                else
                {
                    request = null;
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
            }
        } while (request != null);
    }
    return s3Objects;
}

Leave A Comment

Your email address will not be published. Required fields are marked *