AWS S3 Versioning Status and CloudWatch Dashboard Setup

2024-07-19 843 words 4 minutes

Contents

🇦 🇼 🇸 S3 Versioning Status and CloudWatch Dashboard Setup 🪧 ☁️ 🕙 🧺 📈

This document provides step-by-step instructions to set up a custom CloudWatch Dashboard that shows which S3 buckets have versioning enabled and disabled. It also includes the total number of buckets, separated by their versioning status.

Step 1: Create a Custom Policy for Lambda Function

Create a Custom Policy 🧾

➤ Go to IAM in the AWS Management Console.

➤ Click on Create Policy.

➤ Select the JSON tab and paste the following policy:

for policy access to github repo

json Copy code

 {
   "Version": "2012-10-17",
   "Statement": [
      {
        "Effect": "Allow",
        "Action": [
            "s3:ListAllMyBuckets",
            "s3:GetBucketVersioning",
            "cloudwatch:PutMetricData"
        ],
        "Resource": "*"
      }
   ]
}

Click on Next.

➤ Provide a suitable name for the custom policy and click on Create policy.

Step 2: Create a Lambda Function

Create the Lambda Function ⚙️

➤ Go to Lambda in the AWS Management Console.

➤ Click on Create function.

➤ Provide a suitable function name.

➤ Select runtime (Python 3.8).

➤ Click on Create function.

Assign IAM Role to Lambda Function 🙋

➤ Click on Configuration > Permissions.

➤ Click on the Role name to open the IAM Role page.

➤ Click on Add permissions > Attach policies.

➤ Click on Filter policies and select Customer managed.

➤ Search for the policy created in Step 1, select it, and click on Add permissions.

Add Lambda Function Code

➤ Go back to the Lambda function.

➤ Click on Code.

➤ In the Lambda function section, paste the following code After pasting the code click on Deploy.

➤ Lambda Code for Bucket Versioning Status

for code access to github repo

python

Copy code

import boto3
import json
import datetime

s3_client = boto3.client('s3')
cloudwatch_client = boto3.client('cloudwatch')

def lambda_handler(event, context):
buckets = s3_client.list_buckets()
for bucket in buckets['Buckets']:
    bucket_name = bucket['Name']
    versioning = s3_client.get_bucket_versioning(Bucket=bucket_name)
    status = versioning.get('Status', 'Disabled')

    # Publish metrics to CloudWatch
    cloudwatch_client.put_metric_data(
        Namespace='S3/Versioning',
        MetricData=[
            {
                'MetricName': 'BucketVersioningStatus',
                'Dimensions': [
                    {
                        'Name': 'BucketName',
                        'Value': bucket_name
                    },
                ],
                'Timestamp': datetime.datetime.utcnow(),
                'Value': 1 if status == 'Enabled' else 0,
                'Unit': 'Count'
            },
        ]
    )

return {
    'statusCode': 200,
    'body': json.dumps('Metrics published successfully.')
}

Test the Lambda Function ✅

➤ Click on Test > Configure test event.

➤ Provide an event name.

➤Click on Save.

➤ Click on Test to run the Lambda function.

Step 3: Create a CloudWatch Dashboard

Create a Dashboard 🪧

➤ Go to CloudWatch in the AWS Management Console.

➤ Click on Dashboards > Create dashboard.

➤ Provide a name and click on Create dashboard.

Add Widgets to the Dashboard

➤ Click on Add widget.

➤ Select Number and click on Next.

➤ Select the namespace S3/Versioning and metric

➤ BucketVersioningStatus.

➤ Select all buckets and click on Create widget.

➤ Click on the three dots of the widget and select Edit.

➤ Click on Data table and select Only display summary columns in the table.

➤ In Summary, select Average and click on Update widget.

➤ Create a new Lambda function similar to the first one, but with the following code:

➤ Lambda Code for Total Number of Buckets and Versioning Status

for code access to github repo

python

Copy code

import boto3
import json
import datetime

def lambda_handler(event, context):
s3 = boto3.client('s3')
cloudwatch = boto3.client('cloudwatch')

# Get list of all S3 buckets
buckets = s3.list_buckets()

total_buckets = len(buckets['Buckets'])
versioning_enabled_buckets = []
versioning_disabled_buckets = []

for bucket in buckets['Buckets']:
    bucket_name = bucket['Name']
    versioning = s3.get_bucket_versioning(Bucket=bucket_name)
    
    if 'Status' in versioning and versioning['Status'] == 'Enabled':
        versioning_enabled_buckets.append(bucket_name)
    else:
        versioning_disabled_buckets.append(bucket_name)

# Publish custom metrics to CloudWatch
cloudwatch.put_metric_data(
    Namespace='S3/Versioning',
    MetricData=[
        {
            'MetricName': 'TotalBuckets',
            'Timestamp': datetime.datetime.utcnow(),
            'Value': total_buckets,
            'Unit': 'Count'
        },
        {
            'MetricName': 'VersioningEnabledBuckets',
            'Timestamp': datetime.datetime.utcnow(),
            'Value': len(versioning_enabled_buckets),
            'Unit': 'Count'
        },
        {
            'MetricName': 'VersioningDisabledBuckets',
            'Timestamp': datetime.datetime.utcnow(),
            'Value': len(versioning_disabled_buckets),
            'Unit': 'Count'
        }
    ]
)

# Create a detailed log output
response = {
    'TotalBuckets': total_buckets,
    'VersioningEnabledBuckets': versioning_enabled_buckets,
    'VersioningDisabledBuckets': versioning_disabled_buckets
}

return {
    'statusCode': 200,
    'body': json.dumps(response)
}

➤ Click on the + button in the CloudWatch dashboard.

➤ Select Data table and click on Next.

➤ Select the namespace S3/Versioning.

➤ Select metrics with no dimensions.

➤ Select all metric names and click on Create widget.

➤ Click on the three dots of the widget and select Edit.

➤ Select options as per the previous widget and click on Update widget.