Maximizing Cost Savings with AWS DocumentDB(FinOps): Tips and Tricks for the Budget Conscious
Amazon Web Services (AWS) DocumentDB is a fully managed, highly scalable, and highly available NoSQL database service. It is designed to provide compatibility with the MongoDB API, enabling developers to build, run, and scale applications that utilize MongoDB operations without the need for costly infrastructure management. Despite its many advantages, costs can quickly add up, making it crucial for organizations to explore ways to optimize their AWS DocumentDB usage. In this blog post, we’ll share some practical tips and tricks to help you maximize cost savings for your AWS DocumentDB deployment.
Choose the Right Instance Type
Selecting the appropriate instance type is a crucial aspect of optimizing your DocumentDB costs. AWS offers various instance types with different CPU, memory, and I/O capacity combinations. To maximize cost savings, it’s essential to choose an instance type that closely matches your application’s performance requirements. Regularly monitor your application’s performance and use Amazon CloudWatch to identify the optimal instance type.
import boto3
# Create a DocumentDB client
docdb_client = boto3.client('docdb')
# Modify the instance type of an existing cluster
response = docdb_client.modify_db_instance(
DBInstanceIdentifier='your-instance-identifier',
DBInstanceClass='db.r5.large' # Choose an appropriate instance type
)Use Reserved Instances
Reserved Instances (RIs) are a cost-effective option when you have predictable and consistent workloads. By committing to a one or three-year term, you can save up to 40% or 70% respectively compared to On-Demand pricing. Analyze your usage patterns and reserve instances for long-term workloads to achieve significant cost savings.
Optimize Storage Capacity
DocumentDB allows you to choose between two storage types: General Purpose (SSD) and Provisioned IOPS (SSD). General Purpose storage is ideal for small to medium-sized databases, while Provisioned IOPS is designed for I/O intensive workloads. To minimize costs, opt for the storage type that best fits your application’s needs, and monitor storage usage to ensure you’re not over-provisioning.
Take Advantage of Autoscaling
Autoscaling helps you manage costs by automatically adjusting your DocumentDB cluster’s instance count based on demand. It ensures that you have the appropriate resources to handle your workload without over-provisioning. Enable autoscaling to optimize instance usage and reduce costs associated with idle resources.
# Import the necessary libraries
import boto3
# Create an Application Auto Scaling client
app_scaling_client = boto3.client('application-autoscaling')
# Register a scalable target (your DocumentDB cluster)
response = app_scaling_client.register_scalable_target(
ServiceNamespace='rds',
ResourceId='cluster:your-cluster-identifier',
ScalableDimension='rds:cluster:ReadReplicaCount',
MinCapacity=1,
MaxCapacity=5
)
# Create a scaling policy to scale the number of read replicas
response = app_scaling_client.put_scaling_policy(
PolicyName='your-policy-name',
ServiceNamespace='rds',
ResourceId='cluster:your-cluster-identifier',
ScalableDimension='rds:cluster:ReadReplicaCount',
PolicyType='TargetTrackingScaling',
TargetTrackingScalingPolicyConfiguration={
'TargetValue': 50,
'PredefinedMetricSpecification': {
'PredefinedMetricType': 'RDSReaderAverageCPUUtilization'
},
'ScaleInCooldown': 300,
'ScaleOutCooldown': 300
}
)Implement Data Archiving and TTL
Data archiving and Time-to-Live (TTL) features enable you to remove old or irrelevant data automatically, thus reducing storage costs. Set a TTL policy to delete documents when they reach a specific age or when they’re no longer needed. This approach not only helps you save on storage costs but also streamlines your database maintenance.
from pymongo import MongoClient
# Connect to your DocumentDB cluster
client = MongoClient("mongodb://username:password@your-cluster-endpoint:27017")
# Select the database and collection
db = client['your-db-name']
collection = db['your-collection-name']
# Create a TTL index
collection.create_index("expireAt", expireAfterSeconds=0)To use TTL, you need to add an expireAt field to your documents with the desired expiration datetime:
from datetime import datetime, timedelta
# Insert a document with an expireAt field set to expire in 30 days
expire_at = datetime.utcnow() + timedelta(days=30)
document = {"field1": "value1", "field2": "value2", "expireAt": expire_at}
result = collection.insert_one(document)Use Database Caching
To reduce the load on your DocumentDB instances and lower costs, implement a caching layer using services like Amazon ElastiCache. By storing frequently accessed data in a cache, you can minimize read and write operations, resulting in decreased latency and reduced costs.
Optimize Queries and Indexes
Optimizing your queries and indexes can significantly impact your DocumentDB’s performance and costs. Analyze slow queries using the DocumentDB profiler and fine-tune them to reduce CPU and memory consumption. Additionally, ensure you create appropriate indexes for your queries to minimize the number of read operations and increase efficiency.
from pymongo import MongoClient
# Connect to your DocumentDB cluster
client = MongoClient("mongodb://username:password@your-cluster-endpoint:27017")
# Select the database and collection
db = client['your-db-name']
collection = db['your-collection-name']
# Create an index on the `field1` and `field2` fields
collection.create_index([("field1", 1), ("field2", 1)])
# Query using the created index
result = collection.find({"field1": "value1", "field2": "value2"})Schedule Cluster Start/Stop Times
If you have non-production environments, such as development or testing, you can schedule your DocumentDB clusters to start and stop at specific times to save costs. For example, you can stop your clusters during weekends or off-business hours.
To automate this process, you can use AWS Lambda and Amazon EventBridge:
First, create a Lambda function that starts or stops your DocumentDB clusters:
import boto3
import os
docdb = boto3.client('docdb')
def lambda_handler(event, context):
action = event['action']
cluster_identifier = os.environ['CLUSTER_IDENTIFIER']
if action == 'start':
docdb.start_db_cluster(DBClusterIdentifier=cluster_identifier)
elif action == 'stop':
docdb.stop_db_cluster(DBClusterIdentifier=cluster_identifier)
else:
raise ValueError(f"Invalid action: {action}")Next, create two Amazon EventBridge rules: one to start the cluster and another to stop the cluster.
Start rule:
{
"source": ["aws.events"],
"detail-type": ["Scheduled Event"],
"detail": {
"action": ["start"]
}
}Stop rule:
{
"source": ["aws.events"],
"detail-type": ["Scheduled Event"],
"detail": {
"action": ["stop"]
}
}Set the desired schedule expression for each rule, for example:
- Start rule:
cron(0 12 ? * MON-FRI *)(start at 12:00 UTC on weekdays) - Stop rule:
cron(0 0 ? * MON-FRI *)(stop at 00:00 UTC on weekdays)
Implement Data Compression
Compressing your data before storing it in DocumentDB can lead to significant cost savings. Data compression reduces the amount of storage and network bandwidth required. You can use libraries like Snappy, LZ4, or gzip for compression.
Here’s an example using the gzip library in Python:
import gzip
import json
from io import BytesIO
from pymongo import MongoClient
# Connect to your DocumentDB cluster
client = MongoClient("mongodb://username:password@your-cluster-endpoint:27017")
# Select the database and collection
db = client['your-db-name']
collection = db['your-compressed-collection-name']
def compress_data(data):
buffer = BytesIO()
with gzip.GzipFile(fileobj=buffer, mode='wb') as f:
f.write(json.dumps(data).encode('utf-8'))
return buffer.getvalue()
# Insert compressed data
document = {"field1": "value1", "field2": "value2"}
compressed_data = compress_data(document)
result = collection.insert_one({"data": compressed_data})Monitor and Set Usage Alerts
By setting up usage alerts, you can proactively manage your DocumentDB costs. You can configure Amazon CloudWatch alarms to notify you when your costs exceed a certain threshold.
import boto3
cloudwatch = boto3.client('cloudwatch')
# Create a CloudWatch alarm for estimated DocumentDB costs
response = cloudwatch.put_metric_alarm(
AlarmName='DocumentDBCostAlarm',
AlarmDescription='Trigger an alarm when the estimated DocumentDB costs exceed the specified threshold',
Namespace='AWS/Billing',
MetricName='EstimatedCharges',
Dimensions=[
{
'Name': 'Currency',
'Value': 'USD'
},
{
'Name': 'ServiceName',
'Value': 'Amazon DocumentDB'
}
],
Statistic='SampleCount',
Period=86400,
EvaluationPeriods=1,Implement Connection Pooling
Connection pooling helps you manage the number of connections your application makes to DocumentDB, reducing the need to create new connections for each request. This can improve performance and reduce costs, as you may not require additional read replicas to handle increased connection demands.
Here’s an example using Python and the PyMongo driver:
from pymongo import MongoClient
# Connect to your DocumentDB cluster
connection_string = "mongodb://username:password@your-cluster-endpoint:27017"
client = MongoClient(
connection_string,
maxPoolSize=50, # Set the maximum number of connections in the pool
minPoolSize=10, # Set the minimum number of connections in the pool
waitQueueTimeoutMS=5000 # Set the timeout for waiting for a connection in the pool
)
# Select the database and collection
db = client['your-db-name']
collection = db['your-collection-name']Use Bulk Operations
Bulk operations can help you save costs by reducing the number of round trips between your application and DocumentDB. By performing multiple operations in a single request, you can minimize network latency and the number of read and write operations.
Here’s an example using Python and PyMongo:
from pymongo import MongoClient
from pymongo.errors import BulkWriteError
# Connect to your DocumentDB cluster
client = MongoClient("mongodb://username:password@your-cluster-endpoint:27017")
# Select the database and collection
db = client['your-db-name']
collection = db['your-collection-name']
# Prepare a list of documents for bulk insert
documents = [{"field1": "value1", "field2": "value2"},
{"field1": "value3", "field2": "value4"},
{"field1": "value5", "field2": "value6"}]
try:
# Perform a bulk insert operation
result = collection.insert_many(documents, ordered=False)
print("Bulk insert completed successfully.")
except BulkWriteError as e:
print("Bulk insert failed. Error details:", e.details)Reduce Data Transfer Costs
Data transfer costs can add up quickly, especially when transferring data between different AWS regions or between AWS and the internet. To minimize these costs, ensure that your DocumentDB cluster and the services accessing it are in the same region. Additionally, consider using AWS PrivateLink to establish a private connection between your DocumentDB cluster and other AWS services.
Here’s an example of how to create an Amazon VPC endpoint for your DocumentDB cluster:
import boto3
# Create an EC2 client
ec2 = boto3.client('ec2')
# Create a VPC endpoint for your DocumentDB cluster
response = ec2.create_vpc_endpoint(
VpcEndpointType='Interface',
VpcId='your-vpc-id',
ServiceName='com.amazonaws.<region>.docdb',
SubnetIds=['subnet-1', 'subnet-2'],
SecurityGroupIds=['sg-1', 'sg-2'],
PrivateDnsEnabled=True
)Use AWS Savings Plans
AWS Savings Plans is a flexible pricing model that offers significant discounts compared to On-Demand pricing. By committing to a consistent amount of usage over a one or three-year term, you can save up to 72% on your DocumentDB usage. Savings Plans apply to both compute and storage usage and can be used across AWS services.
To purchase a Savings Plan, visit the AWS Cost Management Console, choose the desired Savings Plan type and term, and specify your commitment amount.
Optimize Index Usage
Indexes can speed up query performance, but they also consume storage and memory resources. To optimize costs, review your indexes and remove any unnecessary or unused indexes. Be cautious when using compound indexes, as they can increase storage and memory usage.
To list all indexes in a collection and remove any unused index, you can use the following Python code with PyMongo:
from pymongo import MongoClient
# Connect to your DocumentDB cluster
client = MongoClient("mongodb://username:password@your-cluster-endpoint:27017")
# Select the database and collection
db = client['your-db-name']
collection = db['your-collection-name']
# List all indexes in the collection
indexes = collection.list_indexes()
for index in indexes:
index_name = index["name"]
print("Index name:", index_name)
# Decide whether to remove the index based on your criteria
if should_remove_index(index):
print("Removing index:", index_name)
collection.drop_index(index_name)Use Amazon CloudWatch Metrics and Logs
Monitoring your DocumentDB cluster using Amazon CloudWatch can help you identify performance bottlenecks, inefficiencies, and resource usage patterns. By analyzing these metrics, you can make informed decisions about resource allocation and cost optimization.
For example, if you notice consistently high CPU utilization, you may need to adjust your instance type or implement query optimization. Similarly, if you observe that your cluster’s storage is underutilized, you can reduce the allocated storage to save costs.
Here’s an example of how to retrieve CloudWatch metrics for your DocumentDB cluster using Python and Boto3:
import boto3
from datetime import datetime, timedelta
# Create a CloudWatch client
cloudwatch = boto3.client('cloudwatch')
# Define the time range for the metrics
end_time = datetime.utcnow()
start_time = end_time - timedelta(hours=1)
# Retrieve CPU utilization metrics for your DocumentDB cluster
response = cloudwatch.get_metric_data(
MetricDataQueries=[
{
'Id': 'cpu_utilization',
'MetricStat': {
'Metric': {
'Namespace': 'AWS/DocDB',
'MetricName': 'CPUUtilization',
'Dimensions': [
{
'Name': 'DBClusterIdentifier',
'Value': 'your-cluster-identifier'
}
]
},
'Period': 60,
'Stat': 'Average'
},
'ReturnData': True
}
],
StartTime=start_time,
EndTime=end_time
)
print("CPU Utilization Metrics:", response['MetricDataResults'])- Use Amazon CloudWatch Metrics and Logs
Monitoring your DocumentDB cluster using Amazon CloudWatch can help you identify performance bottlenecks, inefficiencies, and resource usage patterns. By analyzing these metrics, you can make informed decisions about resource allocation and cost optimization.
For example, if you notice consistently high CPU utilization, you may need to adjust your instance type or implement query optimization. Similarly, if you observe that your cluster’s storage is underutilized, you can reduce the allocated storage to save costs.
Here’s an example of how to retrieve CloudWatch metrics for your DocumentDB cluster using Python and Boto3:
pythonCopy codeimport boto3
from datetime import datetime, timedelta# Create a CloudWatch client
cloudwatch = boto3.client('cloudwatch')# Define the time range for the metrics
end_time = datetime.utcnow()
start_time = end_time - timedelta(hours=1)# Retrieve CPU utilization metrics for your DocumentDB cluster
response = cloudwatch.get_metric_data(
MetricDataQueries=[
{
'Id': 'cpu_utilization',
'MetricStat': {
'Metric': {
'Namespace': 'AWS/DocDB',
'MetricName': 'CPUUtilization',
'Dimensions': [
{
'Name': 'DBClusterIdentifier',
'Value': 'your-cluster-identifier'
}
]
},
'Period': 60,
'Stat': 'Average'
},
'ReturnData': True
}
],
StartTime=start_time,
EndTime=end_time
)print("CPU Utilization Metrics:", response['MetricDataResults'])
Use AWS Budgets
AWS Budgets is a powerful tool that allows you to set custom cost and usage budgets based on your requirements. You can create budgets for your overall AWS costs or specific services like DocumentDB. When your costs or usage exceed the defined thresholds, you receive notifications via email or SMS.
To set up a budget, visit the AWS Budgets Console and follow the guided steps to create a new budget.
Implement Caching
Using a caching layer in front of your DocumentDB cluster can help reduce the load on your database and save costs. Amazon ElastiCache is a popular choice for implementing caching. By caching frequently accessed data, you can minimize read operations and reduce the need for additional read replicas.
To create an Amazon ElastiCache cluster using Python and Boto3:
import boto3
# Create an ElastiCache client
elasticache = boto3.client('elasticache')
# Create an Amazon ElastiCache cluster for Redis
response = elasticache.create_cache_cluster(
CacheClusterId='your-cache-cluster-id',
Engine='redis',
CacheNodeType='cache.t3.small',
NumCacheNodes=1,
VpcSecurityGroupIds=['sg-1', 'sg-2'],
CacheSubnetGroupName='your-cache-subnet-group-name'
)Optimize Data Retention
DocumentDB automatically takes snapshots of your cluster to help you recover from any failures. By default, these snapshots are retained for 7 days, but you can modify this retention period based on your requirements. Reducing the retention period can help save costs on snapshot storage.
To modify the backup retention period for your DocumentDB cluster using Python and Boto3:
import boto3
# Create a DocumentDB client
docdb = boto3.client('docdb')
# Modify the backup retention period for your cluster
response = docdb.modify_db_cluster(
DBClusterIdentifier='your-cluster-identifier',
BackupRetentionPeriod=3 # Set the retention period to 3 days
)Enable Audit Logging Selectively
Audit logging in DocumentDB allows you to track and monitor user activity, which can be helpful for security and compliance purposes. However, audit logs consume storage, and the cost of storing these logs can add up over time.
To save costs, enable audit logging selectively based on your needs. For example, you can enable logging only for specific event categories or users.
To configure audit logging for your DocumentDB cluster using Python and Boto3:
import boto3
# Create a DocumentDB client
docdb = boto3.client('docdb')
# Enable audit logging for specific event categories
response = docdb.modify_db_cluster(
DBClusterIdentifier='your-cluster-identifier',
EnableCloudwatchLogsExports=['audit'],
CloudwatchLogsExportConfiguration={
'EnableLogTypes': [
'creation',
'deletion',
'modification',
'backup',
'restore'
]
}
)Implement Data Archiving
For some applications, you may store historical data that is infrequently accessed in your DocumentDB cluster. This data can consume storage resources, driving up costs. To save on storage costs, consider archiving older data to a more cost-effective storage solution, such as Amazon S3 or Amazon Glacier.
Here’s an example of how to export data from a DocumentDB cluster and store it in Amazon S3 using Python, Boto3, and the PyMongo driver:
import boto3
import json
from pymongo import MongoClient
# Connect to your DocumentDB cluster
client = MongoClient("mongodb://username:password@your-cluster-endpoint:27017")
# Select the database and collection
db = client['your-db-name']
collection = db['your-collection-name']
# Retrieve historical data from the collection
historical_data = collection.find({"date": {"$lt": "your-date-threshold"}})
# Serialize the data to a JSON file
with open("historical_data.json", "w") as f:
json.dump(list(historical_data), f)
# Create an S3 client
s3 = boto3.client('s3')
# Upload the JSON file to Amazon S3
with open("historical_data.json", "rb") as f:
s3.upload_fileobj(f, "your-s3-bucket", "historical_data/historical_data.json")
# Remove the historical data from the DocumentDB collection
collection.delete_many({"date": {"$lt": "your-date-threshold"}})Use AWS Trusted Advisor
AWS Trusted Advisor is a tool that provides real-time guidance on cost optimization, security, fault tolerance, and performance improvement. Trusted Advisor regularly checks your AWS resources and provides recommendations based on AWS best practices.
You can access Trusted Advisor through the AWS Management Console or by using the AWS CLI or SDKs.
To check Trusted Advisor recommendations using Python and Boto3:
import boto3
# Create a Trusted Advisor client
ta = boto3.client('support')
# Retrieve Trusted Advisor checks
checks = ta.describe_trusted_advisor_checks(language='en')['checks']
# Get the check ID for the Cost Optimization category
cost_optimization_check_id = next(
(check['id'] for check in checks if check['category'] == 'cost_optimization'),
None
)
if cost_optimization_check_id:
# Retrieve Trusted Advisor recommendations for Cost Optimization
recommendations = ta.describe_trusted_advisor_check_result(
checkId=cost_optimization_check_id,
language='en'
)
print("Trusted Advisor Cost Optimization Recommendations:", recommendations)
else:
print("Cost optimization check not found.")By adopting these additional cost-saving strategies and continuously monitoring your AWS DocumentDB deployments, you can further optimize your infrastructure costs. Regularly review your database setup and application requirements to fine-tune your configurations and identify new cost-saving opportunities. Keep up-to-date with AWS best practices and new features to get the most out of your deployments.
