How can you view the distribution of AWS spending in one of your AWS accounts?

Features of AWS Cost Explorer                                     

Popular features of AWS Cost Explorer include:

• Filters and grouping of data.
• Ability to access data programmatically through the ad-hoc analytics engine that powers Cost Explorer.
• Forecasting of usage and costs, which utilizes machine learning and rule-based models to predict charges.
• Data is segmented based on charge types such as RI usage or on-demand usage.
• Access to AWS RI purchase recommendations.
• Access to EC2 RI purchase recommendations from the Cost Explorer API.

Types of reports available in AWS Cost Explorer

Users can choose to view a variety of default and specific reports for service costs and usages. Default report types include:  

• Monthly Costs by AWS Service- Which provides a view for costs and usages over the previous six months, grouped by the five most cost-occurring AWS services [additional services used are grouped into a single bar]. This report will also provide a breakdownoneach service in a table format.
• EC2 Monthly Cost and Usage- Which provides a view for AWS costs over the last two months and the current month. This report also allows users to view cost and usages with linked accounts, tags and regions.
• Monthly Costs by Linked Account- Which provides a view for distribution of costs among an organization through grouping linked accounts.
• Monthly Running Costs- Which provides a viewforall costs over the past three months and forecasts the next month’s costs.

Additional report types include:

• AWS Marketplace- Which allows users to view costs through the AWS Marketplace.
• Daily costs- Which allows users to view costs in the time period of the past six months with a one month forecast.
• Monthly EC2 Running Hours Costs and Usage- Which allows users to view the costs and usage of active RIs.
• RI Utilization Reports- Which allows users to view their usage of RIs in Amazon EC2, Redshift, RDS, Elasticsearch ServiceandElastiCache. This report also shows how much the user saves or overspends by usingRIs.

  AWS Billing allows you to track your AWS infrastructure billing forecasts and costs, including CloudWatch usage.

  Nội dung chính Show

  • Installation
  • Metrics collection
  • Log collection
  • Monitoring CloudWatch usage
  • Data Collected
  • Service Checks
  • Troubleshooting
  • No metrics are reporting from the AWS billing integration
  • Missing metrics
  • Some important principles of AWS cost reduction
  • Step 1 - Get All Relevant Data
  • Where’s the money going?
  • Step 2 - Identify Anomalies and Corrective Actions
  • EC2 Compute
  • RDS Compute
  • Data Transfer
  • Step 3 - Confirm the right balance of Performance, Availability and Operational Excellence.
  • a] Identify affected applications, components and deployment stages by cost
  • b] Identify main business transactions supported by identified applications and components
  • c] Identify Performance requirements for identified transactions and stages [dev, QA, production, etc.]
  • d] Identify Availability requirements for identified transactions and stages
  • Step 4 - Apply Corrective Actions
  • Right-size and Optimize Infrastructure
  • Optimize Applications
  • Delete, Terminate or Stop
  • Purchase Discounts
  • Negotiate Discounts
  • Step 5 - Iterate
  • To Summarize
  • Do you need help lowering your AWS cost?

  Enable this integration to see billing metrics in Datadog.

  Setup

  Installation

  If you haven’t already, set up the Amazon Web Services integration first.

  Metrics collection

  Note: AWS Budgets metrics can only be collected from your AWS primary account.

  Log collection

  Enable logging

  Configure AWS billing to send logs either to a S3 bucket or to CloudWatch.

  Note: If you log to a S3 bucket, make sure that amazon_billing is set as Target prefix.

  Send logs to Datadog

  Monitoring CloudWatch usage

  After setting up your AWS permissions to add the budgets:ViewBudget permission, you can monitor CloudWatch billing with this integration.

  AWS billing metrics are available about once every 4 hours. You may have to wait 4 hours for Datadog to collect the metrics.

  Once the metrics are available, look at aws.billing.estimated_charges and aws.billing.forecasted_charges. You can use these metrics to track your CloudWatch usage by filtering the context down to service:amazoncloudwatch. You can break down the spend to each AWS account using max:account_id.

  The metric aws.billing.estimated_charges is what AWS believes to be the CloudWatch bill so far for the current month. This value is reset to 0 at the start of each month. The metric aws.billing.forecasted_charges is your estimated CloudWatch bill for the end of the month based on current usage.

  Data Collected

  Metrics

  aws.billing.estimated_charges
  [gauge]The estimated charges for your AWS usage. This can either be estimated charges for one service or a roll-up of estimated charges for all services.
  Shown as dollaraws.billing.actual_spend
  [gauge]The actual spending costs for your budget period
  Shown as dollaraws.billing.forecasted_spend
  [gauge]The forecasted spending costs for your budget period
  Shown as dollaraws.billing.budget_limit
  [gauge]The spending limit for your budget period
  Shown as dollar

  Each of the metrics retrieved from AWS is assigned the same tags that appear in the AWS console, including but not limited to host name, security-groups, and more.

  Events

  The AWS Billing integration does not include any events.

  Service Checks

  The AWS Billing integration does not include any service checks.

  Troubleshooting

  No metrics are reporting from the AWS billing integration

  Here is a check list that you can apply to troubleshoot your integration:

  1. Ensure your IAM policy has budgets:ViewBudget.
  2. Ensure billing metrics are enabled in your payer account.

  Note: AWS Billing metrics are collected every 4 or 8 hours by Datadog.

  Missing metrics

  If you are missing, aws.billing.actual_spend, aws.billing.forecasted_spend, or aws.billing.budget_limit, create an AWS budget to start receiving the metrics in Datadog.

  Note: AWS Billing metrics are collected every 4 or 8 hours by Datadog.

  One of the most common problems AWS customers face is the dreaded “Bad AWS Billing Surprise”. It’s the beginning of the month, you’ve just received your latest AWS bill and found out that you owe AWS a completely unexpected amount of money [typically a large sum, hundreds or thousands of dollars above what you originally expected, depending on your budget]. There are also situations where you regularly spend far more than you should over an extended period of time, which can be equally as expensive.

  If you’re in either of these situations, a critical question is:

  How do you reduce your AWS cost in a responsible way, without putting at risk your applications and the business transactions they support?

  There’s no easy answer to this problem. That’s why in this article I’ll walk you through a proven process I’ve put together in order to reduce AWS cost…

  Some important principles of AWS cost reduction

  Before you start terminating EC2 instances, spending money on Reserved purchases or making any changes to your AWS infrastructure, keep in mind these important principles:

  • Be patient. A full cost optimization cycle can easily take 3 months or longer. There will be Quick Wins, but getting to a final goal can be an iterative process throughout multiple AWS billing cycles.
  • You have to find the right balance. Based on your application needs, you have to make sure you’re paying for the right amount of Performance, Availability and Operational Excellence that your business requires. Security should be the last area -if at all- to consider when it comes to cost reductions [in most cases, the greatest portion of cost is related to performance and availability anyways].
  • **Cost reductions need to be prioritized [some are not even worth the effort]**. Implementing infrastructure updates can be risky and it can take precious time and effort away from you and your team. Sometimes it happens that the engineering time required to optimize a certain area of your AWS infrastructure is far more expensive than the actual benefits. Calculating whether you should go ahead with a cost reduction project should include the expected savings vs. both the actual engineering hours and the cost of opportunity [i.e. not focusing on building new features due to some cost reduction tasks]. For example, spending $5,000 in engineering time for a $200/month reduction in your AWS bill might not be worth it. I’ll cover this area in more detail below, so keep reading.

  With these principles in mind, we can continue with the actual steps…

  Step 1 - Get All Relevant Data

  The first thing you need to do is get all cost data and relevant system metrics in front of you, so you can find cost-related inefficiencies.

  Below are some important tools that can help you find relevant data:

  • AWS Cost and Usage Reports. These are detailed reports that AWS creates 2-3 times per day and are stored in an S3 bucket that you define. Data can be as granular as 1 hour and they have all the relevant usage information you’ll need. The downside is that these reports can easily turn into many thousands of records, therefore you need to have a scalable way to analyze them. Check out this article I wrote, which describes how to use AWS Athena to analyze Cost and Usage Reports and find your main usage types and resources by cost. AWS Athena is a great tool to drill down on your AWS Cost and Usage data and find areas to optimize.
  • AWS Cost Explorer. This is a graphic interface in the AWS console that allows you to visualize relevant items that contribute to your AWS cost, such as top services, usage types, API operations and others. Even though Cost Explorer is readily available and it’s easy to use, querying Cost and Usage reports can be a much more effective way to find key areas that contribute to AWS cost. For example, Cost Explorer doesn’t give you a full view of AWS resources [EC2 instances, RDS DB instances, S3 buckets, etc.] and the usage types each resource consumes, which is an essential step for getting to the bottom of many cost-related issues.
  • CloudWatch Metrics. Once the top resources by cost are identified, sometimes it’s necessary to look at the actual system metrics for specific AWS components. That’s where a service like CloudWatch is essential in order to identify areas for cost optimizations, such as idle or under-utilized resources. CloudWatch is also important when making informed cost optimization decisions that could potentially affect performance or availability.
  • AWS Trusted Advisor. This AWS service executes periodic checks on your AWS infrastructure and comes up with a list of recommendations. There is a section for Cost Optimizations, which typically contains useful findings, such as idle or under-utilized resources, among others. In order to access these findings, you have to enable AWS Support, which has a cost depending on the plan you choose [e.g. Developer, Business or Enterprise].
  • AWS Compute Optimizer. This AWS feature creates an automated report that analyzes compute resources in your account and comes up with recommendations for EC2 usage optimizations.

  I also recommend installing MiserBot, a Slack and email bot I developed, which gives you daily updates on your AWS cost and a detailed view of where your money is going.

  Now that you know where to find important cost data, it’s time to look for specific information…

  Where’s the money going?

  The first step is to understand the areas that result in the most AWS cost, and that you also identify their percentage relative to your total AWS bill.

  There are three important dimensions to analyze:

  A. Top AWS Usage Types by Cost

  The first meaningful step is to understand the AWS usage types that incur in the most money spent. Some common examples:

  • BoxUsage. Anything in this category refers to an On Demand EC2 instance type. As we’ll cover below, On Demand EC2 usage can be reduced by purchasing EC2 Reserved instances or AWS Savings Plans.
  • TimedStorage. S3 storage cost starts with this label and there are some variations depending on the storage class. Cost can be reduced by allocating objects to a specific storage class, such as Infrequent Access, depending on your application’s needs for availability or access to specific objects in S3.
  • InstanceUsage. On Demand RDS DB compute time. Similarly to EC2, On Demand RDS usage can be optimized by purchasing Reserved RDS instances.
  • HeavyUsage. EC2 and RDS Reserved instances. Usage in this category represents Reserved purchases in your account. It’s important to keep track of these purchases and make sure they’re being used properly. It’s not uncommon to see cases where customers don’t take advantage of Reserved discounts available in their AWS accounts.
  • DataTransfer-Out. This is data transferred out to the internet. For example, from EC2 instances, Load Balancers or CloudFront distributions. In some applications -such as high traffic or media sites- this can represent a top percentage of money spent.

  Using the tools described in this article, you can run the following Athena query:

  SELECT lineitem_productcode, lineItem_UsageType, round[sum[cast[lineitem_unblendedcost AS double]],2] AS sum_unblendedcost FROM billing.hourly WHERE period='' GROUP BY lineitem_productcode, lineItem_UsageType ORDER BY sum_unblendedcost DESC

  Once you have the top usage types, you’ll have a good starting point to uncover areas that are impacting your AWS bill. For example:

  • Why are you spending a particular amount of money on a specific EC2 instance type?
  • Identify anomalies, such as a large RDS or EC2 instance type that you weren’t aware of.

  B. Top AWS Resources by Cost

  This is a list of the top AWS resources, such as specific EC2 instances, RDS DB instances, S3 buckets, CloudFront distributions, etc. This will give you a detailed view of the most expensive AWS components in your application. Once you have this list, focus on the top 5-10 and also on those that are a considerable expense for your budget.

  Using Athena, you can run the following query:

  SELECT lineitem_productCode, lineitem_resourceId, sum[cast[lineitem_unblendedcost AS double]] AS sum_unblendedcost FROM billing.hourly WHERE period='' GROUP BY lineitem_productCode,lineitem_resourceId ORDER BY sum_unblendedcost desc LIMIT 100

  Keep in mind, this is not information you can find in AWS Cost Explorer, that’s why I recommend using Athena to analyze Cost and Usage reports.

  C. Find usage types for the top AWS resources

  Once you find the top AWS resources by cost, it’s time to drill down on each component and see what’s costing your applications the most money. For example:

  • You have an expensive CloudFront distribution. Are you paying a lot of money for Requests, DataTransfer or both?
  • You have an expensive EC2 instance. Are you paying for BoxUsage [compute time], DataTransfer-Out [internet] or DataTransfer-Regional [cross Availability Zones]?
  • You have an expensive RDS DB instance. Are you paying for InstanceUsage [compute time], StorageIOUsage [disk I/O operations] or Storage [disk storage]?
  • You have an expensive S3 bucket. Are you paying for Requests [API calls, billed by usage tier], DataTransfer-Out [data transfer out to the internet], TimedStorage [storage, you should see if it’s Standard, Standard Infrequent Access, One-Zone Infrequent Access, Reduced Redundancy, Glacier, Glacier Deep Archive].

  This is my recommended Athena query to find the usage types for a particular AWS resource:

  SELECT DISTINCT lineitem_usagetype, sum[cast[lineitem_usageamount AS double]] AS sum_usageamount, sum[cast[lineitem_unblendedcost AS double]] AS sum_unblendedcost FROM billing.hourly WHERE lineitem_resourceId = '' AND period='' GROUP BY lineitem_usagetype ORDER BY sum_unblendedcost DESC

  Once you know exactly which resources consume the most amount of money and their usage types, then you are in a position to take some corrective actions. Similarly to finding the top AWS resources by cost, this information is not available in AWS Cost Explorer, only in Cost and Usage Reports.

  Step 2 - Identify Anomalies and Corrective Actions

  Once you have relevant usage data from the first step, here are two questions that need to be asked per each top cost item:

  • Are top AWS usage items justified?
  • What can be done to reduce top usage items?

  The important part in this step is to identify potential corrective actions, which will then be evaluated, prioritized and potentially executed in the following steps.

  Below are some common costly items:

  EC2 Compute

  In my experience, EC2 is almost always one of the top areas in AWS bills. If EC2 BoxUsage [On Demand EC2] is among your top AWS cost items, I recommend looking into the following areas:

  • Find idle EC2 instances. Check CloudWatch metrics for all instances within the affected EC2 instance type [i.e. c5.large, t3.large, etc.] and confirm if they’re being used or not.
  • Find over-provisioned EC2 instances. A common area to optimize is over-provisioned EC2 instances. Sometimes it’s common to assign unnecessary compute capacity to certain application components. Some examples would be having a t3.xlarge for a workload that can be safely handled by a t3.large or t3.medium, or having an EC2 Auto Scaling Group with more instances than required.

  A quick way to do this is to go to the CloudWatch Metrics console and select CPUUtilization for all instances in your account. You’ll see a graph including all EC2 instances in your account and how high [or low] CPU they consume. Any EC2 instances with