Ask HN: AWS CloudWatch Fraud?

7 points by uLogMicheal 10 months ago | 8 comments
Over a year ago I migrated one of my companies away from AWS and deleted all EC2 and Kubernetes instances. I recently noticed that regardless of these cancellations, AWS has consistently billed for "CloudWatch".

The service team sent back a bunch of general information about costs and instructions to remove a "Metric Filter" which from my follow-up I can confirm does not exist on my account.

The detailed billing shows charges like this across US regions:

$0.00 per request - first 1,000,000 requests 239,785 Requests

$0.01 per 1,000 metrics requested using GetMetricData API - US West (Northern California) 295,570 Metrics

Do others have experience with miscellaneous charges that are impossible to cancel? Any tips for dealing with support on this to get answers beyond copy/paste support articles?

  • cddotdotslash 10 months ago
    Based on "GetMetricData", you're not paying for services, but rather something with access to your account is making API requests to CloudWatch. Do you have any third-party monitoring tools (Splunk, Datadog, etc.) in use? Can you check your IAM portal to see if you have any users/roles with recent access?
    • Reubend 10 months ago
      AWS definitely doesn't do any "fraudulent" billing for CloudWatch, but there are sometimes very complicated pricing schemes for AWS products, and people often setup complicated systems that they don't fully understand the cost implications of.

      In your case, I'd guess that some part of your system, or perhaps some integration that you added to your account, is making API calls without you being aware of it.

      • ashitvora 10 months ago
        Not sure about the fraud but recently we had a very heavy bill from AWS (6x of our usual AWS bill).

        After much investigation I realised that one of my dev has setup complicated stuff using some Terraform config he found on Github.

        I feel that AWS has very bad UX.

        • re-thc 10 months ago
          Just in case: the thing people often forget is to change regions when checking if something exists. Did you check us-west-1 when checking if this exists in CloudWatch?

          Maybe try the aws cli to list and delete?

          • uLogMicheal 10 months ago
            Checked all regions, no services on the AWS side. It seems maybe our old Datadog was still spamming the GetMetricData API by default after cancellation. No logs are collected but they still seem to query the API for every service enabled by default.
            • brodouevencode 10 months ago
              Yes, that's how the DD->CW integration works.

              Side note: even if you shut down every service in the account, if something outside of AWS is connected to it, like Datadog, you will still incur charges. I'd recommend deleting the account if you can.

              • uLogMicheal 10 months ago
                Thanks for the insight! I'm sure it costs Datadog a lot to continue to query connected accounts even after the user has cancelled the Datadog service. I reported to them, hopefully they take measures to fix. It would also be nice if integrations could be blocked or de-authorized from the AWS side, but I guess we can't ask for too much in 2024.