Automation for AWS Bills Backup

If you are a DevOps Engineer who is aware of the limitations when it comes to pulling AWS bills via API or through AWS support, this post is certainly for you. In the IT industry, when there are changes in organizational policies or leaders require comprehensive data points to support decisions, the significance of AWS billing data becomes apparent. However, AWS billing data has several restrictions. Despite potentially spending millions of dollars on AWS services, we can only retrieve 37 months of Cost and Utilization Reports via AWS support, and via API, we are limited to the past 12 months.

These limitations can pose a significant challenge when making critical decisions, such as migrating an organization, reducing service usage, or estimating expenditures to facilitate informed decisions. All of these tasks become feasible only when we have access to complete billing reports from the inception of service usage. According to AWS support, the only available method is to manually download each bill from the console. This process can be quite cumbersome, especially when dealing with a decade’s worth of billing data, as it necessitates separate downloads by navigating through each month’s billing page.

What if we can automate all this process so that your computer can downloads all these billing data without you browsing through all the pages manually.

Below is the javascript code that can let us do all the manual tasks and at the end of the day you can get all this bills. Make sure you select the start year and end year as per your requirement. Only pre requisite is for it to work we need a user with billing access and without MFA.

server.js

const puppeteer = require('puppeteer');
// Change AWS account ID 
var accountId = "your AWS account"; 
// Change Username 
var username = "test-user";
var password = "test";

function sleep(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
}

(async () => {
    const browser = await puppeteer.launch({
        headless: false
    }); // Set headless to false for visual debugging
    const page = await browser.newPage();
    // Navigate to the AWS Management Console login page
    await page.goto('https://aws.amazon.com/console/');
    // Wait for and click the "Sign-In" button
    await page.waitForSelector('a[href*="header-signin"]');
    await page.click('a[href*="header-signin"]');
    await page.waitForSelector('input[type="radio"][id="iam_user_radio_button"]');
    await page.click('input[type="radio"][id="iam_user_radio_button"]');
    await page.type('input[type="email"]', accountId);
    await page.click('#next_button');
    await page.waitForSelector('input[id="username"]');
    await page.type('input[id="username"]', username);
    await page.waitForSelector('input[id="password"]');
    await page.type('input[id="password"]', password);
    await sleep(10000);
    await page.click('#signin_button');
    await sleep(10000);

    const startYear = 2016;
    const endYear = 2023;

    for (let year = startYear; year <= endYear; year++) {
        for (let month = 1; month <= 12; month++) {
            const urlvisit = `https://us-east-1.console.aws.amazon.com/billing/home?region=us-east-1#/bills?year=${year}&month=${month}`;
            await page.goto(urlvisit);
            await sleep(3000);
            await page.click('button[data-testid="download-csv-button"]');
            await sleep(3000);
            await page.goto('https://us-east-1.console.aws.amazon.com/console/home');
            await sleep(3000);
        }
    }
})();

You can use the package below by running ‘npm install’; this will install all the dependencies on your local system. To run the JavaScript code above, save it as ‘server.js’ and execute ‘node server.js’.

package.json:

{
  "name": "puppeeter-aws-invoices",
  "version": "1.0.0",
  "description": "billing aws",
  "main": "server.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "node server.js"
  },
  "author": "Hassan",
  "license": "ISC",
  "dependencies": {
    "puppeteer": "^21.1.0",
    "request-promise": "^4.2.6"
  }
}

Please feel free to comment with your findings and other cool things you can achieve with automation. Suggestions are always welcome on the blog.