Your First Automation

Every year, the browser inches closer to becoming infrastucture. No longer is the browser merely for humans communicating with one another, but for machines talking to machines.

This article will walk you through the process of developing your first browser automation, whether your interest is a one-and-done script, a background batch job, or a realtime feature for your existing application. I’ll lead you around the pitfalls and give you some resources for going further.

Set up your workspace

If you’re developing this automation as a one-off script, let’s set up a new repo. If you’re developing this automation as part of an existing application, you can skip this step.

# Create a new directory
mkdir my-first-automation
cd my-first-automation

# Set up a new node project
npm init -y

# Create a new git repo
git init
git add -A
git commit -m "Initial commit"

# Create a new directory
mkdir my-first-automation
cd my-first-automation

# Set up a new python project
python3 -m venv venv
source venv/bin/activate

# Create a new git repo
git init
git add -A
git commit -m "Initial commit"

But everyone’s got to install the dependencies. Playwright is our automation library of choice. It’s got a great API, great devtools, great performance, and great community support. It’s sure to be the go-to SDK for a long time to come.

# Install node dependencies
npm install playwright 
npm install -D typescript tsx

# Install playwright browsers
npx playwright install

# Install python dependencies
pip install playwright

# Install playwright browsers
playwright install

That’s all there is to it. Now let’s actually write some code!

Create your first script

For the sake of simplicity, we’ll write our new automation as a standalone script. Later on, we’ll explore how to incorporate it into an existing application.

mkdir -p scripts
touch scripts/automation.ts

mkdir -p scripts
touch scripts/automation.py

Open the file you’ve just created. We’ll start by importing the Playwright library and creating a new browser instance.

import * as pw from 'playwright';

async function run() {
  // Create a new browser instance
  const browser = await pw.chromium.launch();

  // Do stuff here...
  console.log(browser.contexts().length, 'contexts');

  // Close the browser when we're done
  await browser.close();
}

run();

import asyncio
from playwright.async_api import async_playwright, Playwright

async def run(playwright: Playwright):
    async with async_playwright() as playwright:
        # Create a new browser instance
        browser = await p.chromium.launch()

        # Do stuff here...
        print(len(browser.contexts), 'contexts')

        # Close the browser when we're done
        await browser.close()

if __name__ == '__main__':
    asyncio.run(run())

For a sanity check, let’s run our script and see what happens.

npx tsx scripts/automation.ts

python scripts/automation.py

If you see 0 contexts, you’re good to go. If you see an error, make sure you’ve followed the previous steps and try again.

Assuming everything is working, let’s add some functionality…

Let’s start by navigating to browsercat.com and visiting the pricing page. We’ll then loop through every value of our dynamic pricing slider and scrape the resulting data.

async function run() {
  // Create a new browser instance
  const browser = await pw.chromium.launch();

  // Navigate to a known url
  const page = await browser.newPage();
  await page.goto('https://www.browsercat.com');

  // Navigate by clicking a link
  const $pricingLink = page.getByRole('link')
    .filter({hasText: 'Pricing'})
    .first();
  await $pricingLink.click();
  await page.waitForSelector('input[type="range"]');

  // Scrape data interactively
  const $range = page.getByRole('slider');
  const $savings = page.locator('#price-savings');
  const min = parseInt(await $range.getAttribute('min') ?? '0');
  const max = parseInt(await $range.getAttribute('max') ?? '0');

  for (let i = min; i <= max; i += 2) {
    // Set the slider value within the page context
    await $range.evaluate((el: HTMLInputElement, i) => {
      el.value = i.toString();
      el.dispatchEvent(new Event('input'));
    }, i);
    const savings = await $savings.textContent();
    console.log(savings?.replace(/ \([^)]+\)/u, '. '));
  }

  // Close the browser when we're done
  await browser.close();
}

import re
import asyncio
from playwright.async_api import async_playwright, Playwright

async def run(playwright: Playwright):
    async with async_playwright() as p:
        # Create a new browser instance
        browser = await p.chromium.launch()

        # Navigate to a known URL
        page = await browser.new_page()
        await page.goto('https://www.browsercat.com')

        # Navigate by clicking a link
        pricing_link = await page.locator('role=link[name="Pricing"]').first
        await pricing_link.click()
        await page.wait_for_selector('input[type="range"]')

        # Scrape data interactively
        range_slider = page.locator('role=slider')
        savings_text = page.locator('#price-savings')
        min_value = int(await range_slider.get_attribute('min') or '0')
        max_value = int(await range_slider.get_attribute('max') or '0')

        for i in range(min_value, max_value + 1, 2):
            # Set the slider value within the page context
            await page.evaluate(f'element => element.value = "{i}" && element.dispatchEvent(new Event("input"))')
            savings = await savings_text.text_content()
            print(re.sub(r" \([^)]+\)", ". ", savings))

        # Close the browser when we're done
        await browser.close()

if __name__ == '__main__':
    asyncio.run(run())

Run that code and see what happens. You should expect output that looks like the following. (I cut most of the output for brevity.)

100 credits. Free! Your first 1k/month is on us!
...
6k credits. Save 25% with a right-sized business plan!
...
200k credits. Save 149% with a right-sized business plan!
...
1M credits. Unlock our best price at just $0.001/credit!

If you received some errors inside, this is a great opportunity to brush up on your Playwright scripting skills. The following resources will be essential in your journey:

And if you’re still stuck, contact us. We’re always happy to help!

Now let’s incorporate this code into an existing application…

Respond to API requests

Let’s say you want to create a new API that returns realtime BrowserCat pricing data. (Why? I don’t know. Just go with it…) But rather than return the whole list, instead, you want to parameterize the response to return only the data for a given credit amount.

Let’s build an API endpoint that does that.

import * as pw from 'playwright';
import Express from 'express';

// Create a new Express app
const app = Express();

// Register your API route
app.get('/api/pricing/:credits', async (req, res) => {
  // Start browser, navigate, etc.

  // Find the correct value on the page
  let result: string | null = null;
  const credits = parseInt(req.params.credits);

  for (let i = min; i <= max; i++) {
    await $range.evaluate((el: HTMLInputElement, i) => {
      el.value = i.toString();
      el.dispatchEvent(new Event('input'));
    }, i);
    const savings = await $savings.textContent() ?? '';

    if (credits = parseInt(savings)) {
      result = savings.replace(/ \([^)]+\)/u, '. ');
      break;
    }
  }

  // Clean up...
  await browser.close();
  return res.send(result);
});

import re
import asyncio
from quart import Quart, request
from playwright.async_api import async_playwright

app = Quart(__name__)

@app.route('/api/pricing/<credits>', methods=['GET'])
async def get_pricing(credits):
    result = None
    credits = int(credits)

    async with async_playwright() as p:
        # Start browser, navigate, etc.

        # Find the correct value on the page
        for i in range(min_value, max_value + 1):
            await page.evaluate(f'element => element.value = "{i}" && element.dispatchEvent(new Event("input"))')
            savings = await savings_text.text_content()

            if credits == int(current_savings_text.split()[0]):
                result = re.sub(r" \([^)]+\)", ". ", savings)
                break

        # Clean up...
        await browser.close()

    return result or "No pricing available for given credit amount"

if __name__ == '__main__':
    app.run()

That’s really all there is to it. Playwright will work in just about any context, from a serverless function to a background job to a realtime websocket. And it’s all the same code.

Deploying your scripts

The one pitfall is when it comes to deploying this script to your provider of choice. While Playwright is a fairly light library, browsers are notorious resource hogs. You really don’t want to have to stuff Chromium into your server. It has very different requirements and it scales at a much different rate. Rather than supporting hundreds of parallel connections, most servers will drop to supporting one or two dozen. And that’s just not going to cut it.

This is where BrowserCat comes in. With BrowserCat, rather than hosting your browsers with your code, instead you connect to our fleet of headless browsers on-demand. (At a very affordable cost.)

Jump to our guide on deployment and scaling for more information.