Every year, the browser inches closer to becoming infrastucture. No longer is the browser merely for humans communicating with one another, but for machines talking to machines.
This article will walk you through the process of developing your first browser automation, whether your interest is a one-and-done script, a background batch job, or a realtime feature for your existing application. I’ll lead you around the pitfalls and give you some resources for going further.
Set up your workspace
If you’re developing this automation as a one-off script, let’s set up a new repo. If you’re developing this automation as part of an existing application, you can skip this step.
But everyone’s got to install the dependencies. Playwright is our automation library of choice. It’s got a great API, great devtools, great performance, and great community support. It’s sure to be the go-to SDK for a long time to come.
That’s all there is to it. Now let’s actually write some code!
Create your first script
For the sake of simplicity, we’ll write our new automation as a standalone script. Later on, we’ll explore how to incorporate it into an existing application.
Open the file you’ve just created. We’ll start by importing the Playwright library and creating a new browser instance.
For a sanity check, let’s run our script and see what happens.
If you see 0 contexts, you’re good to go. If you see an error, make sure you’ve followed the previous steps and try again.
Assuming everything is working, let’s add some functionality…
Navigation, interaction, and scraping
Let’s start by navigating to browsercat.com and visiting the pricing page. We’ll then loop through every value of our dynamic pricing slider and scrape the resulting data.
Run that code and see what happens. You should expect output that looks like the following. (I cut most of the output for brevity.)
100 credits. Free! Your first 1k/month is on us!...6k credits. Save 25% with a right-sized business plan!...200k credits. Save 149% with a right-sized business plan!...1M credits. Unlock our best price at just $0.001/credit!
If you received some errors inside, this is a great opportunity to brush up on your Playwright scripting skills. The following resources will be essential in your journey:
And if you’re still stuck, contact us. We’re always happy to help!
Now let’s incorporate this code into an existing application…
Respond to API requests
Let’s say you want to create a new API that returns realtime BrowserCat pricing data. (Why? I don’t know. Just go with it…) But rather than return the whole list, instead, you want to parameterize the response to return only the data for a given credit amount.
Let’s build an API endpoint that does that.
That’s really all there is to it. Playwright will work in just about any context, from a serverless function to a background job to a realtime websocket. And it’s all the same code.
Deploying your scripts
The one pitfall is when it comes to deploying this script to your provider of choice. While Playwright is a fairly light library, browsers are notorious resource hogs. You really don’t want to have to stuff Chromium into your server. It has very different requirements and it scales at a much different rate. Rather than supporting hundreds of parallel connections, most servers will drop to supporting one or two dozen. And that’s just not going to cut it.
This is where BrowserCat comes in. With BrowserCat, rather than hosting your browsers with your code, instead you connect to our fleet of headless browsers on-demand. (At a very affordable cost.)