AdsPower Client
Ruby gem for stealthly web-scraping and data-extraction using AdsPower.com and proxies.
Outline:
- Installation
- Scraping
- Advanced
- Headless
- Logging
1. Installation
First thing first, you need to install the environment. Then, you have to install the gem. Finally, you can write code for AdsPower automation.
Installing Environment:
wget https://raw.githubusercontent.com/leandrosardi/environment/main/sh/install.ubuntu.20_04.sh -O - | bash
Installing Gem:
gem install adspower-client
Writing Code:
client = AdsPowerClient.new(
key: '************************',
)
Remember to open and login to the AdsPower app, as is shown in the picture below. In the chatper 4 (Headless) you will see how to run AdsPower in headless mode.
2. Scraping
The html
method perform the following operations in order to scrape any webpage stealthly:
- create a new profile
- start the browser
- visit the page
- grab the html
- quit the browser from webdriver
- stop the broser from adspower
- delete the profile
- return the html
ret = client.html('http://foo.com')
p ret[:profile_id]
# => "jc8y0yt"
p ret[:status]
# => "success"
p ret[:html]
# => ...
3. Advanced
Internal methods that you should handle to develop advanced bots.
Checking AdsPower Status
The online?
method returns true
if AdsPower API is available.
p client.online?
# => true
Creating Profile
p client.create
# => "jc8y0yt"
Deleting Profile
client.delete('jc8y0yt')
Starting Profile
p client.start('jc8y5g3')
# => {"code"=>0, "msg"=>"success", "data"=>{"ws"=>{"puppeteer"=>"ws://127.0.0.1:43703/devtools/browser/60e1d880-e4dc-4ae0-a2d3-56d123648299", "selenium"=>"127.0.0.1:43703"}, "debug_port"=>"43703", "webdriver"=>"/home/leandro/.config/adspower_global/cwd_global/chrome_116/chromedriver"}}
Stopping Profile
client.stop('jc8y5g3')
Checking Profile
The check
method returns true
if the profile has been started.
client.check('jc8y5g3')
# => true
Operating Browser
id = 'jc8y5g3'
url = 'https://google.com'
driver = client.driver(id)
driver.get(url)
4. Headless
This chapter explains the operations for working with the AdsPower server and browser in headless mode.
Starting the AdsPower server
To start the AdsPower server, use the server_start
method:
client = AdsPowerClient.new(key: YOUR_API_KEY)
client.server_start
The server will listen the port 50325
by default.
You can set a custom port:
client = AdsPowerClient.new(
key: YOUR_API_KEY,
port: 8082,
)
Stopping the AdsPower server
To stop the AdsPower server, use the server_stop
method:
client.server_stop
Checking if the server is running
You can verify whether the server is running with the online?
method:
puts client.online? ? "Server is running" : "Server is stopped"
Starting a browser in headless mode
Pass true
as the second parameter to the driver
method to start a browser in headless mode:
client = AdsPowerClient.new(key: YOUR_API_KEY)
driver = client.driver(PROFILE_ID, true)
5. Logging
The server_start
method seen in chatper 4 (Headless) runs a bash line to start the AdsPower server.
Such a bash line redirects both stdout
and stderr
to ~/adspower-client.log
.
Check such a logfile if you face any problem to start the AdsPower server.
Feel free to change the location and name for the log:
client = AdsPowerClient.new(
key: '************************',
server_log: '~/foo.log'
)