Web Automations and Proxies¶

When developing automations, it is often useful to use proxies to avoid being blocked.

Proxies can be either authenticated or anonymous.

Anonymous proxies are the easiest to use and widely supported by browsers.

Authenticated proxies are also supported by browsers but unfortunately the authentication process is not exposed via command-line arguments as to avoid exposure of the credentials.

On this guide we will walk you through the steps to use anonymous and authenticated proxies.

Anonymous Proxy¶

Via the browser capabilities, you can set the proxy settings for the browser.

With your proxy settings ready such as the proxy server address and port, you can use the following code snippet to set up the proxy on your Web Bot.

PythonJava

# Import for the WebBot
from botcity.web import WebBot, Browser, By

# Uncomment the line for your desired browser below
# from botcity.web.browsers.firefox import default_capabilities
# from botcity.web.browsers.chrome import default_capabilities
# from botcity.web.browsers.edge import default_capabilities
# from botcity.web.browsers.ie import default_capabilities

# Modify the proxy settings here
PROXY = "186.251.64.10:8085"

# Get the default capabilities for the browser
capabilities = default_capabilities()

# Adjust as needed for the proxy
capabilities['proxy'] = {
    "httpProxy": PROXY,
    "sslProxy": PROXY,
    "proxyType": "MANUAL",
}

bot = WebBot()

# Configure proxy in capabilities
bot.capabilities = capabilities

# Opens browser using the proxy defined above
bot.browse("https://whatsmyip.org")

// Uncomment the line for your desired browser below
// import dev.botcity.framework.web.browsers.FirefoxConfig;
// import dev.botcity.framework.web.browsers.ChromeConfig;
// import dev.botcity.framework.web.browsers.EdgeConfig;

// Modify the proxy settings here
String proxy = "186.251.64.10:8085";

// Get the default capabilities for the browser
FirefoxConfig firefoxConfig = new FirefoxConfig();
MutableCapabilities capabilities = firefoxConfig.defaultCapabilities();

// Setting proxy configuration
// Adjust as needed for the proxy
Map<String, String> proxyConfig = new HashMap<String, String>();
proxyConfig.put("httpProxy", proxy);
proxyConfig.put("sslProxy", proxy);
proxyConfig.put("proxyType", "MANUAL");

capabilities.setCapability("proxy", proxyConfig);
setCapabilities(capabilities);

browse("https://whatsmyip.org");

Authenticated Proxy¶

As mentioned above, authenticated proxies are supported by browsers but the authentication process is not exposed via code as to avoid exposure of the credentials.

Here are some alternatives to use an authenticated proxy:

Configure the proxy on your operating system.
Use Firefox with the extension AutoAuth.

Roadmap

Our team will update this guide as we find more alternatives to use an authenticated proxy.

Using Firefox with AutoAuth¶

AutoAuth is a simple Firefox Addon based on WebExtension that submits HTTP authentication credentials automatically.

The first time you open a web page, AutoAuth will prompt you for the credentials.

The login page contains a form with the following fields:

Field	Description	DOM ID
Username	The username to use	username
Password	The password to use	password
Submit	The submit button	submit

In order to use this extension with your Web Bot, you will need to download the xpi file from the AutoAuth repository releases page and save it in a location that is accessible to your Web Bot.

Tip

You can add the xpi file to your bot's resources folder and it will make your code portable across different runtime environments.

With the xpi file in place, you can use the following code snippet to set up the proxy on your Web Bot.

PythonJava

# Import for the WebBot
from botcity.web import WebBot, Browser, By

# Uncomment the line for your desired browser below
# from botcity.web.browsers.firefox import default_capabilities
# from botcity.web.browsers.chrome import default_capabilities
# from botcity.web.browsers.edge import default_capabilities
# from botcity.web.browsers.ie import default_capabilities

# Modify the proxy settings here
PROXY = "186.251.64.10:8085"

USERNAME = "proxyuser"
PASSWORD = "s3cr3tp4ssw0rd"

# Get the default capabilities for the browser
capabilities = default_capabilities()

# Adjust as needed for the proxy
capabilities['proxy'] = {
    "httpProxy": PROXY,
    "sslProxy": PROXY,
    "proxyType": "MANUAL",
}

bot = WebBot()

# Setting to use Firefox
bot.browser = Browser.FIREFOX

# Configure proxy in capabilities
bot.capabilities = capabilities

# Assuming AutoAuth XPI is in the `resources` folder of your bot
autoauth_xpi = bot.get_resource_abspath("autoauth-3.1.1-an.fx.xpi")

# Add the AutoAuth extension to the browser
bot.install_firefox_extension(autoauth_xpi)

bot.browse("https://whatsmyip.org")

# Fill in the login form
username = bot.find_element("login", By.ID)
username.send_keys(USERNAME)

password = bot.find_element("password", By.ID)
password.send_keys(PASSWORD)

# Submit the form
submit = bot.find_element("submit", By.ID)
submit.click()

// Uncomment the line for your desired browser below
// import dev.botcity.framework.web.browsers.FirefoxConfig;
// import dev.botcity.framework.web.browsers.ChromeConfig;
// import dev.botcity.framework.web.browsers.EdgeConfig;

// Modify the proxy settings here
String proxy = "186.251.64.10:8085";

String username = "proxyuser";
String password = "s3cr3tp4ssw0rd";

// Get the default capabilities for the browser
FirefoxConfig firefoxConfig = new FirefoxConfig();
MutableCapabilities capabilities = firefoxConfig.defaultCapabilities();

// Setting proxy configuration
// Adjust as needed for the proxy
Map<String, String> proxyConfig = new HashMap<String, String>();
proxyConfig.put("httpProxy", proxy);
proxyConfig.put("sslProxy", proxy);
proxyConfig.put("proxyType", "MANUAL");

capabilities.setCapability("proxy", proxyConfig);
setCapabilities(capabilities);

// Assuming AutoAuth XPI is in the `resources` folder of your bot
String autoAuthXPI = "./src/resources/autoauth-3.1.1-an.fx.xpi";

// Add the AutoAuth extension to the browser
installFirefoxExtension(autoAuthXPI);

browse("https://whatsmyip.org");

// Fill in the login form
WebElement usernameElement = findElement(By.id("login"));
usernameElement.sendKeys(username);

WebElement passwordElement = findElement(By.id("password"));
passwordElement.sendKeys(password);

// Submit the form
WebElement submit = findElement(By.id("submit"));
submit.click();

Important

Once you have authenticated with AutoAuth, you will not be prompted for credentials again.