In this comprehensive tutorial, you will learn how to utilize cURL (also known as curl) with proxy servers in a step-by-step manner. Starting from the installation process, we will cover all the aspects of using proxy servers, including various options for setting up the proxy. It is important to note that this guide is not limited to any specific proxy service, as it can be applied to any proxy server by simply providing the server details and credentials.

While this tutorial delves into technical aspects, readers are assumed to have a basic understanding of proxies. It is particularly beneficial for those starting out with web scraping, as it offers valuable insights and practical knowledge.

cURL is a powerful command line tool that enables users to send and receive data through URLs. To better understand its functionality, let's explore a basic example of how to use curl. Simply open your terminal or command prompt and enter the following command:

By executing this command, you will obtain the HTML code of the webpage and display it on the console. Additionally, you can retrieve document information by adding the -I flag to the command:

This will display the HTTP/1.1 200 OK message, along with the content type and character encoding. For more information on using cURL, check out our link to GitHub

cURL is widely available on Linux distributions and MacOS, and it can now be accessed on Windows 10 as well.

If your Linux distribution doesn't have cURL pre-installed, you can easily install it by running a simple install command. For instance, on Ubuntu, you can open the Terminal and enter the following command:

For additional information on cURL installation, please visit our link to GitHub. In the event that you are utilizing an older version of Windows or desire to install an alternate version, you can download curl from the official download page.

When it comes to using a proxy server, there are a few key pieces of information that you will need to know, regardless of which proxy service you choose to utilize. These include the proxy server address, port, protocol, and any necessary authentication details such as a username and password.

In this tutorial, we will assume that the proxy server address is 127.0.0.1, the port is 1234, the username is "user," and the password is "pwd." It is worth noting that while we will provide examples of various protocols, the steps outlined in this tutorial are not limited to any specific proxy service, meaning that you can apply this knowledge to any proxy server by simply providing the necessary server details and credentials.

If you happen to be connected to a network that utilizes NTLM authentication, you can use the switch –proxy-ntlm with curl to authenticate your connection. Similarly, –proxy-digest can be used to establish a connection via digest authentication. To discover all of the available options, you can run curl –help.

Throughout this tutorial, we will provide numerous examples of how to use cURL with proxy servers, with a particular focus on the most common scenario - HTTP and HTTPS proxy usage. Whether you are a novice or an experienced user, this tutorial will offer valuable insights and practical knowledge that will help you better understand how to use cURL with proxy servers effectively.

As we discussed earlier, curl is a powerful command line tool that enables you to send and receive data using URLs. One of the most useful applications of curl is its ability to work with proxy servers. Proxy servers act as intermediaries between your computer and the internet, allowing you to browse the web anonymously and access content that may be restricted in your region.

If you want to test out proxies, https://httpbin.org/ip is an excellent website to try. This site allows you to view the output of the origin IP address, making it easy to determine if the proxy is working correctly. When using a proxy, the returned IP address should differ from that of your machine and instead display the proxy's IP address.

There are multiple ways to run curl with a proxy command. One of the most common methods is to send proxy details as a command line argument. This involves specifying several pieces of information, including the proxy server address, port, protocol, username (if authentication is required), and password (if authentication is required). Once you have this information, you can input it into the command line along with the URL you wish to access.

It's essential to keep in mind that all command line options, or switches, are case sensitive. For instance, using the switch -f instructs curl to fail silently, while -F denotes a form to be submitted. By understanding these nuances, you can effectively use curl with proxy and achieve your desired results.

Command line argument to set proxy in cURL

To access the list of options for curl, open your terminal and type "curl --help" followed by the Enter key. The output will display a comprehensive list of curl options, including the "-x" or "--proxy" switch used for proxy details. Note that the switch is case-sensitive and the proxy details can be supplied using either "-x" or "--proxy" switch. The following two commands are equivalent:

curl -x "http://user:pwd@127.0.0.1:1234" "http://httpbin.org/ip"

curl --proxy "http://user:pwd@127.0.0.1:1234" "http://httpbin.org/ip"

It's important to note that if there are SSL certificate errors, you should add the "-k" switch to the curl command to allow insecure server connections when using SSL. Another recommended practice is to surround both the proxy URL and target URL with double quotes to handle special characters in the URL. Additionally, it's worth noting that the default protocol for a proxy is HTTP, so both the commands above will have the same effect.

There's another way to use a proxy with curl, which is to set environment variables http_proxy and https_proxy. However, please note that this method is only applicable for MacOS and Linux users. For Windows users, the next section will explain how to use curlrc file. These variable names show the protocol for which the proxies will be used, and not for the protocol used by the proxy server itself. To use the proxy, simply set the http_proxy variable to the http proxy address and https_proxy variable to the https proxy address. To do this, open the terminal and execute these two commands:

export http_proxy="http://user:pwd@127.0.0.1:1234"

export https_proxy="http://user:pwd@127.0.0.1:1234"

After executing these commands, you can run curl as usual:

If you encounter SSL certificate errors, you can add the -k option to ignore them. Please note that these variables apply system-wide. If you don't want this behavior, you can unset the two variables using the following commands:

In the next section, we will explain how to set the default proxy only for curl, without affecting the entire system.

To enable a proxy for curl while leaving other programs unaffected, you can create a curl config file. On Linux and MacOS, navigate to your home directory in the terminal and open the .curlrc file if it already exists. If not, create a new file using the following commands:

Save the file and curl is now ready to use the proxy. When running curl normally, it will automatically read the proxy from the .curlrc file. On Windows, the file is named _curlrc and can be placed in the %APPDATA% directory. To find the path of %APPDATA%, open the command prompt and run this command:

The resulting directory, such as C:\Users<your_user>\AppData\Roaming, can be used to create the _curlrc file and set the proxy in the same way as on Linux and MacOS.

To modify or bypass the proxy settings for a specific request, the -x or --proxy switch can be used as usual, even if the proxy has been set globally or through the .curlrc file.

To bypass the proxy completely for a request, the cURL noproxy command can be used by passing --noproxy followed by an asterisk (*). This instructs cURL to not use the proxy for any URLs.

For situations where many curl requests need to be executed without changing the system-wide proxy settings, the following section provides a solution.

Bonus tip – turning proxies off and on quickly

For advanced users only, here's a tip to create aliases in the .bashrc file for setting and unsetting proxies. If you're not familiar with this file, feel free to skip this section.

To get started, open your .bashrc file using your preferred editor and add these two lines:

alias proxyon="export http_proxy=' http://user:pwd@127.0.0.1:1234';export https_proxy=' http://user:pwd@127.0.0.1:1234'"

alias proxyoff="unset http_proxy;unset https_proxy"

Once you've added these lines, save the .bashrc file and update your shell to read the changes. To do this, run the following command in the terminal:

Now, when you need to turn on the proxy, simply run the "proxyon" command. You can then execute one or more curl commands, and when you're finished, turn off the proxies by running the "proxyoff" command. Here's an example of how to use these aliases:

proxyon

curl "http://httpbin.org/ip"

curl "http://google.com"

proxyoff

The syntax for setting a proxy server using socks protocol is the same as for other protocols:

curl -x "socks5://user:pwd@127.0.0.1:1234" "http://httpbin.org/ip"

You can use socks4://, socks4a://, socks5:// or socks5h:// depending on the socks version.

Alternatively, you can use the –socks5 switch to set a curl socks proxy. To send the username and password, use the –proxy-user switch. Here's an example:

curl --socks5 "127.0.0.1:1234" "http://httpbin.org/ip" --proxy-user user:pwd

You can use –socks4, –socks4a or –socks5 depending on the version. Note that this tip is more suitable for advanced users.

Undoubtedly, cURL is a highly effective automation tool, offering unparalleled proxy support through its command-line interface. Moreover, owing to the compatibility of libcurl with PHP, several web applications leverage it for their web scraping projects, making it an indispensable tool for any web scraper.

How to Use cURL With Proxy?

Web scraping can be a complex field, but thanks to coding languages like Python, building a basic web scraper is relatively straightforward. Python offers a range of useful libraries that simplify the process, and one such library is Beautiful Soup. Beautiful Soup is a popular Python package used for parsing HTML and XML documents, and in this tutorial, we will focus on using it to parse data in Python.

While we recommend checking out our comprehensive article or video tutorial on Python web scraping for beginners, this tutorial specifically concentrates on parsing data using a sample HTML file. It aims to provide a quick introduction to the value offered by Python and Beautiful Soup v4. By following the examples provided, you will gain an understanding of the fundamental principles of parsing HTML data. The examples cover traversing a document for HTML tags, printing the content of tags, finding elements by ID, extracting text from specific tags, and exporting it to a .csv file.

Before diving into the main topic, let's review some fundamental concepts.

Data parsing refers to the process of analyzing and extracting specific information from a structured or semi-structured data source. It involves breaking down the data into smaller components or fields and interpreting its structure according to predefined rules or patterns.

Parsing is commonly used in various fields, including programming, data analysis, and data integration. In the context of programming, data parsing often refers to the extraction and manipulation of data from formats such as XML, JSON, HTML, CSV, or plain text. It involves identifying the data elements, their relationships, and their corresponding values.

During the parsing process, the data is typically converted into a more usable or structured format that can be further processed or analyzed. This may involve transforming the data into a specific data model, such as a dictionary, list, or object, to facilitate easier access and manipulation.

Overall, data parsing is a crucial step in working with data, as it enables the extraction of relevant information and facilitates further analysis, visualization, or integration with other systems.

Beautiful Soup is a Python library designed for parsing HTML and XML documents. It generates a hierarchical parse tree based on predefined criteria, allowing users to easily extract, navigate, search, and manipulate data from HTML. Beautiful Soup is particularly useful for web scraping tasks. It supports both Python 2.7 and Python 3, making it accessible to a wide range of programmers. With its convenient features, Beautiful Soup can significantly reduce development time for various projects.

Before proceeding with this tutorial, it's essential to have a Python programming environment set up on your machine. For the purpose of this tutorial, we'll assume you're using PyCharm as it provides a convenient option, even for those who are less experienced with Python. However, you can use any IDE of your choice.

If you're using Windows, ensure that you select the "PATH installation" checkbox while installing Python. This will add the Python executables to the default Windows Command Prompt search, allowing you to use commands like "pip" or "python" without specifying the executable's directory. This simplifies the process and makes it more convenient.

Additionally, you need to have Beautiful Soup installed on your system. Regardless of the operating system, you can easily install the latest version of Beautiful Soup by executing the following command in the terminal:

If you're using Windows, it's recommended to run the terminal as an administrator to ensure a smooth installation process.

Lastly, since we'll be working with a sample HTML file, it's beneficial to have some familiarity with the structure of HTML.

To demonstrate the main methods of how Beautiful Soup parses data, we will use a sample HTML file. Although this file is simpler than a typical modern website, it will suffice for the purposes of this tutorial.

<!DOCTYPE html>

<html>
    <head>
        <title>What is a Proxy?</title>
        <meta charset="utf-8">
    </head>

    <body>
        <h2>Proxy types</h2>

        <p>
        There are many different ways to categorize proxies. However, two of the most popular types are residential and data center proxies. Here is a list of the most common types.
        </p>

        <ul id="proxytypes">
            <li>Residential proxies</li>
            <li>Datacenter proxies</li>
            <li>Shared proxies</li>
            <li>Semi-dedicated proxies</li>
            <li>Private proxies</li>
        </ul>

    </body>
</html>

To use this file in PyCharm, simply copy the contents and save it with the .html extension in the directory of your PyCharm project.

Next, open PyCharm and right-click in the project area. Navigate to "New" -> "Python File". Congratulations! You now have a new playground to work with.

To extract a list of all the tags used in our sample HTML file using Beautiful Soup, we can follow these steps:

from bs4 import BeautifulSoup

with open('index.html', 'r') as f:
    contents = f.read()

    soup = BeautifulSoup(contents, features="html.parser")

    for child in soup.descendants:
        if child.name:
            print(child.name)

After running this code, you should see the following output:

html
head
title
meta
body
h2
p
ul
li
li
li
li
li

Now, let's take a closer look at what each line of the code does:

This line imports the Beautiful Soup library.

with open('index.html', 'r') as f:
    contents = f.read()

Here, we open the sample HTML file and read its contents.

soup = BeautifulSoup(contents, features="html.parser")

This line creates a BeautifulSoup object and uses Python's built-in HTML parser to parse the HTML contents.

for child in soup.descendants:
    if child.name:
        print(child.name)

The loop iterates over the descendants of the HTML file and checks if each element has a name (i.e., it is an HTML tag). If it does, the tag name is printed.

This code demonstrates how Beautiful Soup can traverse an HTML file and extract HTML tags. Later in the tutorial, we will explore additional functionalities, such as exporting the results to a .csv file.

from bs4 import BeautifulSoup

with open('index.html', 'r') as f:
    contents = f.read()

    soup = BeautifulSoup(contents, features="html.parser")

    print(soup.h2)
    print(soup.p)
    print(soup.li)

This code will output the HTML tags along with their full content in the specified order. The output should look like this:

<h2>Proxy types</h2>
<p>

          There are many different ways to categorize proxies.  However, two of the most popular types are residential and data center proxies. Here is a list of the most common types.
        </p>
<li>Residential proxies</li>

If you want to remove the HTML tags and print only the text, you can use the text attribute of the tag, like this:

In our case, it will give the following output:

Keep in mind that these methods only print the first instance of the specified tag. To find elements by ID or filter elements using specific criteria, we can use the find_all method, which we will explore in the following sections.

Using Beautiful Soup to find elements by ID

To find elements by ID in Beautiful Soup, there are two similar ways you can use:

print(soup.find('ul', attrs={'id': 'proxytypes'}))

Both of these methods will output the same result in the Python Console:

<ul id="proxytypes">
<li>Residential proxies</li>
<li>Datacenter proxies</li>
<li>Shared proxies</li>
<li>Semi-dedicated proxies</li>
<li>Private proxies</li>
</ul>

These methods allow you to locate elements by their ID attribute and retrieve the corresponding HTML code.

Finding all specified tags and extracting text

The find_all method in Beautiful Soup is a powerful tool for extracting specific data from an HTML file. It allows you to filter data based on various criteria. However, for this tutorial, we'll keep it simple and use it to find all items in our list and print their text content only:

from bs4 import BeautifulSoup

with open('index.html', 'r') as f:
    contents = f.read()

    soup = BeautifulSoup(contents, features="html.parser")

    for tag in soup.find_all('li'):
        print(tag.text)

Residential proxies
Datacenter proxies
Shared proxies
Semi-dedicated proxies
Private proxies

Congratulations! You now have a basic understanding of how Beautiful Soup can be used to parse data. It's important to note that the examples in this tutorial serve as introductory material, and real-world web scraping and data parsing scenarios can be more complex. For a more in-depth exploration of Beautiful Soup, I recommend referring to its documentation, which is an excellent resource to further enhance your knowledge.

A practical application of Beautiful Soup is exporting data to a .csv file for further analysis. Although this topic is beyond the scope of this tutorial, let's briefly explore how it can be accomplished.

First, you'll need to install the pandas library, which helps in creating structured data in Python. You can easily install it using the following command:

Next, add the following line at the beginning of your code to import the library:

Moving forward, let's add some lines of code that export the list we extracted earlier to a .csv file. Here's the updated code:

from bs4 import BeautifulSoup
import pandas as pd

with open('index.html', 'r') as f:
    contents = f.read()

    soup = BeautifulSoup(contents, features="html.parser")
    results = soup.find_all('li')

    df = pd.DataFrame({'Names': results})
    df.to_csv('names.csv', index=False, encoding='utf-8')

This line finds all instances of the <li> tag and stores them in the results object.

df = pd.DataFrame({'Names': results})
df.to_csv('names.csv', index=False, encoding='utf-8')

In these lines, the pandas library comes into play. It stores our results in a table-like structure called a DataFrame and exports it to a .csv file.

If everything went well, a new file named names.csv should appear in your Python project directory. Inside the file, you should see a table with the list of proxy types. Congratulations! You now not only understand how to extract data from an HTML file but also how to programmatically export it to a new file.

Beautiful Soup is an incredibly useful HTML parser that offers a wide range of capabilities. It has a relatively gentle learning curve, allowing you to quickly understand how to navigate, search, and modify the parse tree. Additionally, when combined with libraries like pandas, you gain even more power to manipulate and analyze the data. This combination opens up endless possibilities for collecting and analyzing data in various use cases. Beautiful Soup, along with its integration with other libraries, provides a powerful package for data collection and analysis.

Using Python and Beautiful Soup to Parse Data: Intro Tutorial

In the realm of web scraping, getting started can be straightforward, but sometimes it can be challenging, which is why you're here seeking guidance. Python offers a user-friendly approach, serving as an ideal starting point with its object-oriented nature. Compared to other programming languages, Python's classes and objects are remarkably intuitive to work with. Moreover, Python boasts a wide range of libraries that simplify the process of building web scraping tools.

In this tutorial, we will provide a comprehensive overview of web scraping using Python. We'll walk you through the essential steps to create a basic application that extracts text-based data from web pages, saves it to a file, and organizes the output based on specified parameters. Towards the end, we'll touch upon more advanced features and offer suggestions for their implementation. By following the outlined steps in this tutorial, you'll gain a solid understanding of web scraping using Python.

Web scraping refers to the automated extraction of publicly available data. With the help of a webpage scraper, vast amounts of data can be extracted from target websites within seconds.

This Python web scraping tutorial is compatible with all operating systems. The installation process for Python and development environments may vary slightly, but the core concepts remain consistent across platforms.

This web scraping tutorial is based on Python version 3.4 or higher. Specifically, we used version 3.8.3, but any version from 3.4 onwards should work perfectly fine.

For Windows installations, it is important to select the "PATH installation" option during the Python installation process. This ensures that the executables are added to the default Windows Command Prompt search, allowing you to use commands like "pip" or "python" without manually specifying the executable's directory (e.g., C:/tools/python/.../python.exe). If you have already installed Python without selecting this option, you can simply rerun the installation and choose the "modify" option. On the second screen, select "Add to environment variables" to enable the PATH installation.

Python offers a wide range of powerful libraries specifically designed for web scraping. With thousands of Python projects and over 300,000 projects on PyPI (Python Package Index), the availability of web scraping libraries is extensive. Here are some notable Python web scraping libraries to consider:

These libraries provide various functionalities and features that make web scraping tasks more efficient and effective. Whether you need to make HTTP requests, parse HTML documents, extract data, or automate browser interactions, these libraries offer the tools you need to accomplish your web scraping goals.

The Requests library is a valuable tool for web scraping, particularly when it comes to making HTTP requests. It simplifies the process by reducing the amount of code required, making it easier to understand and debug. To install the library, you can use the following command:

With the Requests library, you can easily send HTTP GET and POST requests. For instance, the get() function allows you to send an HTTP GET request:

response = requests.get('https://flipnode.io/')

If you need to post form data, you can use the post() method and provide the data as a dictionary:

form_data = {'key1': 'value1', 'key2': 'value2'}

response = requests.post('https://flipnode.io/', data=form_data)

print(response.text)

The Requests library also offers convenient support for proxies that require authentication. You can specify the proxy information as follows:

proxies = {'http': 'http://user:password@pr.flipnode.io:7777'}

response = requests.get('https://ip.flipnode.io/', proxies=proxies)

print(response.text)

However, it's important to note that the Requests library does not parse the extracted HTML data into a more readable format for analysis. Additionally, it may not be suitable for scraping websites that heavily rely on JavaScript for their content.

Beautiful Soup is a powerful Python library that specializes in parsing HTML and extracting data from it, even if the markup is invalid. However, it should be noted that Beautiful Soup is primarily focused on parsing and does not have the capability to request HTML data from web servers. As a result, it is commonly used in conjunction with the Python Requests library.

To demonstrate its usage, let's go through an example using Beautiful Soup along with the html.parser module from the Python Standard Library.

First, we use the Requests library to retrieve the HTML content of a webpage:

import requests

url = 'https://flipnode.io/blog'

response = requests.get(url)

Next, we import Beautiful Soup and create a BeautifulSoup object to parse the HTML:

from bs4 import BeautifulSoup

soup = BeautifulSoup(response.text, 'html.parser')

print(soup.title)

This will print the title element of the webpage:

Beautiful Soup simplifies the process of navigating, searching, and modifying the parse tree, making it particularly useful for developers, even those with limited experience. For example, we can use the findAll() method to print all the blog titles on the page. In this case, the blog titles are contained within <h3>, <h4>, and <h5> elements. We can specify these elements to the findAll() method:

blog_titles = soup.findAll(['h3', 'h4', 'h5'])

for title in blog_titles:

    print(title.text)

Beautiful Soup also provides convenient support for CSS selectors. Developers familiar with CSS selectors can utilize them directly instead of using the find() or find_all() methods. Here's an example that removes the "Most popular articles" title, which is contained within an <h5> element, from the blog_titles object using CSS selectors:

blog_titles = soup.select('h3, h4, h5:not(:contains("Most popular articles"))')

for title in blog_titles:

    print(title.text)

Aside from its ability to parse broken HTML, Beautiful Soup offers various functions, including the ability to detect page encoding, which enhances the accuracy of the extracted data.

Furthermore, it can be easily configured with just a few lines of code to extract custom publicly available data or identify specific data types.

lxml is a high-performance parsing library that supports both HTML and XML files. It is known for its speed, power, and user-friendly interface, making it a popular choice for data extraction tasks, especially when dealing with large datasets. However, compared to Beautiful Soup, lxml's parsing capabilities can be affected by poorly structured HTML, causing some limitations.

To install the lxml library, you can use the following pip command in the terminal:

Once installed, you can import the html module from lxml to work with HTML content. Assuming you have already obtained the HTML string using the Requests library as explained in the previous section, you can create an HTML tree using the fromstring method:

from lxml import html

tree = html.fromstring(response.text)

With the tree object in place, you can now utilize XPath to query and extract specific elements. Building on the example from earlier, if you want to retrieve the blog titles excluding the "Most popular articles" title, you can use the following XPath expression:

blog_titles = tree.xpath('//h3 | //h4 | //h5[not(contains(text(), "Most popular articles"))]')

for title in blog_titles:

    print(title.text)

In the above code, the XPath expression //h3 | //h4 | //h5[not(contains(text(), "Most popular articles"))] selects all <h3>, <h4>, and <h5> elements that do not contain the text "Most popular articles".

If you are interested in learning more about using the lxml library and incorporating it into your web scraping workflows, we recommend checking out our comprehensive lxml tutorial, which provides detailed guidance and insights to enhance your expertise.

When it comes to websites built with JavaScript, traditional Python libraries struggle to extract data from dynamic web pages. However, Selenium comes to the rescue and thrives in this scenario.

Selenium is an open-source browser automation tool, often used for test automation, that can execute JavaScript and render web pages just like a regular browser. This makes it a powerful tool for web scraping, as it can handle JavaScript-driven content that standard web crawlers cannot. Selenium has gained widespread popularity among developers due to its versatility.

To get started with Selenium, you need three components:

Web Browser - Selenium supports popular browsers like Chrome, Edge, Firefox, and Safari.

Browser Driver - You can find the appropriate driver for your chosen browser on the Selenium website.

The Selenium package itself - Install it using the following command:

Once installed, you can import the necessary class for your chosen browser and create an object of that class. Here's an example using the Chrome browser:

from selenium.webdriver import Chrome

from selenium.webdriver.common.by import By

driver = Chrome(executable_path='/path/to/driver')

You can then load any web page using the get() method:

Selenium allows you to extract elements using CSS selectors or XPath. The following example prints all the blog titles using CSS selectors, excluding the "Most popular articles" title:

blog_titles = driver.find_elements(By.CSS_SELECTOR, 'h3, h4, h5')

for title in blog_titles:

   if title.text != 'Most popular articles':

       print(title.text)

Finally, don't forget to close the browser when you're done:

It's important to note that Selenium's execution of JavaScript and dynamic rendering can slow down the scraping process, making it less suitable for large-scale data extraction. However, if you're working on smaller-scale projects or prioritize accuracy over speed, Selenium is an excellent choice with its ability to mimic human behavior and handle JavaScript-driven content.

In web scraping, a browser is necessary to connect to the target URL. For beginners, it is highly recommended to use a regular browser (non-headless) during testing. This allows you to observe how the written code interacts with the application, making troubleshooting and debugging simpler and providing a better understanding of the entire process.

Headless browsers can be utilized for more advanced and complex tasks at a later stage. In this web scraping tutorial, we will be using the Chrome web browser, although the process is almost identical with Firefox.

To begin, use your preferred search engine to find the appropriate "webdriver for Chrome" (or Firefox) based on your browser's current version. Once found, download the corresponding webdriver.

If necessary, select the required package, download it, and unzip it. Copy the executable file of the webdriver to a readily accessible directory. We will verify if everything is set up correctly in the subsequent steps.

Finding a cozy place for our Python web scraper

Before we dive into the programming part of this web scraping tutorial, there is one final step to take: setting up a suitable coding environment. You have various options to choose from, ranging from a basic text editor where you can create a *.py file and write your code directly, to a comprehensive Integrated Development Environment (IDE).

If you already have Visual Studio Code installed, opting for this IDE would be the easiest choice. However, for newcomers, I highly recommend using PyCharm due to its user-friendly interface and minimal learning curve. For the remainder of this web scraping tutorial, we will assume that PyCharm is being used.

In PyCharm, you can create a new Python file by right-clicking in the project area and selecting "New -> Python File". Be sure to give your file a descriptive name! This will be your workspace for writing and running the web scraping code.

Now it's time to make use of the Python packages we installed earlier. Let's proceed with the following code:

import pandas as pd

from bs4 import BeautifulSoup

from selenium import webdriver

Note that in PyCharm, these import statements might appear in grey, as the IDE automatically marks unused libraries. However, I recommend not accepting its suggestion to remove unused libraries for now.

Next, we need to define our browser based on the webdriver we selected earlier in the "WebDriver and browsers" section. Depending on your choice, you can use one of the following lines of code:

driver = webdriver.Chrome(executable_path='c:\path\to\windows\webdriver\executable.exe')

driver = webdriver.Firefox(executable_path='/nix/path/to/webdriver/executable')

Make sure to replace the paths with the actual paths to your webdriver executables. This will initialize the browser and prepare it for web scraping tasks.

Before running our first test, let's choose a suitable URL. Since this web scraping tutorial focuses on creating a basic application, it's recommended to select a simple target URL. Keep the following considerations in mind:

Avoid scraping data hidden within JavaScript elements, as they often require specific actions to display the desired data. Scraping such elements requires more advanced Python techniques and logic.

Image scraping is not covered in this tutorial, as Selenium allows direct downloading of images.

Always ensure that you are scraping public data and not violating any third-party rights. Additionally, check the website's robots.txt file for any scraping guidelines.

Once you have chosen a landing page to visit, input the URL as a parameter in the driver.get('URL') method. Remember to include the connection protocol (e.g., "http://" or "https://") in the URL.

driver.get('https://your.url/here?yes=brilliant')

To perform a test run, click the green arrow at the bottom left of your coding environment or right-click the coding environment and select "Run".

If you encounter an error stating that a file is missing, double-check that the path provided in the webdriver.* matches the location of the webdriver executable. If you receive a version mismatch error, make sure to redownload the correct webdriver executable.

Python Web Scraping Tutorial: Step-By-Step

Welcome to this lxml Python tutorial, where we will dive into the powerful lxml library. Throughout this tutorial, we will cover the fundamental aspects of working with XML documents and progress to handling both XML and HTML documents. We will then bring everything together and explore how to extract data using lxml. Each step will be accompanied by practical examples in Python, ensuring a hands-on learning experience.

This tutorial assumes that you have a basic understanding of Python programming. Additionally, familiarity with XML and HTML concepts is beneficial. If you are familiar with XML attributes, you will be able to grasp the concepts presented in this article.

The code snippets in this tutorial are written in Python 3, but with minimal modifications, they can also be adapted to Python 2.

The lxml library is renowned for its exceptional performance and extensive features when it comes to XML and HTML processing in Python. It serves as a convenient interface to the underlying C libraries, libxml2 and libxslt, combining the efficiency of the C implementation with the ease of use provided by Python.

With the Python lxml library, you can effortlessly create, parse, and query XML and HTML documents. It is a crucial dependency for various complex packages, such as Scrapy, enabling advanced web scraping capabilities.

To install the lxml library, the recommended approach is to obtain it from the Python Package Index (PyPI). On Linux (debian-based) systems, you can use the following command:

Alternatively, you can utilize the pip package manager, which is compatible with Windows, Mac, and Linux. Execute the following command:

For Windows users running Python 3, you can simply use the command pip install lxml.

To represent XML or XML-compliant HTML, we can visualize it as a tree structure consisting of a root and various branches. Each branch can further contain additional branches. In the lxml library, these branches and the root are represented as elements.

Here's an example of a simple XML document:

If an HTML document follows XML compliance, it adheres to the same tree structure concept. However, it's important to note that not all HTML documents are XML compliant. For instance, an HTML document with a <br> tag without a corresponding closing tag is still considered valid HTML but not valid XML. In the later part of this tutorial, we will explore how to handle such cases. For now, let's focus on XML-compliant HTML.

To create an XML document using Python lxml, you need to start by importing the etree module from lxml:

In lxml, every XML document starts with a root element. To create the root element, you can use the Element type. This type serves as a flexible container object for storing hierarchical data. In this Python lxml example, we aim to create an XML-compliant HTML, so the root element should have the name "html":

Similarly, an HTML document consists of a head and a body. We can create these elements using the Element type as well:

head = etree.Element("head")
body = etree.Element("body")

To establish parent-child relationships, we can use the append() method:

The XML document can be serialized and printed to the terminal using the tostring() function. This function requires the root element as a mandatory argument. Optionally, you can set pretty_print to True to make the output more readable. It's important to note that the tostring() function returns bytes, which can be converted to a string by calling decode():

print(etree.tostring(root, pretty_print=True).decode())

To improve code readability and simplicity, you can utilize the SubElement type in lxml. It provides a convenient way to create child elements without the need to separately create an Element object and then use the append() function. SubElement takes two arguments: the parent node and the element name. By using SubElement, you can replace the following two lines of code with a single line:

body = etree.Element("body")
root.append(body)

This accomplishes the same result of creating the body element as a child of the root element in a more concise manner.

Setting text and attributes in lxml is straightforward and can be done using the text and set methods provided by the Element and SubElement objects. Here are some examples:

Recent articles

Sign up for our newsletter

News and updates