How To Scrape a Web Page Using PHP

This tutorial will teach you how to collect data from a website or webpage programmatically using PHP, a process known as web scraping or crawling.

To scrape a web page using PHP, we typically use libraries like cURL or Guzzle to make HTTP requests and retrieve the HTML content of the webpage. Then you can use tools like DOMDocument or XPath to parse and extract the desired information from the HTML.

In this tutorial we will see example with php-curl to scrape a website. If you don’t know what is PHP CUrl and how to use then checkout this – PHP CURL Tutorial.

1. Install the php-curl:

Install the php-curl on your system. If you are a xampp user you don’t need to be install, by default this is comes with it.

sudo apt-get install php-curl

2. Basic Web Scraping using PHP cURL:

Here’s a basic example of how you could do it using cURL:

<?php
// URL of the webpage you want to scrape
$url = 'https://example.com';

// Initialize cURL session
$curl = curl_init();

// Set cURL options
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);

// Execute cURL request
$response = curl_exec($curl);

// Check for errors
if ($response === false) {
    echo 'Error: ' . curl_error($curl);
    // Handle error accordingly
} else {
    // Close cURL session
    curl_close($curl);

    // Load the HTML into a DOMDocument
    $dom = new DOMDocument();
    @$dom->loadHTML($response); // Suppress warnings

    // Use XPath to query the DOMDocument for specific elements
    $xpath = new DOMXPath($dom);

    // For example, to extract all <a> tags
    $links = $xpath->query('//a');

    // Loop through the links and output their href attribute
    foreach ($links as $link) {
        echo $link->getAttribute('href') . "<br>";
    }
}

Output:

https://www.iana.org/domains/example<br>

In this code I have used DOMDocument and XPath to parse and extract the all the links from the scraped HTML.

DOMDocument and XPath are both components of PHP that are commonly used together for parsing and navigating XML or HTML documents.

DOMDocument loads the document into memory and represents it as a structured tree, while XPath enables precise selection and extraction of specific elements or data from the document based on criteria defined by XPath expressions.

Remember, before scraping any website, make sure to review its terms of service and robots.txt file to ensure that scraping is allowed. Unauthorized scraping may violate the website’s terms of service or even legal regulations.

3. Enable Cookies When Scraping using cURL:

Sometimes a website needs to enable cookies to access a web page. In this case you can you can use the CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR options to handle cookiest. Here’s an example:

// Enable cookies by specifying a file to read cookies from
curl_setopt($curl, CURLOPT_COOKIEFILE, '/path/to/cookie.txt');
curl_setopt($curl, CURLOPT_COOKIEJAR, '/path/to/cookie.txt');