How to Send HTTP Headers With cURL

Flipnode on Jun 15 2023


Web scraping involves extracting publicly available data from web pages. When you access a web page using your computer's browser, the browser sends an HTTP request that contains various components, including HTTP headers.

HTTP request headers are vital for web scraping as they carry additional information between web servers and clients. By customizing these headers, you can enhance communication between your software and the targeted website.

In this tutorial, we will explore the process of sending and receiving HTTP headers using cURL, a powerful command-line tool that facilitates data transfer using URL syntax.

Sending HTTP headers

HTTP headers are additional pieces of information that accompany every HTTP request and response. These headers contain essential metadata such as content type, language, and caching instructions. Web developers utilize these headers to ensure their websites function correctly and deliver a seamless user experience.

An HTTP header consists of a name-value pair, separated by a colon (:). The name identifies the type of information being sent, while the value represents the actual data.

Several commonly used HTTP headers include User-Agent, Content-Type, Accept, and Cache-Control.

By default, when using cURL to send an HTTP request, it includes the following headers:

  • Host:
  • User-Agent: curl/7.87.0
  • Accept: /

If desired, you can modify the values of these headers when sending a request.

To send custom HTTP headers with cURL, you can utilize the -H or --header option, followed by the header name and value in the format "Header-Name: value":

curl -H "User-Agent: MyCustomUserAgent"

Sending custom HTTP headers

Custom HTTP headers can fulfill various purposes, including authentication, content negotiation, or adding metadata to your requests.

When utilizing cURL to send custom HTTP headers, you can employ the -H option. Simply specify the header name and value following the format demonstrated in the previous section. Consider the following example:

curl -H "Authorization: Bearer my-access-token"

Sending multiple headers

To include multiple headers when sending requests with cURL, you can utilize the -H option multiple times within a single command. Each -H option should be followed by a distinct header name and its corresponding value:

curl -H "User-Agent: MyCustomUserAgent" -H "Accept: application/json"

In the given example, two headers are being sent:

  • A custom User-Agent header.
  • An Accept header specifying a preference for receiving JSON responses.

Feel free to modify the header names, values, and the URL as per your specific requirements.

Get/show HTTP headers

You have a couple of options with cURL to examine response headers from a web server. By using the -I or --head option, you can send a HEAD request that retrieves only the headers without the actual content:

curl -I
curl --head

If you prefer to see both the response headers and the content in the output, you can make use of the -i or --include option:

curl -i
curl --include

Feel free to execute these commands to observe the response headers or obtain a combined output with headers and content, based on your specific requirements.

Advanced tips for working with cURL headers

Sending Empty Headers:

To send an empty header, simply provide the header name followed by a semicolon:

curl -H "User-Agent:;"

Removing Headers:

If you wish to remove a header that cURL automatically adds by default, you can provide the header name followed by a colon without specifying a value. For instance, to remove the User-Agent header:

curl -H "User-Agent:"

Verbose Mode:

To obtain more detailed information about the request and response, including the headers sent and received, you can enable verbose mode using the -v or --verbose option. This is particularly useful for debugging purposes:

curl -v
curl --verbose

Saving Headers to a File:

If you want to save the response headers to a file for further analysis, you can use the -o or --output option along with the -D or --dump-header option:

curl -D headers.txt -o content.txt

In this example, the response headers will be saved to a file named headers.txt, and the content will be saved to a file named content.txt.

Common use cases for custom headers with cURL

In addition to the aforementioned examples, there are various other use cases where sending custom headers with cURL proves advantageous. Here are some common scenarios where custom headers are particularly useful:

Changing Response Format:

When requesting data from an API or web service, you may need to specify the desired response format, such as JSON or XML. In such cases, you can utilize the Accept header to indicate your preference:

curl -H "Accept: application/json"

Conditional Requests:

To request a resource only if it has been modified since a specific date or doesn't match a specific ETag value, conditional headers like If-Modified-Since or If-None-Match come into play. These headers help minimize bandwidth usage and optimize web scraping performance:

curl -H "If-Modified-Since: Sun, 06 Nov 2022 08:49:37 GMT"


Certain websites or APIs require a Referer header to be included in requests, specifying the source of the request. This can be crucial for tracking purposes or to comply with specific API requirements:

curl -H "Referer:"

Custom Authentication

While the Authorization header is commonly used for authentication, some APIs or services may require a custom header for this purpose. In such cases, you need to include a custom header with the appropriate authentication information:

curl -H "X-Api-Key: my-api-key"

Feel free to adapt and utilize these examples based on your specific needs, allowing you to leverage custom headers effectively with cURL.

Troubleshooting common cURL header issues

When working with cURL and HTTP headers, you may come across common issues. Here are some tips to assist you in troubleshooting problems:

Double-check the Header Syntax

Ensure that the header name and value are correctly formatted, with a colon (":") separating them. Be cautious of any extra spaces or typos in the header name or value.

Verify Header Support

Not all headers are supported by every API or web service. Refer to the documentation of the targeted API or website to confirm that the header you are using is accepted and properly implemented.

Check for Case Sensitivity

While HTTP headers are typically case-insensitive, some APIs or services may expect headers with specific capitalization. If you encounter issues, try adjusting the capitalization of the header name to match the API documentation or examples.

Examine the Response for Errors

When facing issues, it's essential to review the response from the server, which might include status codes or error messages that can aid in diagnosing the problem. Use the -i or --include option with cURL to view the response headers and content together. This can help you identify any issues related to the headers you are sending.

By following these tips, you can effectively troubleshoot and resolve common issues that arise when working with cURL and HTTP headers.


In conclusion, customizing HTTP headers in cURL enables effective interaction with APIs and web services. By modifying headers, you have control over request customization, including response formats, conditional requests, referer inclusion, and custom authentication.

However, be cautious of syntax, header support, case sensitivity, and error examination to troubleshoot potential issues. By mastering custom headers in cURL, you optimize interactions with APIs and enhance web scraping and API integration efforts.

News and updates

Stay up-to-date with the latest web scraping guides and news by subscribing to our newsletter.


Related articles

ProxiesHow to Use Chrome Browser Proxy Settings

Learn how to configure and utilize Chrome browser proxy settings effectively for enhanced privacy and optimized web browsing.

author avatar
8 min read
ScrapersWeb Scraping With RegEx

Regular Expressions (RegEx) are powerful pattern matching tools that allow you to filter and extract specific combinations of data, providing the desired output.

author avatar
5 min read
How to Use DataXPath vs CSS Selectors

Read this article to learn what XPath and CSS selectors are and how to create them. Find out the differences between XPath vs CSS, and know which option to choose.

author avatar
12 min read