How to Send GET Requests With cURL

Flipnode on Jun 09 2023

cURL is not only a command-line tool but also a versatile library that enables seamless data transfer between servers using different protocols. Beyond its capability to retrieve public data, including HTML, text, JSON, images, and more, cURL excels at managing sessions and cookies. It seamlessly integrates with APIs and can be effortlessly incorporated into shell commands for automation purposes. These diverse use cases make cURL an excellent choice for web scraping tasks.

In this article, we will guide you through the process of sending GET requests using cURL, enabling you to efficiently extract public data from various sources.

What is a GET request?

A GET request is an HTTP request used to retrieve data from a specific web page. For example, when you type in a URL into your browser and click enter, your browser sends a GET request to the server hosting that URL, requesting to retrieve the associated HTML code.

While other HTTP request methods can also result in a web server sending you HTML, those methods serve different purposes. For instance, HTTP POST is utilized for posting data, such as when you submit a form containing your username and password. Although you may still see a web page, the POST request method is primarily for sending the form data to the server. If you're interested in learning about sending POST requests, you can refer to our blog post on POST requests with cURL.

On the contrary, a GET request does not send any data. It is solely used to request a page or other resources, such as images. Here's a simple example of a cURL GET request:

curl http://flipnode.io

The result of this request will be the HTML code returned by the web server.

Steps of sending GET requests with cURL

Now that we have the fundamentals covered, let's dive into the process of sending a GET request using cURL. In this tutorial, we will use httpbin.org, a straightforward HTTP request and response service, as our example. We will explore various aspects of cURL, including:

• Simple cURL GET requests
• Sending a GET request with parameters
• Retrieving GET HTTP headers
• Receiving responses in JSON format
• Following redirects
• Sending cookies with a GET request

By the end of this tutorial, you will have a solid understanding of how to utilize cURL for performing GET requests and handling different scenarios effectively. Let's get started!

Step 1 - Simple cURL GET requests

If the default request method is GET, you can omit the --request option and simply send a GET request using the following command:

curl http://httpbin.org/get

Step 2 - GET request with parameters

Sending a GET Request with Parameters

When you need to include additional data within the URL of your GET request, cURL offers two useful options: -d and -G.

Keep in mind that using -d without -G will result in a POST request. Similarly, if you specify the -X option with the value GET, the data provided with -d will be ignored.

To properly send parameters with a GET request, you should use -G along with -d. Take a look at the example command below:

curl -G -d "param1=value1" -d "param2=value2" http://httpbin.org/get

In the above command, 'param1' and 'param2' represent the parameter keys, while 'value1' and 'value2' are their respective values. You can use the -d option multiple times to send multiple parameters.

Alternatively, you can include the GET parameters directly in the URL, like this:

curl 'http://httpbin.org/get?param1=value1&param2=value2'

Step 3 - GET HTTP Headers

Retrieving HTTP Headers in a GET Request

HTTP headers enable the exchange of additional information between the client and server during an HTTP request. To obtain the HTTP headers along with the response body, you can utilize the -i or --include option in the curl GET command:

curl -i http://httpbin.org/headers

By executing this command, you will retrieve the HTTP response headers, including details such as the server, date, content type, and content length. These headers provide valuable insights into the nature and specifics of the response data.

It's worth noting that the long parameter for fetching response headers is --head:

curl --head http://httpbin.org/headers

Step 4 - Retrieve Data in JSON Format

Working with JSON Data in cURL

JSON has emerged as a widely adopted standard for data exchange in the modern web development ecosystem. When communicating with APIs, it is essential to request data in JSON format. You can instruct cURL to expect the response in JSON format by using the -H option followed by "Accept: application/json":

curl -H "Accept: application/json" http://httpbin.org/get

However, it is important to note that specifying "Accept: application/json" does not guarantee that the response will be returned in JSON format. The actual format of the response depends on whether the website supports returning a JSON response.

Step 5 - Follow Redirects

In some cases, the URL you request may redirect to another URL. By default, cURL does not automatically follow these redirects. However, you can explicitly instruct cURL to follow redirects by using the -L or --location option:

curl -L http://httpbin.org/redirect-to?url=http://httpbin.org/get

With the -L option, cURL will follow the redirection and provide you with the response from the final URL. This enables you to seamlessly handle scenarios where the initial URL redirects to a different location.

Step 6 - Send Cookies with a GET Request

In certain situations, it may be necessary to include cookies with your GET request, particularly when dealing with websites that rely on user sessions or track user behavior. To accomplish this, you can utilize the -b or --cookie option followed by the cookie's name and value:

curl -b "username=John" http://httpbin.org/cookies

By specifying the cookie using the -b option, cURL will include it in the GET request. This allows you to provide the necessary cookie information when interacting with websites that require it for authentication or personalized functionality.

cURL GET Request Arguments

To learn how to send GET requests with cURL, here are some key arguments and their corresponding options and descriptions:

• -I or --head: Retrieves HTTP headers only.
• -i or --include: Includes the HTTP response headers in the output.
• -A or --user-agent: Specifies the User-Agent string to send to the server.
• -X or --request: Specifies the request type to use (GET, POST, PUT, DELETE, etc.).
• -L or --location: Follows redirects in the server's response.
• -b or --cookie: Sends cookies from a string or file. Format of the string should be NAME=VALUE; another=anotherval.
• -v or --verbose: Provides more information (debug info).
• -s or --silent: Silent mode. Don't output anything.
• -o or --output: Writes output to a file instead of stdout.
• -H or --header: Passes a custom header to the server. To get JSON, use -H "Accept: application/json".

These options allow you to customize your GET requests with cURL based on your specific requirements.

Conclusion

We trust that this guide has provided you with a solid understanding of cURL GET requests. However, like any skill, practice is essential for mastery. We encourage you to dedicate some time to practicing sending requests with cURL, and before long, you will gain confidence and proficiency in using this powerful tool.