[LINUX] An article that just tries a little HTTP request with the curl command
Introduction
Hello! Long time no see. I want to make a "batch that just sends JSON to the app server when it is executed" [^ 1].
[^ 1]: This article is a continuation of the following article, but if you know "What is Linux or shell?", You can read it from this article as well
When I asked my senior how to implement it, he said, "That's right, curl choice", so I looked up the curl command and created a batch!
I would like to summarize the knowledge I gained at that time.
In this article, I will introduce the basics of curl in the following order.
--What is the curl command? --Basic usage of curl ――I want to try it out quickly to see how it works! (You can try it!) --You can do many things by adding options after curl!
What is the curl command?
The curl command is ** various, as introduced in this article. A command that allows you to * transfer data using a protocol.
It supports various communication protocols such as HTTPS, FTP, Telnet, SMTP, IMAP, and POP3, as well as HTTP GET and POST.
I think that it is included in recent terminals from the beginning and you can use the curl command immediately in the terminal, but if it is not included, Installation is required / 53bca48416030ed2c599).
Environment and version in this article
macOS Catalina curl 7.64.1
$ curl --version
curl 7.64.1 (x86_64-apple-darwin19.0) libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.39.2
Release-Date: 2019-03-27
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets
Just by listening to the version information, the supported protocols and version features are also displayed, which is kind!
This time, we will use only HTTP communication among the protocols!
Basic usage of curl
The basic form of the curl command is as follows.
$ curl {URL}
You can now display the contents of the URL on the console.
Let's try curl Google.
$ curl https://www.google.com/
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="ja"><head><meta content="世…
... Omitted ...
…google.pmc=JSON.parse(pmc);})();</script></body></html>
If you get Google with curl, you'll get this terribly long HTML.
Google's response is long and I don't understand! It doesn't come with a pin! If so, why don't you refer to the homepage of Hiroshi Abe?
$ curl http://abehiroshi.la.coocan.jp/
<html>
<head>
... Omitted ...
<title>Hiroshi Abe's homepage</title>
</head>
... Omitted ...
<body></body>
... Omitted ...
</html>
If you get the simple and beautiful HTML as above, it's working!
(Even if the title is garbled, it's okay because the communication itself is working well's homepage F12 / option + command + I Please press and compare the contents!)
If you see a message like the one below, curl may not be included by default, so try installation.
zsh: command not found: curl
I want to try it out quickly to see how it works! (You can try it!)
Now, I'm going to write a curl command, but don't you care what kind of reply you get when you actually communicate with the app server?
If you just want to give it a quick try, you can take a look at Google and other famous sites.
"What will be returned when an error occurs?" "I'm just getting, but I really want to POST ... [^ 2]"
At times like this, you shouldn't rely on Mr. Abe forever ...
[^ 2]: The difference between GET and POST is easy to understand this Qiita article!
In such a case, httpbin is recommended!
This is a website that returns various responses when you make an HTTP request. You can easily check whether the request is written correctly and whether it can perform HTTP communication properly!
You can also GET and POST respectively by changing the URL a little, and you can arbitrarily set the exit code to 400 or 500. Not limited to curl, this web service is recommended when you want to try out only the code that communicates HTTP / HTTPS.
In this article as well, I will use httpbin.
You can do a lot with options after curl!
Well, until now I've been playing with simple orthodox beautiful and pure curl commands, but in fact you can set ** options ** behind curl!
This makes it possible to make various detailed settings such as using GET and POST properly, setting headers, bodies, and responses in detail, and not displaying error messages.
[Method change] curl -X {request method} “URL”
First, let's try the simplest request method change.
As in Google's example, if you do not add any options, it will be a GET method, but if you want to specify a request method explicitly **, [-X {request method} “URL”](https: // You can change it with curl.haxx.se/docs/manpage.html#-X).
##########
#GET method
# （curl http://httpbin.org/Same result as get)
##########
$ curl -X GET "http://httpbin.org/get"
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "httpbin.org",
"User-Agent": "curl/7.64.1",
"X-Amzn-Trace-Id": "Root=12345678901234567890"
},
"origin": "xxx.xxx.xxx.xxx",
"url": "http://httpbin.org/get"
}
Here is an example of the POST method [^ 3].
[^ 3]: In httpbin, origin etc. is put out in the response so that it is convenient for debugging etc., but since there is personal information, it is hidden in the article!
##########
#POST method
##########
$ curl -X POST "http://httpbin.org/post"
{
"args": {},
"data": "",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Host": "httpbin.org",
"User-Agent": "curl/7.64.1",
"X-Amzn-Trace-Id": "Root=12345678901234567890"
},
"json": null,
"origin": "xxx.xxx.xxx.xxx",
"url": "http://httpbin.org/post"
}
[Header change] curl -H “{parameter}”: “{value}”
Add ** HTTP header ** by adding -H “{parameter}”: “{value}” I can. (From here, I will try it with the POST method!)
$ curl -X POST "http://httpbin.org/post" -H "accept: application/json"
{
"args": {},
"data": "",
"files": {},
"form": {},
"headers": {
"Accept": "application/json",
"Host": "httpbin.org",
"User-Agent": "curl/7.64.1",
"X-Amzn-Trace-Id": "Root=12345678901234567890"
},
"json": null,
"origin": "xxx.xxx.xxx.xxx",
"url": "http://httpbin.org/post"
}
What do you think?
If you compare it carefully with the previous one, you can see that the "Accept" item in "headers" has changed!
In the previous example with only -X, it was "Accept": "* / *", but in this time -X -H, it is "Accept": "application / json"!
[Content change] curl -X POST -d ‘{“parameter ”}: {“value ”}’
Now let's send JSON with the POST method!
By adding "content-type: application / json" to the HTTP header and adding -d option to POST method You can ** request JSON format data with the POST method **.
$ curl -H "content-type: application/json" -X POST -d'{"asa_gohan":"misosiru", "oyatsu":"karl"}' http://httpbin.org/post
{
"args": {},
"data": "{\"asa_gohan\":\"misosiru\", \"oyatsu\":\"karl\"}",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Content-Length": "41",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "curl/7.64.1",
"X-Amzn-Trace-Id": "Root=12345678901234567890"
},
"json": {
"asa_gohan": "misosiru",
"oyatsu": "karl"
},
"origin": "xxx.xxx.xxx.xxx",
"url": "http://httpbin.org/post"
}
Speaking of breakfast is miso soup, and speaking of snacks is karl!
[Detailed communication output] curl -v
-v option is an option to ** output HTTP communication details **.
Since you can see the request header, response header and body, it is convenient to add it when debugging.
$ curl -v -H "content-type: application/json" -X POST -d'{"asa_gohan":"misosiru", "oyatsu":"karl"}' http://httpbin.org/post
* Trying xxx.xxx.xxx.xxx...
* TCP_NODELAY set
* Connected to httpbin.org (xxx.xxx.xxx.xxx) port xx (xx)
> POST /post HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.64.1
> Accept: */*
> content-type: application/json
> Content-Length: 41
>
* upload completely sent off: 41 out of 41 bytes
< HTTP/1.1 200 OK
< Date: Mon, 30 Mar 2020 xx:xx:xx GMT
< Content-Type: application/json
< Content-Length: 486
< Connection: keep-alive
< Server: xxxxxxxx/xxxxxxxx
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
<
{
"args": {},
"data": "{\"asa_gohan\":\"misosiru\", \"oyatsu\":\"karl\"}",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Content-Length": "41",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "curl/7.64.1",
"X-Amzn-Trace-Id": "Root=12345678901234567890"
},
"json": {
"asa_gohan": "misosiru",
"oyatsu": "karl"
},
"origin": "xxx.xxx.xxx.xxx",
"url": "http://httpbin.org/post"
}
* Connection #0 to host httpbin.org left intact
* Closing connection 0
The 0 on this last line is the exit code of the Linux command.
The Linux command returns 0 at normal termination and a positive integer from 1 to 255 at abnormal termination, but curl is HTTP communication. Sometimes, even if a 400 series error or a 500 series error occurs, the exit code will be 0.
This is a type of command that makes you look as if you were successful. I don't know.
[Exit code change] curl -f
It's a mistake concealment man curl who ends normally even if an error occurs, but if you add -f option, it will end abnormally It ends with an exit code of 22! Yay!
Use the Status codes in httpbin.
First, an error without the -f option
##########
# 400
##########
$ curl -X POST "http://httpbin.org/status/400" -H "accept: text/plain"
#Nothing comes out ... Did you succeed? Did you fail? (Actually it fails with 400 errors)
# echo $?The command can display the exit code of the last executed command.
$ echo $?
0
You see ... I'm lying right away ...
However, if you add -f, it will throw an error properly!
##########
# 400
##########
$ curl -f -X POST "http://httpbin.org/status/400" -H "accept: text/plain"
curl: (22) The requested URL returned error: 400 BAD REQUEST
#Yay! Easy to understand!
# echo $?Check it with a command.
$ echo $?
22
Yatter! !! !! !!
Whether it's a 404 or 500, the curl error code and HTTP error code are displayed, so it's very easy to understand!
$ curl -f -X POST "http://httpbin.org/status/404" -H "accept: text/plain"
curl: (22) The requested URL returned error: 404 NOT FOUND
$ curl -f -X POST "http://httpbin.org/status/500" -H "accept: text/plain"
curl: (22) The requested URL returned error: 500 INTERNAL SERVER ERROR
However, curl -f alone is not fail-safe, and authentication errors such as 401 and 407 errors may be ignored [Official document description](https://curl.haxx.se/docs/manpage. There is also html # -f).
Therefore, [Add curl -sS and display error message only when there is an error](https://superuser.com/questions/590099/can-i-make-curl-fail-with-an-exitcode-different-than -0-if-the-http-status-code-i) There seems to be a way too!
in conclusion
curl is difficult ...! I didn't have the concept of an exit code in particular, so I was surprised by all the new things!
Also, if you have any misunderstandings, I would appreciate it if you could let me know ...! ＞ ＜ Thank you.
By the way, the "batch that just sends JSON to the app server when it is executed" has been completed successfully, so I think I'll have a drink at the ramen shop introduced in the previous article!
Thank you for all your comments! I'll eat 500 million cups ~
Recommended Posts