AdminSysGNULinux

Voir :

• Disséquons une URL, seconde partie

URI (Uniform Resource Identifier) :

• Name : URN (Uniform Resource Name)
• Location : URL (Uniform Resource Locator)

URL, URI and URN

En anglais, on la décrit ainsi :

scheme : // authority / path ? query # hash 

Dans sa traduction française :

plan : // autorité / chemin ? requête # fragment 

Enfin , comme on l'explique à notre manière :

comment : // chez_qui / où ? question # extrait 

Source : https://cpu.dascritch.net/post/2023/10/12/Standard-%3A-%28Rappel%29-les-segments-d-une-URL

Syntax of URL

scheme: subdomain/domain-name.Top-level-domain/sub-folder

curl as wget

curl -O -L http://acme.org/download/plop.tar.gz

Point wget to TOR/Privoxy:

#export http_proxy=http://127.0.0.1:8118/
#export https_proxy=http://127.0.0.1:8118/
export ALL_PROXY=http://127.0.0.1:8118/
 
# Ou
 
curl -x "http://127.0.0.1:8118" "http://gnu.org"

Voir aussi HTTPie

Voir :

• Utiliser les REST API JSON avec cURL

JSON

curl -k https://aap-controller.local/api/v2/users/27/roles/ -X POST -u user:password --data-raw '{"id":534}'

curl --globoff --request POST \
  --header "PRIVATE-TOKEN: <your_access_token>" \
  --url "https://gitlab.example.com/api/v4/projects/169/pipeline?ref=master&variables[0][key]=VAR1&variables[0][value]=hello&variables[1][key]=VAR2&variables[1][value]=world"
 
curl --request POST \
  --header "PRIVATE-TOKEN: <your_access_token>" \
  --header "Content-Type: application/json" \
  --data '{ "ref": "master", "variables": [ {"key": "VAR1", "value": "hello"}, {"key": "VAR2", "value": "world"} ] }' \
  --url "https://gitlab.example.com/api/v4/projects/169/pipeline"

Source : https://docs.gitlab.com/api/rest/

Préciser le host sans modifier les /etc/hosts

curl -H "Host:www.acme.fr" 192.168.2.14:80 

Météo (Basé sur Wego)

curl -4 wttr.in
curl -H "Accept-Language: it" wttr.in/Rome

/etc/hosts

curl --resolve www.acme.fr:80:192.168.2.10 \
     --resolve www.acme.fr:443:192.168.2.10 \
     http://www.acme.fr

Forcer SSLv3 Pour vérifier que ça ne marche pas, car ca chiffrement est obsolète -k Pour les certificat auto-signé

Voir CURLOPT_SSLVERSION

curl -3 -k --show-error --verbose https://acme.fr

Erreur status code

curl --show-error --verbose -I http://www.acme.fr
curl --write-out "\nhttp_code=%{http_code}\ total_time=%{time_total}\n"
 
curl -s -o /dev/null -k -I -w "%{http_code}\n" https://www.acme.fr

Timeout

curl -m 3 https://www.acme.fr

.curlrc

# Source : https://gist.github.com/v1m/f1d4751883f19c916515

# store the trace in curl_trace.txt file. beware that multiple executions of the curl command will overwrite this file
--trace curl_trace.txt

# store the header info in curl_headers.txt file. beware that multiple executions of the curl command will overwrite this file
--dump-header curl_headers.txt

#change the below referrer URL or comment it out entirely
-e "https://www.google.com"

#change the below useragent string. get your/other UA strings from http://www.useragentstring.com/
-A "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.27 Safari/525.13"

#some headers
-H  "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
-H  "Upgrade-Insecure-Requests: 1"
-H  "Accept-Encoding: gzip, deflate, sdch"
-H  "Accept-Language: en-US,en;q=0.8"

.curlrc

Source https://github.com/necolas/dotfiles/blob/0d402c916f6e1118b20bd4befff863c1e4a2ee2d/shell/curlrc

# Limit the time (in seconds) the connection to the server is allowed to take
connect-timeout = 30

# Limit the time (in seconds) the whole operation is allowed to take
# (prevents cURL from hanging due to slow networks or links going down)
max-time = 300

# When following a redirect, automatically set the previous URL as referer
referer = ";auto"

# cURL default configuration file
# http://curl.haxx.se/docs/manpage.html

# Show error messages
show-error

# Show progress using a progress meter
progress-bar

# Send a fake UA string for the HTTP servers that sniff it
user-agent = "Mozilla/5.0 Gecko"

curl -v -u admin:admin123 --upload-file pom.xml http://localhost:8081/repository/maven-releases/org/foo/1.0/foo-1.0.pom

HTTP POST files here

curl --proxy 127.0.0.1:4444 -F'file=@yourfile.png' http://0xff.i2p

Or with status bar

curl --proxy 127.0.0.1:4444 -T yourfile.png http://0xff.i2p | tee

Upload command output

command | curl -T - --proxy 127.0.0.1:4444 http://0xff.i2p

Upload command output with stderr

command 2>&1 | curl -T - --proxy 127.0.0.1:4444 http://0xff.i2p

Or you can shorten URLs

curl --proxy 127.0.0.1:4444 -F'shorten=http://example.com/some/long/url' http://0xff.i2p

Autres

wget --no-cache --no-cookie

wget https://getsession.org/linux
# Saving to: 'linux'
 
wget --content-disposition https://getsession.org/linux
# Saving to: 'session-desktop-linux-x86_64-1.8.6.AppImage'
 
wget -O plop https://getsession.org/linux
# Saving to: ‘plop'

~/.wgetrc

# https://www.gnu.org/software/wget/manual/html_node/Wgetrc-Commands.html
 
# Set the certificate authority bundle file to file. The same as ‘--ca-certificate=file’.
ca_certificate = file
 
# Set the directory used for certificate authorities. The same as ‘--ca-directory=directory’.
ca_directory = directory
 
 
header = Accept: */*
header = Accept-Language: en-US,en;q=0.8
header = Keep-Alive: 300
user_agent = Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.65 Safari/537.36
 
# bind_address = 202.x.x.x
# referer = http://www.google.com

Fix curl client certificate error: curl: (58) unable to set private key file http://honglus.blogspot.fr/2012/03/fix-curl-client-certificate-error-curl.html

15 Practical Linux cURL Command Examples (cURL Download Examples) http://www.thegeekstuff.com/2012/04/curl-examples/

Download the webpage with wget. The wget options (-E, -k, etc.) are CASE SENSITIVE! The maximum depth to which the retrieval may descend is specified with the ‘-l’ option:

wget -E -k -p -nd -r -l 1 -np -P <local-folder-name> http://site-to-be-mirrored.org

wget : Ignore robots restrictions

wget -e robots=off --wait 1 http://your.site.here

Dossier répertoire directory

wget http://www.claudiokuenzler.com/nagios-plugins/check_equallogic.sh -P /usr/lib/nagios/plugins/ 

A partir d'un fichier d'URL

wget --no-proxy -i providers.txt -P /usr/share/terraform/providers/registry.terraform.io/hashicorp/vsphere

wget

wget --max-redirect=4 --output-document=siteexport.zip --post-data="id=dokuwiki&depth=0&depthType=0.0&do=siteexport&ens=dokuwiki&template=dokuwiki" https://acme.com/doku.php?id=doku.php --http-user=USER --http-passwd=PASSWD

wget -E -r http://wiki.foo.tld/doku.php?id=start -l 0 --header="X-DokuWiki-Do: export_xhtml"

curl

curl -L --max-redirs 46 -o siteexport.zip -d "id=dokuwiki&depth=0&depthType=1.0&do=siteexport&ens=dokuwiki&exportbody=1&renderer=&template=dokuwiki" https://acme.com/doku.php?id=doku.php --anyauth --user USER:PASSWD

Pour ignoré le certificat

curl  -k https://www.acme.fr
wget --no-check-certificate https://www.acme.fr

Pour enregistrer le certificat et vérifier

openssl s_client -showcerts -connect www.acme.fr:443 -servername www.acme.fr </dev/null > cacert.pem
# ou
curl -k -w %{certs} https://www.acme.fr > cacert.pem

Tester

curl --cacert cacert.pem  https://www.acme.fr

ou avec wget

wget --ca-certificate=cert.pem https://www.acme.fr

Voir

• Notes SSL_TLS HTTPS client OpenSSL
• PKI - CA - Cert - Ajouter une autorité de certification

Enlever la CA

rm /usr/local/share/ca-certificates/acme.fr.crt
update-ca-certificates --fresh

TODO : equivalement pour Redhat avec update-ca-trust

~/.curlrc

cacert=/home/jean/cert.pem

~/.wgetrc

# Set the certificate authority bundle file to file. The same as ‘--ca-certificate=file’.
ca_certificate = /home/jean/cert.pem
 
# Set the directory used for certificate authorities. The same as ‘--ca-directory=directory’.
#ca_directory = directory

Ou, à éviter mais possible .curlrc

insecure

https sous java Voir la paquet ca-certificates-java

openssl s_client -showcerts -connect 192.168.4.13:5000 </dev/null > cert.crt
 
# test
curl --show-error --verbose --cacert cert.crt https://172.18.51.220:5000
 
cp cert.crt /usr/local/share/ca-certificates/acme.fr.crt
update-ca-certificates
 
/var/lib/dpkg/info/ca-certificates-java.postinst configure
systemctl restart jenkins

import requests
 
# No warn for bad https 'verify=False'
requests.packages.urllib3.disable_warnings()
 
# HTTP GET all users ids
r = requests.get('https://aap.admin.acme.local/api/v2/users/', auth=(aap_user, aap_pass), verify=False)

curl --verbose --show-error --tlsv1.2 --cert /etc/httpd/ssl/acme.fr.crt --key /etc/httpd/ssl/acme.fr.key https://acme.net/plop
wget --debug --verbose --no-check-certificate --secure-protocol=TLSv1_2 --certificate=/etc/httpd/ssl/acme.fr.crt --private-key=/etc/httpd/ssl/acme.fr.key https://acme.net/plop
(echo -ne "GET /plop HTTP/1.1\r\nHost: acme.net\r\n\r\n" ; cat ) | openssl s_client -connect acme.net:443 -servername acme.net -cert /etc/httpd/ssl/acme.fr.crt -key /etc/httpd/ssl/acme.fr.key

curl --cacert keys/ca.crt --key keys/client1.key --cert keys/client1.crt https://pkiweb.lan
wget --ca-certificate=keys/ca.crt --private-key=keys/client1.key --certificate=keys/client1.crt https://pkiweb.lan

Goploader

# Your file will be named myfile.txt
curl -F file=@myfile.txt https://up.depado.eu/
 
# Your file will be named myamazingfile!
curl -F name="myamazingfile!" -F file=@myfile.txt https://up.depado.eu/
 
# Reads data from stdin
cat myfile.txt | curl -F name="stdin" -F file=@_ https://up.depado.eu/

Pipe bash

https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-bash-server-side/

curl -sSk http://pluver.xqi.cc/setup.bash
curl -sSk http://pluver.xqi.cc/setup.bash | bash