Ceci est une ancienne révision du document !

Table des matières

Notes JSON et YAML

Notes JSON et YAML

jc	JSON Convert JSONifies the output of many CLI tools and file-types
jq	Command-line JSON processor
json_pp	JSON Pretty Printer
JMESPath	JMESPath is a query language for JSON (utilisé par `json_query` d'Ansible
jp	The `jp` command is a command line interface to JMESPath

Voir :

JSONPath et go_template

Voir Ansible :

Voir aussi :

Monter un fichier JSON comme en système de fichier avec ffs

sudo apt-get install jc
dig www.google.com | jc --dig -p

[
  {
    "id": 17598,
    "opcode": "QUERY",
    "status": "NOERROR",
    "flags": [
      "qr",
      "rd",
      "ra"
    ],
    "query_num": 1,
    "answer_num": 1,
    "authority_num": 0,
    "additional_num": 1,
    "opt_pseudosection": {
      "edns": {
        "version": 0,
        "flags": [],
        "udp": 4096
      }
    },
    "question": {
      "name": "www.google.com.",
      "class": "IN",
      "type": "A"
    },
    "answer": [
      {
        "name": "www.google.com.",
        "class": "IN",
        "type": "A",
        "ttl": 38,
        "data": "142.250.178.132"
      }
    ],
    "query_time": 8,
    "server": "208.67.222.123#53(208.67.222.123) (UDP)",
    "when": "Sun Oct 16 21:54:04 CEST 2022",
    "rcvd": 59,
    "when_epoch": 1665950044,
    "when_epoch_utc": null
  }
]

cat fic.json |python3 -m json.tool |native2ascii -encoding UTF-8 -reverse

Extension Firefox http://jsonview.com/

sudo apt-get install yajl-tools

yajl-tools

json_reformat
json_verify

mongoimport --host localhost --db database --collection collection <fichier.json --jsonArray

Convert JSON to YAML

yq -P sample.json

Ou encore

python -c 'import sys, yaml, json; yaml.safe_dump(json.load(sys.stdin), sys.stdout, default_flow_style=False)' < /opt/stack/keystone/etc/policy.v3cloudsample.json > /etc/keystone/policy.yaml

pip install jsbeautifier
js-beautify file.js

Dico vers JSON ?

cat plop.t |tr \' \" |sed -e 's/None/""/g' |jq .

VIM - Formater le JSON

:%!python -m json.tool

Requête (Query)

Avec curl

curl -k https://aap-controller.local/api/v2/users/27/roles/ -X POST -u user:password --data-raw '{"id":534}'

Avec jq voir : https://hyperpolyglot.org/json

Top level sections

cat single-instance.json |jq 'keys[]'

Exemple wildcard

 
$ curl -s -u "${AAP_USER}:${AAP_PASS}" "https://aap.acme.local/api/v2/job_templates/" | jq '.results[0].id'
1680

$ curl -s -u "${AAP_USER}:${AAP_PASS}" "https://aap.acme.local/api/v2/job_templates/" | jq '.results[].id' | tail -3
1572
1569
2343

jq retours sans guillemet (quote) :

jq -r

docker inspect portainer/portainer-ce |jq '.[0].RepoTags'

ansible -i testsrv, -m setup all |sed -e 's/^.*=>//' |jq -r '.ansible_facts.ansible_mounts[] |.mount'

Filtres

Conversion string integer avec tonumber

Exemple

jq '.blockdevices[].size | tonumber'

Autres

Ansible - Utilisation de boolean en extra vars via JSON

ansible-playbook playbook.yml -i inventory.ini -e '{ "remove_vhost": false }'

YAML

yq	portable command-line YAML processor
yb	YAML Parser in bash

Voir :

https://stackoverflow.com/questions/3790454/how-do-i-break-a-string-in-yaml-over-multiple-lines/21699210#21699210

Contrôle syntaxe :

Yamale

Ce que jq est à json, yq l'est pour le yaml.

yq permet également de convertir des fichiers JSON en YAML

cat .kube/config |egrep -v "\-data"  |yq .

# pip install --user yq
sudo apt-get install yq

openstack image show IMAGE1 -c properties -f yaml |yq '.properties.direct_url'

Mofifier un fichier YAML

avec yq :

En Python et autres :

https://stackoverflow.com/questions/63581308/edit-yaml-file-with-bash

Avec Ansible : https://github.com/kwoodson/ansible-role-yedit

Exemple non-specific tag

configuration: !include config.d/*.yaml

Reférences

vars:
  webapp:
      version: &my_version 1.0
      custom_name:
          - "ToDo_App"
          - *my_version

Exemple Python

import yaml
 
with open("filter.yml", "r") as yamlfile:
    #filters = yaml.load(yamlfile, Loader=yaml.FullLoader)
 
    filters = yaml.safe_load(yamlfile)

Échapper certains caractères

Passer un fichier YAML à une API REST qui fonctionne en JSON

cat inv.yaml |sed -e 's/$/\\n/' -e 's/"/\\"/g' |tr -d '\n'

Exemple YAML vers JSON

inv.yaml

---
server1:
  hosts:
    server1.inf.acme.lan:
  vars:
    SET_LINGERING_LIST:
      - SET_LINGERING_USER: testplop
        SET_LINGERING_ENABLED: "false"
      - SET_LINGERING_USER: testplop
        SET_LINGERING_ENABLED: "true"

type_host:
  hosts:
    server1.inf.acme.lan: {}
  vars:
    type: host

#! /bin/bash
set -euo pipefail
 
YAML="$(cat inv.yaml |sed -e 's/$/\\n/' -e 's/"/\\"/g' |tr -d '\n')"
 
curl -v -k -u user1:'P@ssw0rd' -H 'Content-Type: application/json' -X POST https://awx.acme.fr/api/v2/job_templates/81/launch/ -d '{
"extra_vars": {
"foo1": "bar1",
"foo2": "bar2",
"inventory_content": "'"${YAML}"'"
 
}
}'

Ou avec un heredoc

YAML="$(cat <<'EOF' | sed -e 's/$/\\n/' -e 's/"/\\"/g' |tr -d '\n'
---
server1:
  hosts:
    server1.inf.acme.lan:
  vars:
    SET_LINGERING_LIST:
      - SET_LINGERING_USER: testplop
        SET_LINGERING_ENABLED: "false"
      - SET_LINGERING_USER: testplop
        SET_LINGERING_ENABLED: "true"
 
type_host:
  hosts:
    server1.inf.acme.lan: {}
  vars:
    type: host
EOF
)"

En convertissant le YAML en JSON

En passant un JSON dans un JSON en string

#JS="$(ansible-inventory -i inv.yaml --list)"
JS="$(cat inv.js)"
 
JS="$(echo $JS |sed 's/"/\\"/g')"
URL=''
 
curl -v -k -u user1:'P@ssw0rd' -H 'Content-Type: application/json' -X POST https://awx.acme.fr/api/v2/job_templates/81/launch/ -d '{
"extra_vars": {
"foo1": "bar1",
"foo2": "bar2",
"inventory_content": "'"${JS}"'"
 
}
}'

Exemple de yaml

CRON_HOST_TARGETS:
  all:
    - CRON_FILENAME: crontab_app1
      CRON_USER: app1
      CRONS:
        - DESCRIPTION: "topmem.sh exécuté toutes les heures"
	        SCHED: "0 * * * *"
	        CMDLINE: "~/scripts/topmem.sh >> ~app1/logs/topmem.out 2>&1"

  sc01:
    - CRON_FILENAME: crontab_app1_sc01
      CRON_USER: app1
      CRONS:
        - DESCRIPTION: "archivage de certains fichiers applicatifs"
	        SCHED: "06 03 * * *"
	        CMDLINE: ". ~/scripts/archive_plopsc.sh -l${LINE} >> ~app1/logs/archive_sc_${LINE}.out 2>&1"

        - DESCRIPTION: Performances
	        SCHED: "05 03 * * *"
	        CMDLINE: ". ~/scripts/appexec.sh appenv.sh ${LINE} > /dev/null; ~/scripts/stat_report_run.sh ${LINE} > /dev/null"

Pb

Compile Error - Échapper certains caractères

plop.yml

rec:plop:
  stage: plop
  variables:
    ENV: "RECETTE"

prd:plop:
  stage: plop
  variables:
    ENV: "PRODUCTION"

$ cat plop.yml |yq '.rec:plop'
jq: error: syntax error, unexpected ':', expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.rec:plop    
jq: 1 compile error

Solution

cat plop.yml |yq '.["rec:plop"]'