Banner of Harnessing jq: A Comprehensive Tutorial for COVID-19 Data Retrieval in Your Terminal

jq tutorial : How To Get COVID stats and data from your terminal?

Date: September 2020
Views: 1.48K

This Article is as much about getting the stats of Covid for any country worldwide as it is a tutorial about how to handle and effectually use and manipulate JSON data from the terminal or a shell script using jq the Command Line JSON proccessor

First the API for the COVID stats:

All the covid stats and data for all countries are provided via the following website : https://corona-stats.online/ . However it is not meant to be visited from your regular website. we get the data by issuing a curl command to this website and we can choose either to view it in the terminal in a colorful table or the raw data in JSON format and this is what we are interested in

if we run the following command in our terminal :
curl -s https://corona-stats.online/
the -s argument for curl is just to suppress the output of curl while performing the http request, a silent mode if you will. the result will be like this:

The command above returns all the data about all countries, however we can request the data fo a single country by issuing the following command:
curl -s https://corona-stats.online/Italy/
and we will have the COVID stats for Italy

JQ the command line JSON proccessor

in this second part of my article I will show you just how flexible jq is. we will see that we can manipulate JSON arrays and objects and extract data however we want.

The syntax of the jq command is similar to that of sed or awk:


jq 'body of jq command' jsonfile

the body of the jq command is enclosed by single quotes or double quotes, and single quotes are prefered if we want to use the double quotes at some point. an empty body '' will only show the entire json data without any filter applied to it but it will print the data in a pretty well formated manner.


curl -s https://corona-stats.online/\?format\=json | jq ''

will produce the following lines and a few tousand lines more :


{
  "data": [
    {
      "updated": 1600182380422,
      "country": "USA",
      "countryInfo": {
	"_id": 840,
	"iso2": "US",
	"iso3": "USA",
	"lat": 38,
	"long": -97,
	"flag": "https://disease.sh/assets/img/flags/us.png"
      },
      "cases": 6752122,
      "todayCases": 2833,
      "deaths": 199166,
      "todayDeaths": 166,
      "recovered": 4029477,
      "todayRecovered": 1651,
      "active": 2523479,
      "critical": 14107,
      "casesPerOneMillion": 20374,
      "deathsPerOneMillion": 601,
      "tests": 92932261,
      "testsPerOneMillion": 280419,
      "population": 331404570,
      "continent": "North America",
      "oneCasePerPeople": 49,
      "oneDeathPerPeople": 1664,
      "oneTestPerPeople": 4,
      "activePerOneMillion": 7614.5,
      "recoveredPerOneMillion": 12158.79,
      "criticalPerOneMillion": 42.57,
      "confirmed": 6752122,
      "countryCode": "US"
    },

you could see that the json data is returned from the website in an array named "data" containing objects "dictionaries" with many "key : value" pairs and now we will begin filtering the data to get anything we want. for a better practice we request the data only once and save it to a file say "covid.json".
I should note that the '' empty body is simillar to '.[]' which is an iterator though all the json objects in our file/stream and print them.

The first element of the array


jq '.data[0]' covid.json

notice the "." before the name of our array "data" and of course the 0 index to get the first element , in this case the object containing all the stats of "USA" :


{
  "updated": 1600182979832,
  "country": "USA",
  "countryInfo": {
    "_id": 840,
    "iso2": "US",
    "iso3": "USA",
    "lat": 38,
    "long": -97,
    "flag": "https://disease.sh/assets/img/flags/us.png"
  },
  "cases": 6752122,
  "todayCases": 2833,
  "deaths": 199166,
  "todayDeaths": 166,
  "recovered": 4029477,
  "todayRecovered": 1651,
  "active": 2523479,
  "critical": 14107,
  "casesPerOneMillion": 20374,
  "deathsPerOneMillion": 601,
  "tests": 92932261,
  "testsPerOneMillion": 280419,
  "population": 331404570,
  "continent": "North America",
  "oneCasePerPeople": 49,
  "oneDeathPerPeople": 1664,
  "oneTestPerPeople": 4,
  "activePerOneMillion": 7614.5,
  "recoveredPerOneMillion": 12158.79,
  "criticalPerOneMillion": 42.57,
  "confirmed": 6752122,
  "countryCode": "US"
}

get the value of a key


$ jq '.data[0].cases' covid.json
6752122

$ jq '.data[0]|.country' covid.json
"USA"

we can get the value with the "." then the name of the key, or we can pipe | the outpout of one filter to the next, basically the first filter '.data[0]' will get the the data index 0 , then we pipe it to the second filter '.country' that will get us the country name, if we did not specify the 0 index, the second filter will be applied to all the elements of the array, ie get all the countries. like so


$ jq '.data[]|.country,.cases,.recovered' covid.json
"USA"
6752122
4029477
"India"
4963097
3887371
"Brazil"
4349544
3613184
"Russia"
1073849
884305
...

In the above example, I selected multiple keys separated by comma ","

search for pattern


jq '.data[]|select(.continent|test("Europe"))|.country,.cases' covid.json

The above command will select only the objects that have the continent equal to "Europe": inside the "select" filter/method we apply the "test" filter on the ".continent" key by .continent|test("Europe"). then then once we get the objects that match the test(.continent|test("Europe") filter, we pipe | the results to the next filters to get the ".country" and ".cases", and the result will be as follows:


"Russia"
1073849
"Spain"
593730
"France"
387252
"UK"
371125
"Italy"
289990
"Germany"
263954
"Ukraine"
159702
"Romania"
105298
"Belgium"
94306
"Sweden"
87345
...

The "test" filter can use regular expressions as well : select(.country|test("^I.")) will return data about all countries beginning with the letter "I"

Construction objects and arrays

A slightly different version of the the precedent command will allow us to better organize its output by creating a "key: value" pair out of it:


jq '.data[]|select(.continent|test("Europe"))|{(.country):.cases}' covid.json

enclosing a filter in curly brackets "{}" will make the output an object, and square brackets "[]" will make an array out of the output. we can specify a single constant key name with a string, but to use an already defined key we enclose it in brackets "()".
the result:


{
  "Russia": 1073849
}
{
  "Spain": 593730
}
{
  "France": 387252
}
{
  "UK": 371125
}
{
  "Italy": 289990
}
{
  "Germany": 263954
}
{
  "Ukraine": 159702
}

remove or delete keys:


jq '.data[]|select(.continent|test("Europe"))|del(.countryInfo,.updated)' covid.json

the del(.countryInfo,.updated) filter will remove the keys from the objects of the "data" array

we can also use select() to select a specific key. if we append |select(.Russia) to the command above we will have:


$ jq '.data[]|select(.continent|test("Europe"))|{(.country):.cases}|select(.Russia)' covid.json
{
  "Russia": 1073849
}

the Final example will build an array and access a range of elements in it, to do so we will enclose the entire body of the command in square brackets so that the result will be an array then we append "| .[3:8]" to get elements between 3 and 8 :


jq '[ .data[]
      |select(.continent|test("Europe"))
      |{(.country):.cases}
    ] |.[3:8]
   ' covid.json

[
  {
    "UK": 371125
  },
  {
    "Italy": 289990
  },
  {
    "Germany": 263954
  },
  {
    "Ukraine": 159702
  },
  {
    "Romania": 105298
  }
]

Or even create an associative array :


$ jq '[.data[]|select(.continent|test("Europe"))]|reduce .[] as $e ({}; .[$e.country] = $e.cases)' covid.json
{
  "Russia": 1073849,
  "Spain": 593730,
  "France": 387252,
  "UK": 371125,
  "Italy": 289990,
  "Germany": 263954,
  "Ukraine": 159702,
  "Romania": 105298,
  "Belgium": 94306,
  "Sweden": 87345,
  "Netherlands": 84778,
  "Poland": 75134,
  "Belarus": 74552,
  "Portugal": 65021,
  "Switzerland": 47751,
  "Moldova": 43734,
  "Czechia": 37222,
  "Austria": 34305,
  "Serbia": 32511,
  "Ireland": 31192,
  "Bosnia": 23929,
  "Denmark": 20571,
  "Bulgaria": 18061,
  "Macedonia": 15925,
  "Hungary": 13879,
  "Croatia": 13749,
  "Greece": 13730,
  "Norway": 12330,
  "Albania": 11672,
  "Finland": 8725,
  "Luxembourg": 7244,
  "Montenegro": 6900,
  "Slovakia": 5768,
  "Slovenia": 3831,
  "Lithuania": 3397,
  "Estonia": 2722,
  "Malta": 2454,
  "Iceland": 2174,
  "Latvia": 1482,
  "Andorra": 1438,
  "San Marino": 722,
  "Channel Islands": 633,
  "Faroe Islands": 423,
  "Isle of Man": 339,
  "Gibraltar": 334,
  "Monaco": 177,
  "Liechtenstein": 111,
  "Holy See (Vatican City State)": 12
}