Tech Blog

Using jq to Manipulate JSON in the Shell

Alma APIs support XML and JSON for both the request and response payloads. The choice between these formats is left to the preference of the institution or individual programmer. Sometimes the choice of language or platform informs the choice of format, as in the natural fit between Node.js and JSON.

Most environments offer tooling which supports either format. Many system librarians or developers who write shell scripts to automate processes are familiar with the xmllint tool for XML processing. However there is a tool which is perhaps even more powerful for working with JSON in the shell: jq. The jq website calls it “a lightweight and flexible command-line JSON processor.” Its features include:

  • Parsing & extracting values from JSON
  • Manipulating the value or structure of an existing JSON object
  • Working with JSON arrays
  • Conditional programming
  • And much more

There are many good tutorials and cookbooks on the web for getting started with jq. In this post, we’ll show two examples of using jq with Alma APIs.

Automating ERP Integration with Alma APIs

In a recent blog post we announced the release of APIs which support integration between Alma and ERP financial systems. In this integration, invoices are approved for payment in Alma and then sent to the institutional ERP for processing. Once paid (or rejected), the invoice can be updated in Alma to reflect the action taken in the ERP.

The flowchart below demonstrates the flow along with the relevant APIs called in each step:

In this flow, we retrieve the list of invoices which are in the “Waiting to be sent” status. We then want to loop over the invoices and process each one by sending it to the ERP and updating the status in Alma.

Calling the “Retrieve Invoices” API will return a JSON object with an array of invoice objects:

$ curl -s -H "Authorization: apikey $ALMA_APIKEY" -H "Accept: application/json" https://api-na.hosted.exlibrisgroup.com/almaws/v1/acq/invoices?invoice_workflow_status=Waiting%20to%20be%20Sent&view=brief

{"invoice":[{"number":"PO-32313","vendor":{"value":"B&TA","desc":"BAKER & TAYLORA"},"owner":...

To convert this text into a list of PO line numbers that we can process in a bash loop, we feed the API response into jq. This command tells jq to take the “invoice” property (the array) and transform (map) it into an array of only the “number” field. The ‘[]‘ command returns the contents of the array only (without the wrapping square brackets) and the -r flag removes the surrounding quotes (“raw”).

jq -r '.invoice | map(.number)[]'

The result is a list of PO numbers that can be looped over in bash, such as:

invoices=`curl -s --fail -H "Authorization: apikey $ALMA_APIKEY" -H "Accept: application/json" "https://api-na.hosted.exlibrisgroup.com/almaws/v1/acq/invoices?invoice_workflow_status=Waiting%20to%20be%20Sent&view=brief" | jq -r '.invoice | map(.number)[]'`;

for number in $invoices; do
  # Call ERP and update in Alma
done

In this case, jq allowed us to manipulate the JSON returned by Alma to process the result in a bash loop. The full code for this example can be found in this Github Gist.

Adding members to a set

The “Manage set members” API allows you to add/replace/clear the members of a set. The API accepts a set object with the “members” property populated with the IDs you wish to add to the set. In this example, we use jq to manipulate the result of the “Search users” API and transform it into the payload required by the “Manage set members” API.

First, we call the “Search users” API to retrieve all users with the job category of “GeneralAdministrator”:

/almaws/v1/users?q=job_category~GeneralAdminisrator

Then we pipe the result into jq, and we build a new object with a members.member array, the contents of which is a list of primary_ids from the previous response. We pass in the -c parameter to request “compact” JSON which can be used in a subsequent curl request:

jq -c '{members:{member: (.user | map({id:.primary_id}))}}'

We can then call the “Manage set members” API and set the payload as the output from our jq command:

MEMBERS=`curl -s -H "Authorization: apikey $ALMA_APIKEY" -H "Accept: application/json" "https://api-na.hosted.exlibrisgroup.com/almaws/v1/users?q=job_category~GeneralAdministrator" | jq -c '{members:{member: (.user | map({id:.primary_id}))}}'`
curl -s -o /dev/null -X POST -H "Authorization: apikey $ALMA_APIKEY" -H "Content-type: application/json" -H "Accept: application/json" --data $MEMBERS https://api-na.hosted.exlibrisgroup.com/almaws/v1/conf/sets/$SET_ID?op=replace_members

The jq tool is incredibly powerful- we’ve just scratched the surface of what’s possible in this post. If you do any work with shell scripts, it deserves a place in your toolbox.