Ready to Learn?Ex Libris products all provide open APIs

Tech Blog

 

Serverless Workflows with Azure Logic Apps and Alma Webhooks

Josh Weisman on July 5th, 2017

As more workflows and applications are deployed to the cloud, many organizations are interested in adopting serverless computing. In this model, code is deployed to the cloud provider in the form of discrete functions, and the cloud provider ensures there are ample compute resources to execute the functionality when needed. A previous blog post showed an example of using Lambda, the AWS serverless offering.

Azure, the Microsoft cloud service, offers Azure Logic Apps, a retooled version of their Biztalk integration platform. Logic apps take serverless one step further by providing connectors which allow for integration with many services without the need to write and maintain code. Logic Apps come with many connectors out of the box, and Microsoft is continually adding to the list. And in cases where the built-in connectors aren’t sufficient, Logic Apps integrates with Azure Functions to allow for custom code. While Logic Apps are a relatively new offering as of this writing, it already provides the ingredients necessary to solve many workflow scenarios.

In this post, we'll cover a workflow common in many institutions- manipulating files produced by Alma in publishing or export jobs. The actual manipulations are varied, such as renaming files according to a convention, moving files to a different location, or creating new files with a particular structure. Specifically, this example will acquire the files produced by the job and enumerate them in a new file which will be consumed by another service. The steps required are:

  • Accept a webhook call from Alma when a job ends
  • Validate the signature to ensure the message wasn’t tampered with
  • Confirm the job is valid:
    • Job is of type EXPORT or PUBLISHING
    • Job was completed successfully
  • List files in SFTP location and compare to expected pattern
  • Create the pickup file
  • Respond to the webhook with a success message

These steps are pictured in the diagram below:

Webhook - Azure Logic App Flow

Getting started

Microsoft provides documentation online on getting started with Logic Apps. Logic Apps are moving quickly so it’s worthwhile to review that material- the screenshots provided here might be out of date. This blog post is not meant as a step-by-step tutorial on creating logic apps- there are plenty of those online too. But we will walk through each step and examine the actions and workflow logic required for our scenario.

Behind every logic app workflow is a JSON representation of the app using Microsoft’s workflow definition language schema. The entire workflow definition is available at the Github Gist referenced below. So you can follow along with your own new logic app, or use the provided definition as a starting point and customize it.

Logic App trigger

Logic Apps provide templates for common integration scenarios. In this case we started with the Blank Logic App template. Every logic app needs a trigger which defines how the app is run. We want our app to be triggered by a webhook from Alma, so we’ll select the Request/Response trigger. The designer will give us a URL to call after it’s saved.

The request trigger takes an optional JSON schema. The schema allows the workflow designer to suggest field names to be used later in the workflow. You can use a sample of the webhook job end payload and the logic app designer will create a schema automatically.

Looking back at our flow above, we will skip over the signature validation for now and get to work on confirming that the webhook content can be processed.

Validate the job

We want to perform a few validations here:

  • The webhook is of type JOB_END
  • The job that completed was either an EXPORT or PUBLISHING job
  • The job completed successfully

Our validations are a bit complex, so we need to configure our condition in “advanced mode” and use the JSON below. This syntax defines the conditions we outlined above:

@and(
  equals(triggerBody()?['action'], 'JOB_END'),
  and(
    or(
      equals(triggerBody()['job_instance']['job_info']['category']['value'], 'EXPORT'),
      equals(triggerBody()['job_instance']['job_info']['category']['value'], 'PUBLISHING')
    )
    equals(triggerBody()['job_instance']['status']['value'], 'COMPLETED_SUCCESS')
  )
)

The designer gives us space to define what happens in both the “yes” and the “no” branches. If our condition is not met, we don’t want to do anything, so that’s easy. If our condition is met, we want to continue with our workflow.

Work with the SFTP Connector

The next step in the flow we defined above is to find with the files created by the publishing job. Logic Apps provide an SFTP connector with the actions that we’ll require. To begin, we’ll need to create an SFTP connection.

Add an SFTP connection

In the “yes” area following our condition, we add a “List files in folder” action. The logic app designer will prompt to create a new SFTP connection with all of the details required to connect to our SFTP server.

Webhook - Azure Logic App SFTP Connection

The “list files in folder” action requires a folder to search in, which of course needs to be consistent with the configuration of our publishing job in Alma.

Filter the files according to the job details

The previous action returns an array of all of the files from the specified folder. We need to narrow that list down to the files which were produced by our job. To do that, we’ll use a workflow action called “filter array”. Here we specify the array which was the output of our “list files in folder” action. The file name should contain the external ID of the job instance, which is how Alma names files in publishing jobs.

Webhook - Azure Logic App Filter Array

Flatten the file list

We now have an array limited to the files we’re interested in. The next step is to create an array of only the file names that we can use to create our new pickup file. This step is a bit technical as we need to use the actions available in logic apps to manipulate the array. We want to iterate through each value in the array and select the “name” field. We can do this with a combination of the “For each” and “Compose” actions.

Webhook - Azure Logic App For Each Compose

Create the pickup file in the SFTP location

Finally, we can write our new pickup file by joining the output of the compose action with a new line between each file name.

Webhook - Azure Logic App Create File

Add an Azure Function to Validate Signature

The last element we need for our workflow to be complete is to validate the webhook signature before we begin processing it. Of course there’s no built-in action in Logic Apps to validate our signature so we need a way to write some custom code to perform the validation. Luckily Azure Functions allows just that. We’ve created a function in Node.js which receives the webhook trigger body and the secret. The function performs an HMAC signature using the provided secret and returns the value:

module.exports = function (context, data) {
  var hash = require('crypto').createHmac('SHA256', data.secret)
    .update(JSON.stringify(data.body))
    .digest('base64');
  context.res = {
    body: { signature: hash }
  };
  context.done();
}

Our logic app calls the function and compares the output to the X-Exl-Signature header received with the webhook request. If it matches, we’re validated.

Webhook - Azure Logic App Validate Signature

Putting it together

To put everything together, there’s just a bit more work to do:

  • Configure a webhook integration profile in Alma with the URL provided by our logic app. Alma doesn’t challenge listeners hosted by Azure Logic Apps, so we don’t need to handle the GET requests
  • Test our logic app by executing an export or publishing job in Alma via the UI or API. Alternatively, we can write a small script which sends the webhook content directly to our logic app for easy testing. Such a script is available in the Github Gist referenced below.
  • Monitor the results in the logic app “Runs history” panel in the "Overview" blade. For each run of our logic app, we get all of the parameters, raw inputs and outputs, and status of each step in the workflow.

Not surprisingly the final logic app workflow is similar to the flowchart we presented at the beginning of this post.

Webhook - Azure Logic App Run

Summing up

Azure Logic Apps provide a cost-efficient way to automate tasks without investing in infrastructure or a full blown application. There are many use cases where this approach could be relevant, and this blog post demonstrates one such workflow.

The workflow description and test files are available in this Github Gist.

Github