Deep Search (Legacy)

General Purpose

Deep Search (previously called Third Node) is a Primo plug-in that enables customers to replace or enhance the standard Primo search engine with their own search engine functionality using the standard Primo services and view.

This plug-in allows sites to create multiple deep search plugins, each plugin performing searches against a different external search engine (Google, Ebsco, …).

For example, the searches against Primo Central are enabled in Primo by implementing a deep search plugin that works against the Primo Central API.

Plugin Flow

The following are the usual steps to implement in the plugin:

  1. Translate query
    You will receive a query in the Lucene syntax.
    You should translate the query to a query which is supported by the remote engine you are working against.
  2. Transform result
    The result returned to Primo should be in a specific format (described below).
    You will need to transform the result returned from the remote engine you are working against to a Primo result.

For both of the steps above it is recommended to use some kind a mappings file.

Your first option is to create your own file and store it as part of the JAR file you will be creating (see below).

An additional option, is to keep these mappings in the “Deep Search Plugin parameters” mapping table as parameters sent to the plugin, see explanation on how to configure this table in the next section.

Integration Steps

1.     Copy the following JAR file into your working environment:

primo_library-common-4.5.0.jar
primo_common-infra-4.5.0.jar
jaguar-client-4.5.0.jar
jaguar-infra-4.5.0.jar
xbean.jar (or xmlbeans-<version>.jar depends on the Primo version you are on)

The location of these JAR files depends on the Primo version you have.

 

a) up to Primo 4.5:
$primo_dev/ng/primo/home/system/thirdparty/openserver/server/search/deploy/primo_library-app.ear/lib/
b) from Primo 4.5 and on:
$primo_dev/ng/primo/home/system/tomcat/search/webapps/primo_library#libweb/WEB-INF/lib/

 

2.     Extend the AbstractDeepSearch class.

See explanation below.

NOTE: The class you are implementing MUST reside in the following package: com.exlibris.primo.thirdnode.thirdnodeimpl

3.     Wrap your implementation in a JAR file
4.     Copy your JAR file to $primo_dev/ng/primo/home/system/search/conf/thirdNodeImpl
5.     Configure your plugin in the following mapping tables:
1.     Deep-Search Plugins
2.     Deep Search Plugin Parameters

 

The Deep-Search Plugins table is used to configure your plugin.
In the following screenshot you can see an example for deep-search plugins defined in the installation level:
Under the “Plugin ID” column set the ID of the plugin (note for users of primo pre April 2015 release: This is the ID as you’d configure it in the old thirdnode-config.xml file)
Under the “Plugin Class” put the exact and full name of the plugin class that you have implemented in step #2.
Note (again), the class name must reside in the pakcage: com.exlibris.primo.thirdnode.thirdnodeimpl. The value must only be the class name without the package name, as seen in the example above.
Once you want the plugin to be enabled make sure to check the checkbox in the “Enabled” column on the left hand side of the relevant plugin row.
Under the “Plugin Display Name” column, place the text you want to be presented in the Primo View Wizard.

 

The Deep Search Plugin Parameters table is used to configure your plugins’ parameters.
In the following screenshot you can see an example for a configuration of this mapping table:
Please note that in order to be able to configure parameters for a plugin, the plugin itself needs to be enabled.
Check the “Enabled” checkbox if you want the parameter to be functional.
In the “Param name” column either select the parameter name from the drop down-list or type it in manually, if it’s not in the list.
Note: We used “datalist” to implement the append-able list of parameters – it’s an HTML5 element and it’s supported in Chrome, Firefox and IE10+ (not supported in Safari and IE9-).
In the “Param value” column enter the Parameter value. If the value should be encrypted (such as password or WCkey) use the Param value – Encrypted” column.
Make sure to select the correct and relevant plugin from the drop-down in the “Plugin” column.

 

Below you will find list of parameters which are supported by all implementations.
You can use these parameters to configure any of the adaptors.
NameExplanation
primo_rankIf set to true, the ranking will be created by Primo and not by the plugin. If set to false, the ranking will be taken from the plugin results. Be aware that if set to true, the ranking is determined only based on the subset of results returned to Primo. If the plugin returned 20 records, the ranking will be based on those 20 records only. It is recommended not to set this value to true and be able to return the ranking from the original engine.
primo_facetsIf set to true, the facets will be built by Primo. If set to false it is assumed the result set includes a facets section. If set to true, we assume each returned PNX record includes a facet section, and the result set facets section will be based on the facet section of the PNX.
primo_highlightingWhether to perform highlighting on the returned results.
warmupIf set to true, Primo will send a certain amount of queries to your plugin before adding it to Primo’s search. This is in order to warm up your plugin. For example you might be using some kind of a data type, that needs to get filled up.
cacheEnable result caching.
NOTE: When disabled, every full display initiates a new search.You can configure the amount of results stored in the cache. Do not configure a large number. 10 – 20 is a logical number to use.
Important notes for users of Primo pre April 2015 release:
  1. The code table “Third Node Adaptors” is not relevant anymore and changing its content will not affect anything.
  2. The file “thirdnode-config.xml” is not relevant anymore and changing its content will not affect anything.

Deploying to Front-End(s)

After completing filling up the relevant mapping tables and saving them, you’ll need to deploy the new Deep-Search configuration to the Front-End(s):

Go to “Deploy All” screen (you can find it under “Deploy & Utilities” menu, or directly from the home screen in the Back Office).

Check only the “Deep Search Configuration”  checkbox and hit the “Deploy” button.

Once all processes are finished, everything is ready and you can start using the new deep-search class.

Implementing the AbstractDeepSearch class

You should extend the AbstractDeepSearch class and implement the following 3 functions:

  1. init()
  2. search()
  3. countFacets()

init()

This method is called by Primo once, the first time Primo uses the plugin.

This function receives a Map holding the parameters configured in the thirdnode-config.xml.

For example, you can get a parameter value by:

public void init(Map parameters) {

        String url = (String) parameters.get("URL");

}

search()

This method is called each time a search is performed from a Primo view that has Deep Search enabled.

This function receives a Lucene query, and is responsible for translating the query to the needed syntax supported by the remote engine you are working against.

At the end, the plugin needs to transform the response received from the remote engine to a Primo response.

The Primo response is described by the following  jaguar XSD.

The following are the parameters this function receives:

ParameterTypeDescription
vidStringThe Primo view ID
queryStringThe query requested. The query syntax sent is in the Lucene format:http://lucene.apache.org/core/3_1_0/queryparsersyntax.html
fromintNumber of first record requested
tointNumber of last record requested
authorizationMapSee example below, of how to use this map:

End user authorization parameters:
• institution
• group (user group)
• ip (end user IP)
• pds_handle (SSO ID)
• session_id (Unique HTTP session ID)
• on_campus (Boolean)
• User groups with restricted access to PC/DS (Boolean) (Primo V4.8 and higher)

For example:
String sessionId = (String)parameters.get("session_id")
sortStringSort by field. Possible values are:

1) stitle – perform a Title sort.
2) scdate – perform a Date sort.
3) screator – perform a Author sort.
4) popularity – Perform a Popularity sort.

The order of the sort is hardcoded and is defined in the following mapping table:

 

Allow searches only for signed in users

Use the following code snippet to check if user is signed in or on campus
(the map sent to the function is the map received in the search function):

public boolean isSearchAllowed(Map<String, String[]> authorization) {
        Object pdsHandle = authorization.get(ThirdNodeConst.PDS_HANDLE);
        boolean signedInUser = (pdsHandle != null && !"".equals(pdsHandle));

        Boolean isOnCampus = (Boolean)authorization.get(ThirdNodeConst.ON_OFF_CAMPUS);

        return signedInUser || isOnCampus;
}

countFacets()

For blended search queries, Primo uses a specific algorithm that determines which facets to show on the results page.

In order to present a count number next to each facet value, Primo has to “ask” each relevant adapter, for the count of each of the facets.

This function receives the query and all the facets to be displayed, and should return the count for each facet.

Usually this will not be possible by most search engines as they might not have this ability.

The meaning of leaving this function unimplemented, is wrong counts for the facets, meaning when clicking on a facet,
the number of results returned will probably not match the count displayed next to the facet.

The following are the parameters this function receives:

ParameterTypeDescription
queryStringThe query requested. The query syntax sent is in the Lucene format:http://lucene.apache.org/core/3_1_0/queryparsersyntax.html
sortStringSee the search() function for details
facetsMapThe key is the facet group name and the value is an array of all the values for this group
For example: [creator]->[James E.], [Art F, ]…
authorizationMapSee the search() function for details
searchTokenStringdeprecated

The outcome of this function should be something like:

<sear:FACETLIST ACCURATE_COUNTERS="true">
       <sear:FACET NAME="creator" COUNT="19">
         ...
         <sear:FACET_VALUES KEY="Lund, Karen" VALUE="5"/>
         <sear:FACET_VALUES KEY="Webmaster Id" VALUE="1"/>
         <sear:FACET_VALUES KEY="Biro Corneliu" VALUE="4"/>
         <sear:FACET_VALUES KEY="Miller, Matt" VALUE="23"/>
       </sear:FACET>
       <sear:FACET NAME="lang" COUNT="47">
         ...
         <sear:FACET_VALUES KEY="ger" VALUE="303457"/>
         <sear:FACET_VALUES KEY="chi" VALUE="381872"/>
         <sear:FACET_VALUES KEY="dan" VALUE="1787"/>
         <sear:FACET_VALUES KEY="jpn" VALUE="118031"/>
         <sear:FACET_VALUES KEY="baq" VALUE="7"/>
         <sear:FACET_VALUES KEY="cze" VALUE="3352"/>
       </sear:FACET>
      
  </sear:FACETLIST>

Plugin Example:

YahooThirdNode XML Example
In this example the YahooThirdNode plug-in utilizes the following methods to implement a Yahoo search adaptor for Primo.
#init reads the parameters from the thirdnode-config.xml file and sets the client.
#buildQuery, which is called from the search method, converts the search query in Primo format to Yahoo format.
#map2pnx, which is called from the search method, converts the query results in Yahoo format to Primo PNX format.
#search performs the Yahoo search and returns the results in Primo PNX format.

public class YahooThirdNode extends AbstractDeepSearch {

    SearchClient client = null;

Init: get parameters if needed:

public void init(Map parameters) {

    String id = (String) parameters.get("APPLICATION_ID");

    client = new SearchClient(id);

}

Build Query: Conversion of a query from Primo format to Yahoo format.

private String buildQuery (String query) {

    String yahooQuery = query;

    for (int i = 0; i < SearchFields.SEARCH_FIELDS.length; i++) {

        Matcher matcher = Pattern.compile(SearchFields.SEARCH_FIELDS[i]+":

            \\((.*?)\\)").matcher(metalibQuery);

        if (matcher.find()) {

            // remove field name (title:(test)->(test))

            yahooQuery = matcher.replaceAll("$1");

        }

    }

    return yahooQuery;

}

Map2PNX: Conversion of a query result from Yahoo format to Primo PNX format.

private PrimoNMBibDocument map2pnx(WebSearchResult yResult) throws Exception {

    PrimoNMBibDocument primoRecord =  PrimoNMBibDocument.Factory.newInstance();

    RecordType rt = primoRecord.addNewPrimoNMBib().addNewRecord();

    DisplayType dt = rt.addNewDisplay();

    dt.addType("book");

    dt.addTitle(yResult.getTitle());

    dt.addDescription(yResult.getSummary());

    dt.addType(yResult.getMimeType());

    FacetsType ft = rt.addNewFacets();

    ft.addLanguage("eng");

    ft.addLanguage("fre");


    return primoRecord;

}

Searchfunction complete:

public PrimoResult search(String id, String query, int from, int to,

Map authorization, String sort) throws Exception {

    // Search Start Time

    long start = System.currentTimeMillis();


    // Convert Primo query format to yahoo query Format

    String yahooQuery = buildQuery(query);

    // Run search in Yahoo search engine

    WebSearchRequest request = new WebSearchRequest(yahooQuery);

    request.setStart(BigInteger.valueOf(from))

    WebSearchResults yahooResults = client.webSearch(request);

    // Search End Time

    long end = System.currentTimeMillis();

    // Prepare Primo results structure

    PrimoResult primoResult = PrimoResult.Factory.newInstance();

    SEGMENTSDocument.SEGMENTS returnSegment =

    SEGMENTSDocument.Factory.newInstance().addNewSEGMENTS();

    RESULTDocument.RESULT results =

        returnSegment.addNewJAGROOT().addNewRESULT();

    DOCSETDocument.DOCSET docSet = results.addNewDOCSET();

    // calculates search duration

    docSet.setHITTIME((end-start));

    // Set total hits

    docSet.setTOTALHITS(yahooResults.getTotalResultsAvailable().intValue());

    //set number of first result

    docSet.setFIRSTHIT(new Integer(from).toString());

    //set number of last result

    docSet.setLASTHIT(new Integer(from +

    yahooResults.listResults().length - 1).toString());

    for (int i = 0; i < yahooResults.listResults().length; i++) {

        // Convert Yahoo result to Primo PNX format

        PrimoNMBibDocument pnx = (map2pnx(yahooResults.listResults()[i]));

        DOCDocument.DOC hit = docSet.addNewDOC();

        hit.setPrimoNMBib(pnx.getPrimoNMBib());

        hit.setNO(String.valueOf(from + i));

    }

    primoResult.setSEGMENTS(returnSegment);

    return primoResult;

}

Restricted user group

From Primo 4.8 and higher, a new mapping table has been added (User groups with restricted access to PC/DS) which lets you configure groups of restricted users per adapter.

If you want your Deep Search to take this into account, you can add to the code snippet above (“Allow searches only for signed in users”), the following:

public boolean isSearchAllowed(Map<String, Object> authorization) {
        Object pdsHandle = authorization.get(ThirdNodeConst.PDS_HANDLE);
        boolean signedInUser = (pdsHandle != null && !"".equals(pdsHandle));

        boolean isOnCampus = (Boolean)authorization.get(ThirdNodeConst.ON_OFF_CAMPUS);
        
        Boolean isResGroup = (Boolean)authorization.get(ThirdNodeConst.RESTRICTED_USER_GROUP);

        if ( isOnCampus || (signedInUser && !isResGroup) ) {
            return true;
        }

        return false;
}