Mapping fields from WoS API => LOM

This is my first attempt at mapping the fields returned from the WoS API to LOM – hopefully this can form the basis of an appropriate XSLT transformation.

As my template I’ve used the first record from Mike’s full query to the API which happens to be a book review. It might have been better to start with a journal article but we’ll obviously need to look at the full range of records (N.B. Will we actually be able to return <doctype> when API restrictions are applied?)

Metadata	WoS API field[s]	LOM field[s]
Record unique identifier	<ut>ISI UT identifier</ut>	<lom:identifier><lom:entry>ISI UT identifier</lom:entry></lom:identifier>
		<ns0:bibliographicCitation xmlns:imsmd=”http://www.imsglobal.org/xsd/imsmd_v1p2″>
Source title	<source_title>TITLE</source_title>	<ns0:title><ns0:string >TITLE</ns0:string></ns0:title>
Volume/issue/pages/Date (month/year)	<bib_id>31 (7): 1116-1117 NOV 2009</bib_id>	<ns0:volume>31</ns0:volume><ns0:issue>7</ns0:issue><ns0:date>NOV 2009</ns0:date>
Pages	<bib_pages begin=”1116” end=”1117” pages=”2“>1116-1117</bib_pages>	<ns0:spage>1116</ns0:spage><ns0:epage>1117</ns0:epage>
		</ns0:bibliographicCitation>
Date(year)/volume	<bib_issue year=”2009” vol=”31” />	<ns0:volume>31</ns0:volume><ns0:date>NOV 2009</ns0:date>
Item title	<item_title>ITEM TITLE</item_title>	<lom:title><lom:string >ITEM TITLE</lom:string></lom:title>
Author(role/name)	<authors count=”1“><primaryauthor>Robertson, S</primaryauthor><fullauthorname> <AuRole>Author, Reprint Author</AuRole> <AuLastName>Robertson</AuLastName> <AuFirstName> Steve</AuFirstName> <address number=”1“>Leeds Metropolitan Univ, Ctr Mens Hlth, Leeds LS1 3HE, W Yorkshire, England</address> <AuCollectiveName> Robertson, Steve</AuCollectiveName> </fullauthorname></authors>	<lom:lifeCycle><lom:contribute><lom:role><lom:source>LOMv1.0</lom:source><lom:value>author</lom:value></lom:role><lom:entity>BEGIN:vcard&#xD;FN:Steve Robertson&#xD;END:vcard</lom:entity></lom:contribute>   </lom:lifeCycle>

The XML for the record is reproduced below with relevant fields emphasise in red/bold:

<REC inst_id="0" recid="184349093" hot="yes" sortkey="3067387552" timescited="0" sharedrefs="0" inpi="false">
 <item issue="184349006" recid="184349093" coverdate="200911" sortkey="3067387552" refkey="5892051" dbyear="2009">
  <ut>000271043700017</ut>
  <i_ckey>ROBE1116090031SS</i_ckey>
  <i_cid>0130983640</i_cid>
  <source_title>SOCIOLOGY OF HEALTH & ILLNESS</source_title>
  <source_abbrev>SOCIOL HEALTH ILL</source_abbrev>
  <item_title>Men’s Health: Body, Identity and Social Context</item_title>
  <sq>06031J0</sq>
  <bib_id>31 (7): 1116-1117 NOV 2009</bib_id>
  <bib_pages begin="1116" end="1117" pages="2">1116-1117</bib_pages>
  <bib_issue year="2009" vol="31"/>
  <doctype code="B">Book Review</doctype>
  <editions full="SSCI"/>
  <languages count="1">
   <primarylang code="EN">English</primarylang>
  </languages>
  <reviewed_work>
   <rw_langs count="1">
    <rw_lang code="EN">English</rw_lang>
   </rw_langs>
   <rw_authors count="2">
    <rw_author>Broom, A</rw_author>
    <rw_author>Tovey, P</rw_author>
   </rw_authors>
   <rw_year>2009</rw_year>
  </reviewed_work>
  <authors count="1">
   <primaryauthor>Robertson, S</primaryauthor>
   <fullauthorname>
    <AuRole>Author, Reprint Author</AuRole>
    <AuLastName>Robertson</AuLastName>
    <AuFirstName>Steve</AuFirstName>
    <address number="1">
    Leeds Metropolitan Univ, Ctr Mens Hlth, Leeds LS1 3HE, W Yorkshire, England
    </address>
    <AuCollectiveName>Robertson, Steve</AuCollectiveName>
   </fullauthorname>
  </authors>
  <reprint>
   <rp_author>Robertson, S</rp_author>
   <rp_address>Leeds Metropolitan Univ, Ctr Mens Hlth, Leeds LS1 3HE, W Yorkshire, England</rp_address>
   <rp_organization>Leeds Metropolitan Univ</rp_organization>
   <rp_suborganizations count="1">
    <rp_suborganization>Ctr Mens Hlth</rp_suborganization>
   </rp_suborganizations>
   <rp_city>Leeds</rp_city>
   <rp_state>W Yorkshire</rp_state>
   <rp_country>England</rp_country>
   <rp_zips count="1">
    <rp_zip location="AC">LS1 3HE</rp_zip>
   </rp_zips>
  </reprint>
  <research_addrs count="1">
   <research>
    <rs_address number="1">Leeds Metropolitan Univ, Ctr Mens Hlth, Leeds LS1 3HE, W Yorkshire, England</rs_address>
    <rs_organization>Leeds Metropolitan Univ</rs_organization>
    <rs_suborganizations count="1">
     <rs_suborganization>Ctr Mens Hlth</rs_suborganization>
    </rs_suborganizations>
    <rs_city>Leeds</rs_city>
    <rs_state>W Yorkshire</rs_state>
    <rs_country>England</rs_country>
    <rs_zips count="1">
     <rs_zip location="AC">LS1 3HE</rs_zip>
    </rs_zips>
   </research>
  </research_addrs>
  <abstract avail="N"/>
  <refs count="1">
   <ref>184349094</ref>
  </refs>
 </item>
</REC>

As Mike points out, we should bear in mind that there are variations in the fields that come back from the WoS API, for example: bib_id – depends on what bib data has been entered into WoS; authors – The level of detail varies here too, sometimes there is no full author name and so on….all of which will impact on the sophistication we require from the XSLT.

Quick sketch #2

The diagram below is Arthur’s update of my earlier quick sketch to illustrate what Bibliosight will aim to achieve by the formal #jiscri deadline.

It is numbered and colour coded – stages 1 – 3 (shades of blue) are within the #jiscri timeframe; stages 2 (green) & 5 (buff) will require ongoing work beyond the deadline.

(N.B.  Click on the image for a full size view in a separate browser window.)

Thinking out loud…

As the deadline for #jiscri draws close I have just returned to work after a month away from Bibliosight and I’m now desperately trying to catch up with the project and determine exactly what we can aim to achieve by the end of November…The candid truth is that we have only very recently got to the point where Mike can actually do some coding and begin to put together a prototype that fulfills the requirements of our (still formative) use-case[s].

Yesterday morning I had a stab at completing a more detailed template for a primary use-case (this comprises a narrative and the use case itself); then in the afternoon I sat down with Mike to catch up with his progress from a technical perspective and to brain-storm around precisely what functions we require from our prototype and how this may be achieved; there are also some outstanding issues of clarity pertaining to Thomson Reuter’s API documentation, specifically “WoS Search Retrieve Codes and Descriptions” in that we currently have unrestricted access to the API but it is my understanding that the free* service will actually be restricted.  We are not certain:

a)  Precisely which of the fields are associated with the restricted subset that we will be able to query and/or return under the current terrms of our WoS subscription*

b)  What some of the fields actually are as they lack a description in the documentation

*Free to us under existing subscription

Disclaimer:  I’m very much thinking out loud here and attempting to translate what I understand are ongoing conceptual issues for Mike as he works through the documentation.

Note:  I’ve continued to refer to ResearcherID – see http://bibliosightnews.wordpress.com/2009/10/02/visit-from-thomson-reuters/ – though it is not a service we plan on implementing as part of Bibliosight, and not necessarily even in the longer term, I’m pretty sure we are likely to require some sort of unique identifier for authors – a subject that is currently receiving a lot of attention from the repository community.

Anyway…looking back over the blog it seems that:

The requesting system can query the Web of Science using the following fields:

• Address (including Street, City, Province, Zip Code, or Country)
• Author
• Conference (including title, location, data, and sponsor)
• Group Author
• Organization or Sub-organization
• Source Publication (journal, book or conference)
• Title
• Topic
• Year Published

The service will support the AND, OR, NOT, and SAME Boolean operators.

The Web of Science Web Service returns five fields to the requesting system:

• Article Title
• Authors — All authors, book authors, and corporate authors
• Source — Includes the source title, subtitle, book series and subtitle, volume, issue, special issue, pages, article number, supplement number, and publication date
• Keywords — all author supplied keywords
• UT — A unique article identified provided by Thomson Reuters

The test queries that Mike has submitted to the API have returned XML that appears to be both more granular than indicated and that includes fields other than those that constitute these five (e.g. abstract) so the first thing to do, perhaps, is to contact Thomson Reuters and see if they can apply the restrictions that we will ultimately need to work with, if only to remove some of the noise and make it easier to see the wood for the trees.

The API documentation actually lists over 100 “fields”; only a handful of these are actually described in the documentation, however, and while many are reasonably transparent, others are a little less so and some look like they may duplicate information – or are they perhaps used as alternatives? (e.g. bib_id = Volume, issue, special, pages and year data / bib_issue = Volume and year data).  There is also some lack of consistency in this bibliographic info on a record by record basis; we need to ensure that we have consistent XML being returned for all records – hopefully we can then develop a template in intraLibrary itself that reflects that consistent XML as closely as possible such that we can devise an XSLT style-sheet to perform the approriate transformation.

Mike already has a desktop client that will take XML and perform an XSLT transformation so, once we have clarified the LOM format we require (an action for me from the last meeting), it *should* be relatively straightforward to plug into the WoS API to retrieve XML from the Web of Science which can then be transformed into appropriate LOM.

Then we need to ingest that LOM into intraLibrary, preferably using SWORD…which I shall think about another time!

Visit from Thomson Reuters

On Wednesday afternoon Mike and I were finally able to sit down with Jon and…Gareth? (sorry, I’m terrible with names) from Thomson Reuters to discuss Bibliosight and the work we are doing with the WoS API, it probably goes without saying just how useful this was, especially so soon after our Tuesday meeting.

As we have come to appreciate, Thomson are still very much in an ongoing process of developing their suite of tools and commercial services around the extraction of data from WoS using their API and, overall, I was given the impression that the company are currently practising something of a balancing act to weigh their commercial interests against providing appropriate value added services to their subscribers under existing licensing agreement – which is, of course, entirely reasonable.  Jon suggested that the Bibliosight project is something of a pioneer in using this technology and a useful case-study for the company, which certainly puts some of our early difficulties into context – though he did indicate that numerous other folk are also actively investigating the API; in particular he mentioned Queens College Belfast, an institution in Birmingham and R4R at Kings College London in collaboration with EPrints’ Les Carr at Soton.  R4R is the only project that I was hitherto aware of and have had any contact with; it would be really useful if we were able to communicate with others also using the API.

Thomson Reuter’s flagship commercial product is called InCites and “supplies all the data and tools you need to easily produce targeted, customized reports… all in one place. You can conduct in-depth analyses of your institution’s role in research, as well as produce focused snapshots that showcase particular aspects of research performance.” We discussed how, though such a service will be invaluable for the research oriented Russell Group institutions, it is likely to be overkill for a million plus institution like Leeds Met; nevertheless we do require a certain level of functionality to help us analyse our research performance which, alongside our traditional strengths in teaching and learning, is increasingly important, especially in view of the REF.  Hopefully this is where the developing ‘suite of tools’ comes in and our guests were keen to get a handle on precisely what we are hoping to achieve with Bibliosight (aren’t we all!).  I outlined our preliminary use-cases for them as a foundation for our discussion and was also keen to ask some of the specific questions that had arisen during the previous day’s meeting.  First of all I asked about the wording of the documentation that appears to suggest that it is only possible to return 100 records with a single query using the API – they weren’t aware of such an issue and agreed that the way it was expressed in the documentation was a little ambiguous; Jon will follow this up for us though Mike may also be able to elucidate the situation when he has investigated further.  They were able to say that another user had discovered that the API could be called twice every second, however, so didn’t anticipate any problems with extracting all the data we need.

The major issue that came up at the meeting on Tuesday was how best to return all of the articles for a given institution with the most appropriate field to query apparently being the address field.  It is not clear, however, how consistent the institutional address actually is and Jon confirmed that it is derived from information harvested from individual journals/papers which preliminary manual searching of WoS has already demonstrated to be idiosyncratic  – at least in the case of Leeds Metropolitan University and almost certainly other institutions aswell (leeds metropolitan university; leeds met [uni]; lmu etc).  Jon suggested that the safest and most effective method of returning all records would actually be by using ResearcherID though this would require all institutional authors to be registered and an additional paid subscription to ResearcherID download (as opposed to upload which is free) – in lieu of this, however, he did confirm that the address field was the only way and that it may be necessary to build a catch-all query to ensure that we don’t miss anything – precisely how we achieve this is still a little bit of a moot point, though he did indicate that some work has been done on disambiguating institutional address formats within WoS and will follow up on this for us in due course.

Through our discussion, Article Match Retrieval is finally beginning to make more sense to me now, and Jon confirmed that this is the method that would be used in conjunction with the API to provide numbers of citations to an individual article – AMR can be queried by numerous fields including DOI and UT Identifier (A unique identifier for a journal article assigned by Thomson Reuters.); in terms of the current project, I think it makes sense to focus initially on extracting bibliographic data first before worrying about citation metrics; via the API, we can also extract the UT identifier and then use this to query AMR.

We also touched on Terms & Conditions and Thomson, again reasonably, expect WoS as data source to be clearly acknowledged on each individual record – Mike wasn’t initially certain how this could easily be achieved from a technical perspective, at least in the case of bibliographic citation information (which may have been added manually); we have a few ideas on how this could actually be achieved but is really just something to be aware of at this stage.

All in all I now feel that the overall shape project is beginning to be resolved and, in addition to the technical work required to extract, store, parse, convert (XML) records and then pass them somewhere else (intraLibrary/EndNote), a large part of Bibliosight will necessarily focus on developing use-cases for our institutiona research administration which is likely to continue well beyond the designated 6 month life-cycle of the #jiscri project!

Quick sketch

I’ve just had a meeting with Sue and Sam (from URO) about use cases.  It was useful, as always, to put our heads together and we came up with a quick sketch of a potential infrastructure for Bibliosight that we would like feedback on:

A quick sketch

N.B.  This assumes it is possible to programmatically bulk upload XML records to EndNote – I have no idea if this is possible – and to intraLibrary which should be possible based on discussion at the meeting on Tuesday.

The sketch arose from the fact that research administrators currently use EndNote which they wish to continue using as dedicated citation management software for its high level functionality and the need to simplify their workflow rather than expect them to add records to two separate systems.  Such an approach could also inform our developing use cases to a) auto-populate the repository with metadata from WoS and b) alert repository/research administrators when an article is published so we can pursue an appropriate full text version for the repository.