Page 1

14 July 2011

Today’s Tabbloid PERSONAL NEWS FOR



Just noticed that our campus network failures seem to coincide with the #jiscad online meetings :-S

User response

JUL 13, 2011 05:24P.M.

We think this has been amongst the strongest part of the project. So we’ve had engagement with users through a survey and through a series of 1:1 evaluations with users. As you can see from the graph of survey results, the majority of users are finding that recommendations are useful.


RT @jiscprojects: SUSHI Starters (Standardized Usage Statistics Harvesting Initiative exciting new project: [#jiscad]

When we asked people about the relevance of the recommendations then we found that a high proportion were relevant (50%) with 31% not relevant. That may reflect that the system had been running for only a short period of time and may benefit from more data.

JUL 13, 2011 04:30P.M.


RISE measuring success JUL 13, 2011 04:23P.M. As we reach the end of the RISE project it’s a good time to reflect back on the success of the project. At the start we said that we were going to measure the success in several specific ways (shown in the table below). So how have we done?

We’ve setup Google Analytics to be able to track which types of recommendation and which number recommendation is being used. We’ve done some basic work in looking at the analytics data but there is much more that could be done. The data shows that search recommendations are more likely to be used than other types (but the caveat with RISE is that not all recommendations are being shown equally)

How measured What success looks like User response Survey and informal feedback from students and academics. Analytics data. Majority of users agree that recommendations are useful and enhanced their use of the search system. Analytics shows positive impact. Take-up of tools and data Usage of tools and data, downloads of tools and data. Tools are being downloaded several times a week and there are some comments about the tools. Community feedback Feedback. Wider discussions with community about potential of tools & ways to use the


Today’s Tabbloid PERSONAL NEWS FOR

14 July 2011

Community feedback We’ve had a little over 20 comments on blog posts and some feedback at Activity Data events. We’ve also had 25 people at the Innovations in Activity Data for Academic Libraries event at the Open University in July. RISE was also asked to present at a ‘Subscribed Resources’ workshop that formed part of the SCONUL Shared Services programme. One of the advantages with the Activity Data projects is that we have had the Synthesis project actively working alongside us. We’ve also had to leave until later in the project some of the dissemination activities. But it has seemed difficult to get as much engagement with the wider community as we would have liked. Overall We’re happy with the engagement with users, something that is often difficult to achieve bearing in mind that we are a distance-learning instituion. We probably would have hoped for more engagement with the community but many of the people who are working in this area are already pretty busy with other projects on activity data. But overall, within the constraints of a six month project we are reasonably satisfied with what we’ve been able to do.

Although in comments users have suggested that we show more recommendations, analytics clearly shows that the first two recommendations are much more likely to be viewed than any others. Take up of tools and data We haven’t been able to release any data but both the RISE web interface and RISE Google Gadget have been available for a few months. Usage of the tools shows a steady stream of users even though we haven’t done too much promotion of it given the prototype nature. With over 11,000 page views (12% of them through the Gadget) we have reached a good number of users in a short period of time.


Listening to a presentation on the RAPTOR project (JISC). Usage data on ejournals. #jiscad JUL 13, 2011 03:48P.M.

Downloads and use of the Gadget hasn’t been so easy to track even though there is a Google Gadget Dashboard. We haven’t however had any comments or ratings by users. We are expecting to publish the Gadget on the OU Gadget directory in the near future so this will drive the uptake of the Gadget significantly.


Today’s Tabbloid PERSONAL NEWS FOR

14 July 2011



ADS Programme Virtual Meeting 3 – an AGtivity Perspective

RT @richardn2009: New blog post on activity data and open metadata [#jiscad #ukdiscovery]

JUL 13, 2011 03:38P.M. Catch up session on Elluminate this time for an Online Exchange EventSound and video screen-sharing is reasonably high quality and responsive.

JUL 13, 2011 02:05P.M.

1. JISC Collections (Ross MacIntyre). 90+ institutions signed up for for harvesting from/through the JISC collection archives.


New blog posts for RISE Benefits and a couple of posts with some details about what we do with EZProxy #ourise #jiscad

Aside 1: interesting that users have asked for some form of visualisation – includes certain graphs. Request as a downloadable – eg pdf – which is similar to user requirements for AGtivity. Aside 2: benchmarking requires anonymisation sytems 2. Raptor (Nicole Harris) AIM data capture – relatively simple format and may have better governance rules on use. Aside 1: was also talked about on the AIM topics as overlapping themes. Assume identity management stats are some of the most important to data mine.

JUL 13, 2011 12:11P.M.



“we became increasingly interested about how you could tell stories with data” @rattlecentral #datavis [#jiscad]

EZProxy and activity data JUL 13, 2011 12:07P.M. EZProxy pros and cons

JUL 13, 2011 02:29P.M.

What RISE has demonstrated to us is that using proxy server logfiles from EZProxy as the source of your recommendations has some major limitations (in comparison with OpenURL data at least). In part this is due to limitations in the data that is being handled, but particularly in the way that we, at the OU, are using EZProxy. The first limitation relates to how we use EZProxy and particularly how


Today’s Tabbloid PERSONAL NEWS FOR

14 July 2011

we use it now we have implemented the Ebsco Discovery Solution. At the Open University most of our students use our services off-campus, so we push every electronic resource we can through EZProxy. So when we came to define the project EZProxy seemed like a good place to draw our recommendations from as it saw the greatest coverage of our overall traffic.

typically being 3-4 seconds it isn’t practicable to send up to a dozen API calls just to populate a single page of recommendations and results. So we’ve ended up at the moment with using the EDS metadata as a key to retrieve data from Crossref that we are licensed to store locally. Fortunately we have found quite a high overlap between the data sources so have been able to get data for most of our recommendations. So article level metadata, where you can get it from and what you can do with it, seems to be a major issue.

Now, at the time when we defined the project we were using a federated search system and just swopping to a discovery system. With federated search each of the search targets appeared individually within the EZProxy logfiles with their own URLs so an analysis of the logfiles would show which search target was supplying your content. But, when we switched over to the discovery solution we decided that we would put that through EZProxy. So most of our searches now go to EBSCO and that pulls the full text of the article from the content supplier. Consequently as far as our EZProxy logfile is concerned all it sees is a search to EBSCO not to the final content provider.

Open article level metadata There does however seem to be some differences of opinion between providers of article level metadata (although in the case of aggregators it may be that they themselves are actually licensing it rather than creating it) and Rights and Legal experts over exactly what you can and cannot do with article level metadata. Whether as essentially a statement of fact it is possible to restrict what can be done with this data and whether extracting selected data into another database is allowable or not.

As far as recommendations are concerned that isn’t a major issue but it does mean that analysing the logfiles to find out useful usage data may not work for us (so we need to test it to be sure).

Certainly for RISE it brings in added complications. We’ve pretty much run out of time to do too much more. We can think of a couple of alternative approaches using OpenURL data from EDINA or data from Mendeley that might allow us to match data to the RISE recommendations in a way that would allow the full dataset to be openly released. But realistically that may not be able to be achieved by the time the project ends this month. At the moment we are left with potentially being able to release the EZProxy data without bibliographic data and that may be of limited value. But we will get as far as we can.

EZProxy and article level metadata On the plus side having the Ebsco Discovery Solution API has meant that we are at least able to do something that addresses a major limitation of the EZProxy logfile data. Generally there is very little blibliographic metadata within the logfile (certainly in comparison with OpenURL logfiles). To be able to display sensible recommendations you do need to be able to show some descriptive element to help users understand what is being recommended. As a minimum you would want to show an article title and ideally you would want to show a journal title, date and maybe authors and a DOI.


Benefits – what we expected and what we’ve been able to achieve JUL 13, 2011 12:05P.M.

Your EZProxy logfile data already has a URL you can use to link to the content but some form of bibliographic description is essential as otherwise users cannot choose which recommendations are relevant.

For some reason we didn’t manage to finish off our blog post on the intended benefits of the RISE project. Like the Users post we thought it might be more useful to do it towards the end of the project so we could compare what we hoped we would achieve with what we actually achieved.

Now to be able to display an article title for your recommendations if you don’t have that data in your original logfile requires you to do some postprocessing. In the case of RISE, because the majority of our logfile data relates to EBSCO then we can use the Ebsco Discovery Solution API to retrieve some basic metadata about the article, such as the DOI or article title.

Before we started the project the expected benefits were: Benefits for OU The project links to the Open University’s strategic priority Focus Area 2 Learning and Teaching Efficiency by improving the search experience for users and developing tools that can be used across multiple platforms. Within the Library the project links to Strategic Priorities to ‘improve the search experience across all e-collections by implementing intuitive and integrated search’.

But this starts to raise some complications, especially if your end-game is to be able to openly release your search data (more later). Under our license terms we aren’t permitted to store that data within the RISE database. Now theoretically we already have an internal record ID so we could technically pull the article title in real-time using the API and display it within the RISE interface. However with API response times


Today’s Tabbloid PERSONAL NEWS FOR

14 July 2011

By exploiting data that can be recorded by the existing EZProxy system the OU can start to explore whether providing recommendations to users of e-resources will increase the use of e-resources, broaden the range of resources being used, help users to find material that they would otherwise not have located and improve the search and discovery experience.

has taken part in and presented at a workshop on Subscribed Resources run with JISC and the SCONUL Shared Services initiative.

The Google Gadget has been created and released. It is available here In a short period of time we think we achieved much of what we set out to do. We certainly know a lot more about recommendations and what people want, how they can be used to improve the user experience of discovery systems and what the challenges are around EZProxy data.

Benefits for wider community By using the EZProxy system, which is in widespread use across the HE sector and worldwide, as the source for recommendations data the project will ensure that there is increased value to the community as users of that system will have access to a ready-made toolkit to allow them to exploit this data. With over 100,000 annual unique users of e-resources the OU e-resource search data will provide a large pool of openly accessible attention data about the use of e-resources, this is likely to be of high value to other institutions planning recommendation services, of use to any national/regional scale initiatives, and as the MOSAIC developer competition showed, of value to the wider community in discovering new and innovative ways to use the data. The attention data of e-resource searches may be of interest to support the development of Shared Services around the management of e-resources, e.g. as part of the SCONUL Shared Services initiative. The reports of the processes and issues will be of value to those in the community seeking to follow this route. The Google Gadget developed will be freely available to be adapted by other institutions to access their own search systems. So how far have we been able to get in realising these benefits? We’ve been able to develop both a search interface and a Google Gadget. The search gadget will be taken up by the OU alongside the other Google Gadgets created by the DOULS project in a student dashboard of tools. RISE has certainly allowed us to do much more work in evaluating the place of recommendations as a tool to support the use of search. We haven’t yet been able to judge whether recommendations increases the use of e-resources, but comments in the evaluations would seem to indicate that users are getting some resources they would not otherwise have found. In general users like the idea of having search recommendations. We have had some interest from other EZProxy users in using the tools and will be making the code available before the end of the project. We have built a configuration tool to allow users to specify the format of their logfiles when they setup the code. Although RISE hasn’t yet been able to release any search data, RISE


JISC AD Tabbloid: 14 July 2011  
JISC AD Tabbloid: 14 July 2011  

JISC Activity Data project blogs and #jiscad tweets