ALA Redux – Ready Reference in Chat – Not Dead Yet! PLUS 1 Foolproof Way to Increase Accuracy in Chat Reference

July 10, 2007 at 10:31 am 2 comments

“I’m not dead yet!”

I wanted to update the LG faithful on the 5 (5!) very different presentations I gave at ALA in Washington DC, based on the research we are doing on the Seeking Synchronicity IMLS grant studying live chat virtual reference at Rutgers University, SCILS and OCLC.

It was wonderful to speak to so many people at ALA who were eager (yes, actually eager) to hear about research in Virtual Reference! Feedback was amazingly positive and I plan to blog about each of these presentations, just to give some highlights, and to point you to the PowerPoint presentation and handouts if you want to get more information.

For the 13th RUSA New Reference Research Forum at ALA, Lynn Silipigni Connaway of OCLC and I presented “Not Dead Yet! Ready Reference in Live Chat Reference.” We had pulled a random sample from almost 500,000 QuestionPoint transcripts from 2004-2006 and done an analysis of the types of questions and an evaluation of accuracy in ready reference. We found that early reports of the death of ready reference (due to user’s ability to search Google and Wikipedia independently for factual type questions) are premature and that these types of questions surprisingly still comprise nearly 30% of chat questions. Here are some of our findings as a tease to invite you to go to the Seeking Synchronicity site for more detail:

What Types of Questions are Being Asked in Chat?

Of 915 total chat reference questions:

  • 243 (27%) were Ready Reference (short, factual, questions)
  • 293 (32%) were Subject Searches (largest group)
  • 10 (1%) were Inappropriate (comforting in such a low number)

What Types of Libraries/Consortia are Getting more Ready Reference Questions?

Of 162 chat transcripts:

  • 59% of the Ready Reference questions came from Public Libraries
  • 45% from Public & Academic Consortia
  • 35% from Adacemic and Law Libraries

How about Accuracy?

How accurate are chat librarians in answering ready reference questions? We used the Arnold & Kaske (2005) method to classify chat ready reference answers and found 78% (141 of 180) to be correct. 69% (125 of 180) meeting the gold standard of correct with citation.

Increase Accuracy! Our Most Important Finding!

For the 20% (35 of 180) incorrect questions, the most frequent reason was that the chat librarian sent only general information (usually a website on the topic). The general website did not contain the specific information requested.

So the “One Foolproof Way” to increase accuracy for ready reference chat questions is as follows: before you push a general information page, make sure it has the specific, exact answer to the user’s question.


Clidk here for the Not Dead Yet! PowerPoint presentation and here for the Not Dead Yet! handout that details coding procedures for type of question and accuracy.

Cited Reference: Arnold, J. & Kaske, N. (2005). Evaluating the quality of a chat cervice, portal: Libraries and the Academy, 5(2), 177-193.

Special thanks to Janet Torsney, an MLIS student at Rutgers University, SCILS, for her help in doing the accuracy analysis and Susanne Sabolcsi-Boros for intercoder check. Also thanks to Nick Belkin of SCILS for suggesting the accuracy analysis.


About these ads

Entry filed under: Uncategorized. Tags: .

The "New" Librarian in the New York Times and The Sun Congrats to Amy!

2 Comments

  • 1. Cynthia  |  July 11, 2007 at 11:29 am

    Very interesting numbers, thanks for the update. How did you decide what transcripts out of the 500,000 to evaluate?

    I finish my time at SCILS in December and one of my goals is to do Chat Reference. I wish there had been an opportunity at SCILS to do Chat Reference as part of the program so I would feel better prepared for when I am out in the world.

  • 2. Marie L. Radford  |  July 12, 2007 at 3:14 pm

    Hi there! We pulled a totally random sample of 30-50 transcripts per month over an 18 month period so we feel that we can generalize results to the population over time.

    I am working on providing opportunities for SCILS students to try chat during their studies, or at a minimum to shadow chat librarians. If you contact me via email I can arrange for you to shadow a QandANJ chat librarian.

    Thanks for your comment!


Creative Commons

Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License.
Disclaimer: The thoughts expressed on this blog are those of the authors and are not intended to reflect the views of our employers.

A Note on the history of posts

Please note that all Library Garden posts dated earlier than September 13,2009 originally appeared on our Blogger site. These posts have been imported to this site as a convenience when searching the entire site for content.

If you are interested in seeing the original post, with formatting and comments in tact, please bring up the original post at our old Blogger site.

Thanks for reading Library Garden!

wordpress
visitors

Follow

Get every new post delivered to your Inbox.

Join 37 other followers

%d bloggers like this: