Parsing XML by using nested loops and returning multiple values

Question

I have a large XML file having the following structure:

<brain>
<q>
  <question> What are your hobbies? </question>
  <question> What do you do for sake of fun? </question>
  <question> How do you spend your spare time? </question>
  <question> What are your interests? </question>
  <question> What do you enjoy most? </question>
  <answer> I like [personal_info/hobby] </answer>
  <answer>[personal_info/hobby]</answer>
  <answer>I enjoy [personal_info/hobby] </answer>
</q>

<q>
  <question> Where do you live? </question>
  <question> What city do you live in? </question>
  <question> Where are you from? </question>
  <question> Where are you living? </question>
  <question> Where is your residence? </question>
  <answer> I live at [personal_info/loc] </answer>
  <answer> I am living in [personal_info/loc]</answer>
  <answer> At [personal_info/loc]</answer>
  <answer>   [personal_info/loc]</answer>
</q>
.
.
.
</brain>

As you might have guessed, it is a database for a chatbot. The idea is that the user will enter a question (or any sentence for that matter) and our java-based chatbot will run an XQuery over this file. The XQuery implementation that I am using (known as nux) provides a fuzzy matching of sentence similarity and so will return sentences that partially match. Here is some code to illustrate this:

Nodes results = XQueryUtil.xquery(doc, "declare namespace lucene = \"java:nux.xom.pool.FullTextUtil\"; "
    + "for $q in /brain/q "
    + " for $question in $q/question"
    + " let $score := lucene:match($question, \"How are you\") "
    + " where $score > 0.1 "
    + " order by $score descending "
    + "return $q/answer");

This code is supposed to loop through each brain/q and then q/question and if its similarity score is more than 0.1, it should return <answer>'s of that are in that <q>. The problem is that it returns ALL answer tags. For example if "What are your hobbies?" is asked, it should return

  <answer> I like [personal_info/hobby] </answer>
  <answer>[personal_info/hobby]</answer>
  <answer>I enjoy [personal_info/hobby] </answer>

but returns all the answer tags found in the file. It also repeats them again and again for unpredictable number of times.

Can you please help me on this?

The dataset was generated by running various scripts and were collected and manually checked by me. If necessary, I can change the structure of XML to solve this problem but will prefer not if it is possible.

Thanks for taking time to read my question and thinking to help.

Yhn · Accepted Answer · 2011-06-21 08:22:42Z

2

I know from experience with Lucene (or actually Solr); that a similarity score of 0.1 is achieved pretty quickly. Which would explain why all answers in the file are returned.

In the search system I'm using Solr for, I use a boundary of about 0.4~0.6 (depending on the searched fields).

You could try to show the returned scores per question (if the XQueryUtil allows that); to see how much the correct ones match for lines you want to match. That way you can select a good/better boundary.

Another way would be to just try using some higher values, and see if you get less answers back, and try to find a correct value by trial and error.

answered Jun 21, 2011 at 8:22

Yhn

2,8252 gold badges15 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Muhammad Yasir Over a year ago

In fact this was the first thing I tried to achieve (printing $score value) but could not get to it. May be someone experienced in XQuery might be able to tell how to do it. Now, through trial and error, I understood that similarity of "What are your hobbies" with itself is just 0.26. (wierd I think). Secondly it returns repeated values as I mentioned in my question. How to solve that problem?

Yhn Over a year ago

I don't really know XQuery; but when looking at the query and trying to logically reason about it... if you have a <question> in a <q> matching enough, I would expect it to return all <answer>s for that <question>. So if two <question>s match; you would get all <answer>s twice. Maybe XQuery has some kind of DISTINCT functionality like SQL?

Muhammad Yasir Over a year ago

Got distinct-values() function after some research. Thanks for help but still don't understand what value is best to use... As I said a score of 0.26 is too low.

Yhn Over a year ago

Glad to hear at least the distinct stuff is helped. I don't have much experience with XQuery, so I don't really know why the lucene score is that low for exact matches.. maybe there is a configuration possible as for what analyzers should be used for indexing/searching like in lucene? Those can affect the score returned.

Team Pannous · Accepted Answer · 2011-06-21 21:10:38Z

0

Important: scores in solr are always relative. You should never compare them to a fixed value since they differ from query to query and they are not normalized.

Why not use a normal query, set rows to 1 or 10 and ( automatically) order by score? You can make answers a multi valued field and create one document per q.

You should run the best matches through your own quality function anyways to take care of at least minimal semantic matching.

answered Jun 21, 2011 at 21:10

Team Pannous

1,0941 gold badge8 silver badges11 bronze badges

1 Comment

Muhammad Yasir Over a year ago

Can you please elaborate a little more? As you can see from my XQuery, I am sorting by score already. May be you can explain your point through an example?

Collectives™ on Stack Overflow

Parsing XML by using nested loops and returning multiple values

2 Answers 2

4 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related