0

I am trying to count the number of same words in an Urdu document which is saved in UTF-8.

so for example I have document containing 3 exactly same words separated by space

خُداوند خُداوند خُداوند

I tried to count the words by reading the file using the following code:

        file_obj = codecs.open(path,encoding="utf-8")
        lst = repr(file_obj.readline()).split(" ")
        word = lst[0]
        count =0
        for w in lst:
            if word == w:
                count += 1
        print count

but the value of count I am getting is 1 while I should get 3.

How does one compare Unicode strings?

Improve this question
7
  • 1
    What does lst print as? I have [u'\u062e\u064f\u062f\u0627\u0648\u0646\u062f', u'\u062e\u064f\u062f\u0627\u0648\u0646\u062f', u'\u062e\u064f\u062f\u0627\u0648\u0646\u062f'] and those are exactly identical (your code works). But if there are any denormalized forms then they won't be identical. Commented Nov 3, 2013 at 10:14
  • See Normalizing Unicode for the proper way to handle Unicode values with denormalized codepoints. Commented Nov 3, 2013 at 10:15
  • And remove the repr(). You just added u' and ' to the start and end of the string. So word is now "u'\u062e\u064f\u062f\u0627\u0648\u0646\u062f, lst[1] is '\u062e\u064f\u062f\u0627\u0648\u0646\u062f' and list[2] is "\u062e\u064f\u062f\u0627\u0648\u0646\u062f'". These strings are obviously not equal. Commented Nov 3, 2013 at 10:25
  • removing repr() gives me an error: File "C:\Python27\lib\encodings\cp437.py", line 12, in encode return codecs.charmap_encode(input,errors,encoding_map) UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-7: character maps to <undefined> Commented Nov 3, 2013 at 10:29
  • Not on that line it won't. Are you trying to print these values somewhere too? Commented Nov 3, 2013 at 10:30

3 Answers 3

Reset to default
3

Remove the repr() from your code. Use repr() only to create debug output; you are turning a unicode value into a string that can be pasted back into the interpreter.

This means your line from the file is now stored as:

>>> repr(u'خُداوند خُداوند خُداوند\n').split(" ")
["u'\\u062e\\u064f\\u062f\\u0627\\u0648\\u0646\\u062f", '\\u062e\\u064f\\u062f\\u0627\\u0648\\u0646\\u062f', "\\u062e\\u064f\\u062f\\u0627\\u0648\\u0646\\u062f\\n'"]

Note the double backslashes (escaped unicode escapes) and the first string starts with u' and the last string ends with \\n'. These values are obviously never equal.

Remove the repr(), and use .split() without arguments to remove the trailing whitespace too:

lst = file_obj.readline().split()

and your code will work:

>>> res = u'خُداوند خُداوند خُداوند\n'.split()
>>> res[0] == res[1] == res[2]
True

You may need to normalize the input first; some characters can be expressed either as one unicode codepoint or as two combining codepoints. Normalizing moves all such characters to a composed or decomposed state. See Normalizing Unicode.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

Try removing the repr?

lst = file_obj.readline().split(" ")

The point is that you should at least print variables like lst and w to see what they are.

Improve this answer

3 Comments

Even with the repr() the sample input still works; of course the OP should not be using that, however.
Ah, no, the first string will have u', the second has no quotes, the third a trailing '.
Your opening sentence Try removing the repr? suggested otherwise.
0

Comparing unicode strings in Python:

a = u'Artur'
print(a)
b = u'\u0041rtur'
print(b)

if a == b:
    print('the same')

result:

Artur
Artur
the same
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.