10

I would like to split a text into single words using PHP. Do you have any idea how to achieve this?

My approach:

function tokenizer($text) {
    $text = trim(strtolower($text));
    $punctuation = '/[^a-z0-9äöüß-]/';
    $result = preg_split($punctuation, $text, -1, PREG_SPLIT_NO_EMPTY);
    for ($i = 0; $i < count($result); $i++) {
        $result[$i] = trim($result[$i]);
    }
    return $result; // contains the single words
}
$text = 'This is an example text, it contains commas and full-stops. Exclamation marks, too! Question marks? All punctuation marks you know.';
print_r(tokenizer($text));

Is this a good approach? Do you have any idea for improvement?

Thanks in advance!

Improve this question

6 Answers 6

Reset to default
31

Use the class \p{P} which matches any unicode punctuation character, combined with the \s whitespace class.

$result = preg_split('/((^\p{P}+)|(\p{P}*\s+\p{P}*)|(\p{P}+$))/', $text, -1, PREG_SPLIT_NO_EMPTY);

This will split on a group of one or more whitespace characters, but also suck in any surrounding punctuation characters. It also matches punctuation characters at the beginning or end of the string. This discriminates cases such as "don't" and "he said 'ouch!'"

Improve this answer
Sign up to request clarification or add additional context in comments.

10 Comments

+1, not sure, tho, how this will deal with äöüß. Does regex normally classify äöüß as word characters?
Thank you. This would't probably work for English texts but I also want to extract German umlauts (ä, ö, ü), the "ß" and numbers in a string. The "\W" wouldn't extract "Fri3nd", would it?
Seems it does not, but updated answer with something similar that works.
Updated answer works with perl (which php regex are based on): $ echo "äöüß, test" | perl -e 'while (<>) { if (/([\p{P}\s]+)/) { print "$1\n"; } }' ,
Should one split don't into don and t?
|
14

Tokenize - strtok.

<?php
$text = 'This is an example text, it contains commas and full stops. Exclamation marks, too! Question marks? All punctuation marks you know.';
$delim = ' \n\t,.!?:;';

$tok = strtok($text, $delim);

while ($tok !== false) {
    echo "Word=$tok<br />";
    $tok = strtok($delim);
}
?>
Improve this answer

4 Comments

This won't work if you get a : or ; or any other punctuation character you haven't accounted for.
@marcog, I added : and ;. Doesn't {P} catch apostrophe and hyphen?
What about cases such quoting? My updated answer discriminates between these cases.
Excellent idea. Added +1. The only thing is that there should be double quotes around $delim = " \n\t,.!?:;"; With the single quotes it does not work correctly, it splits by the letter n too.
3

I would first make the string to lower-case before splitting it up. That would make the i modifier and the array processing afterwards unnecessary. Additionally I would use the \W shorthand for non-word characters and add a + multiplier.

$text = 'This is an example text, it contains commas and full stops. Exclamation marks, too! Question marks? All punctuation marks you know.';
$result = preg_split('/\W+/', strtolower($text), -1, PREG_SPLIT_NO_EMPTY);

Edit   Use the Unicode character properties instead of \W as marcog suggested. Something like [\p{P}\p{Z}] (punctuation and separator characters) would cover the characters more specific than \W.

Improve this answer

2 Comments

Thanks, the idea to perform strtolower() before is very good. I'll use this.
What purpose does strtolower() serve if you are splitting with \W? Do you want to add a u pattern modifier? A note to researchers... \W will not match an underscore.
1

you can also use PHP strtok() function to fetch string tokens from your large string. you can use it like this:

 $result = array();
 // your original string
 $text = 'This is an example text, it contains commas and full stops. Exclamation marks, too! Question marks? All punctuation marks you know.';
 // you pass strtok() your string, and a delimiter to specify how tokens are separated. words are seperated by a space.
 $word = strtok($text,' ');
 while ( $word !== false ) {
     $result[] = $word;
     $word = strtok(' ');
 }

see more on php documentation for strtok()

Improve this answer

2 Comments

what is the difference between this and explode(' ', $text);
The code sample in the question is a tokenizer, my answer was implying that PHP has a string tokenizer built-in. Also explode() will return all of the words of the text at once, but using strtok() the caller has the choice to stop searching for words in the text, as soon as a desired condition is met. Other than this, I can't think of any other difference.
1

Do:

str_word_count($text, 1);

Or if you need unicode support:

function str_word_count_Helper($string, $format = 0, $search = null)
{
    $result = array();
    $matches = array();

    if (preg_match_all('~[\p{L}\p{Mn}\p{Pd}\'\x{2019}' . preg_quote($search, '~') . ']+~u', $string, $matches) > 0)
    {
        $result = $matches[0];
    }

    if ($format == 0)
    {
        return count($result);
    }

    return $result;
}
Improve this answer

4 Comments

Thanks but this wouldn't work. "Fri3nd" wouldn't be extracted but it should.
I don't understand why "Fri3nd" should be extracted. Removed from the array, broken down into "Fri3" and "nd" (or similar)? O.o
If you want to consider numbers as words just do str_word_count_Helper($string, 1, '0123456789');
Native PHP functions that allow double-dot range syntax demonstrates that str_word_count($string, 1, '0..9') will do.
1

You can also use the method explode : http://php.net/manual/en/function.explode.php

$words = explode(" ", $sentence);
Improve this answer

1 Comment

not works with 2 or more consecutive spaces. you have to use a foreach with explode(" ", $sentence) within if($word == "") continue; so you could avoid empty words.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.