Regex to match tokens in a string using string.gmatch

Question

I need a regex to use in string.gmatch that matches sequences of alphanumeric characters and non alphanumeric characters (quotes, brackets, colons and the like) as separated, single, matches, so basically:

str = [[
    function test(arg1, arg2) {
        dosomething(0x12f, "String");
    }
]]

for token in str:gmatch(regex) do
    print(token)
end

Should print:

function
test
(
arg1
,
arg2
)
{
dosomething
(
0x121f
,
"
String
"
)
;
}

How can I achieve this? In standard regex I've found that ([a-zA-Z0-9]+)|([\{\}\(\)\";,]) works for me but I'm not sure on how to translate this to Lua's regex.

Egor Skriptunoff · Accepted Answer · 2016-06-20 16:03:46Z

1

local str = [[
    function test(arg1, arg2) {
        dosomething(0x12f, "String");
    }
]]

for p, w in str:gmatch"(%p?)(%w*)" do
   if p ~= "" then print(p) end
   if w ~= "" then print(w) end
end

answered Jun 20, 2016 at 16:03

Egor Skriptunoff

23.8k2 gold badges38 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

user6245072 Over a year ago

What does %p match?

Egor Skriptunoff Over a year ago

%w - alphanumeric, %p - punctuation (punctuation is all ASCII7 except alphanumeric, space and control characters)

user6245072 Over a year ago

This is really useful - I could for example add all the tokens to a table and then process them - but how could I preserve in a unique token strings delimited by the character "?

Egor Skriptunoff Over a year ago

@user6245072 - It would be more complex program. It should correctly handle comments, escaped quotes inside string literals, etc.

Paul Kulchenko Over a year ago

@user6245072, if handling of quoted fragments is needed, then something like this may be used: stackoverflow.com/questions/28664139/…

|

Wiktor Stribiżew · Accepted Answer · 2016-06-20 14:59:28Z

1

You need a workaround involving a temporary char that is not used in your code. E.g., use a § to insert it after the alphanumeric and non-alphanumeric characters:

str = str:gsub("%s*(%w+)%s*", "%1§") -- Trim chunks of 1+ alphanumeric characters and add a temp char after them
str = str:gsub("(%W)%s*", "%1§")     -- Right trim the non-alphanumeric char one by one and add the temp char after each
for token in str:gmatch("[^§]+") do  -- Match chunks of chars other than the temp char
    print(token)
end

See this Lua demo

Note that %w in Lua is an equivalent of JS [a-zA-Z0-9], as it does not match an underscore, _.

answered Jun 20, 2016 at 14:59

Wiktor Stribiżew

631k41 gold badges502 silver badges632 bronze badges

Collectives™ on Stack Overflow

Regex to match tokens in a string using string.gmatch

2 Answers 2

6 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related