3

I have a (probably very basic) question about how to construct a (perl) regex, perl -pe 's///g;', that would find/replace multiple instances of a given character/set of characters in a specified string. Initially, I thought the g "global" flag would do this, but I'm clearly misunderstanding something very central here. :/

For example, I want to eliminate any non-alphanumeric characters in a specific string (within a larger text corpus). Just by way of example, the string is identified by starting with [ followed by @, possibly with some characters in between.

[abc@def"ghi"jkl'123]

The following regex

s/(\[[^\[\]]*?@[^\[\]]*?)[^a-zA-Z0-9]+?([^\[\]]*?)/$1$2/g;

will find the first " and if I run it three times I have all three. Similarly, what if I want to replace the non-alphanumeric characters with something else, let's say an X. 

s/(\[[^\[\]]*?@[^\[\]]*?)[^a-zA-Z0-9]+?([^\[\]]*?)/$1X$2/g;

does the trick for one instance. But how can I find all of them in one go?

Improve this question
4
  • What is your expected output from [abc@def"ghi"jkl'123]? Commented Oct 17, 2018 at 0:37
  • 1
    Your understanding is correct, however each of those multiple instances is your whole match. Since your first instance consumes the whole example string...it's done after one iteration. So that's the problem. Which is easier to describe than fix in your case. It could be solved with a variable-length look behind, which perl does not support, or recursion, which it does. Hopefully someone smarter than me will come along with a more straightforward approach. Commented Oct 17, 2018 at 0:52
  • @Nick it would be [abc@defghijkl123] in the first scenario and [abc@defXghiXjklX123] in the second (when replacing with "X") Commented Oct 17, 2018 at 0:53
  • 1
    @zzxyz That makes sense ... even to a non-expert like me ... and when you write it's "easier to describe than fix" that makes me think at least my question wasn't as basic as I thought :-P Commented Oct 17, 2018 at 1:11

3 Answers 3

Reset to default
3

The reason your code doesn't work is that /g doesn't rescan the string after a substitution. It finds all non-overlapping matches of the given regex and then substitutes the replacement part in.

In [abc@def"ghi"jkl'123], there is only a single match (which is the [abc@def" part of the string, with $1 = '[abc@def' and $2 = ''), so only the first " is removed.

After the first match, Perl scans the remaining string (ghi"jkl'123]) for another match, but it doesn't find another [ (or @).

I think the most straightforward solution is to use a nested search/replace operation. The outer match identifies the string within which to substitute, and the inner match does the actual replacement.

In code:

s{ \[ [^\[\]\@]* \@ \K ([^\[\]]*) (?= \] ) }{ $1 =~ tr/a-zA-Z0-9//cdr }xe;

Or to replace each match by X:

s{ \[ [^\[\]\@]* \@ \K ([^\[\]]*) (?= \] ) }{ $1 =~ tr/a-zA-Z0-9/X/cr }xe;

We match a prefix of [, followed by 0 or more characters that are not [ or ] or @, followed by @.

\K is used to mark the virtual beginning of the match (i.e. everything matched so far is not included in the matched string, which simplifies the substitution).

We match and capture 0 or more characters that are not [ or ].

Finally we match a suffix of ] in a look-ahead (so it's not part of the matched string either).

The replacement part is executed as a piece of code, not a string (as indicated by the /e flag). Here we could have used $1 =~ s/[^a-zA-Z0-9]//gr or $1 =~ s/[^a-zA-Z0-9]/X/gr, respectively, but since each inner match is just a single character, it's also possible to use a transliteration.

We return the modified string (as indicated by the /r flag) and use it as the replacement in the outer s operation.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

this indeed seems the most straightforward solution ... thank you for the detailed explanations, very helpful
Quick question, just as I'm learning: what's the difference between \[ [^\[\]\@]* \@ and \[ [^\[\]]*? \@? And are the spaces between mainly for readability or do they have any other function? (is the latter related to using {}{} instead of ///?)
@jan The spaces are for readability. The /x flag lets you format the regex however you want (you can even use multiple lines and add comments). I think {} delimiters look better than // when the replacement part is a block of code (technically you can use any punctuation character you want). A nice side effect is that it frees up / to be used inside the code (e.g. for tr/// in my case).
@jan As for [^\[\]]*?, I generally distrust non-greedy quantifiers. Whenever possible I try to write my regexes in a way that makes it irrelevant whether quantifiers are greedy or not. In this concrete case there's not much difference, but you have to be careful: E.g. \[ [^\[\]\@]* \@ A and \[ [^\[\]]*? \@ A behave differently when applied to a string of [foo@bar@A; the former regex will not match, the latter regex matches the whole string. That's because the [...]*? part in [...]*? X can match X if necessary to make the whole regex succeed whereas [^X]* will never match X.
Rather than distrusting non-greedy quantifiers (their behavior is just as deterministic as greedy quantifiers), just remember that the regex engine goes from left to right then backtracks to try alternatives when necessary, and it's easier (less attempts) for it to "match as many characters that aren't X as it can find" than for it to "match as few characters as it can find while still having X afterward". It happens that there will very often be an easier (for the engine) alternative to using non-greedy quantifiers.
|
1

So...I'm going to suggest a marvelously computationally inefficient approach to this. Marvelously inefficient, but possibly still faster than a variable-length lookbehind would be...and also easy (for you):

The \K causes everything before it to be dropped....so only the character after it is actually replaced.

perl -pe 'while (s/\[[^]]*@[^]]*\K[^]a-zA-Z0-9]//){}' file

Basically we just have an empty loop that executes until the search and replace replaces nothing.

Slightly improved version:

perl -pe 'while (s/\[[^]]*?@[^]]*?\K[^]a-zA-Z0-9](?=[^]]*?])//){}' file

The (?=) verifies that its content exists after the match without being part of the match. This is a variable-length lookahead (what we're missing going the other direction). I also made the *s lazy with the ? so we get the shortest match possible.

Improve this answer

4 Comments

That's great. I had no idea of the existence of the \K that makes many things much easier (in general) ... always learning. Just out of curiosity: when you write [^]a-zA-Z0-9] why don't you have to escape the ] like this [^\]a-zA-Z0-9]?
Because a [] set requires at least one character. So Perl interprets a ] there as literal vs illegal. On my phone where I have no backticks. Hopefully makes sense.
@jan - definitely the most delightful way I've ever had an answer unaccepted, and I think I'll follow suit in my future questions. Melpomene's answer is definitely superior, too, so good call on that.
:) ... it's sometimes difficult, I often profit from multiple answers/comments ... sure, one can upvote, but sometimes it'd be great to accept more than one answer.
1

Here is another approach. Capture precisely the substring that needs work, and in the replacement part run a regex on it that cleans it of non-alphanumeric characters

use warnings;
use strict;
use feature 'say';

my $var = q(ah [abc@def"ghi"jkl'123] oh); #'
say $var;

$var =~ s{ \[ [^\[\]]*? \@\K ([^\]]+) }{
    (my $v = $1) =~ s{[^0-9a-zA-Z]}{}g;
    $v
}ex;

say $var;

where the lone $v is needed so to return that and not the number of matches, what s/ operator itself returns. This can be improved by using the /r modifier, which returns the changed string and doesn't change the original (so it doesn't attempt to change $1, what isn't allowed)

$var =~ s{ \[ [^\[\]]*? \@\K ([^\]]+) }{
    $1 =~ s/[^0-9a-zA-Z]//gr;
}ex;

The \K is there so that all matches before it are "dropped" -- they are not consumed so we don't need to capture them in order to put them back. The /e modifier makes the replacement part be evaluated as code.

The code in the question doesn't work because everything matched is consumed, and (under /g) the search continues from the position after the last match, attempting to find that whole pattern again further down the string. That fails and only that first occurrence is replaced.

The problem with matches that we want to leave in the string can often be remedied by \K (used in all current answers), which makes it so that all matches before it are not consumed.

Improve this answer

4 Comments

That also gets rid of the [abc@ part.
@melpomene Ugh, thakn you - fixed (along with antother piece)
OP's code finds a match in "[\n\@a.b]"; your code doesn't.
@melpomene I didn't worry about the exact nature of "possibly with some characters in between" and used the . ... replaced it now by OP's code. Thank you

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.