Python tokenizing strings

Question

I'm new to python and would like to know how I can tokenize strings based on a specified delimiter. For example, if I have the string "brother's" and I would like to turn it to ["brother", "\s"] or a string "red/blue" to ["red", "blue"], what would be the most appropriate way to do this? Thanks.

I would start with pydoc str and work from there.

larsks
– larsks

2014-02-05 06:17:53 +00:00
Commented Feb 5, 2014 at 6:17 — larsks
– larsks, Commented Feb 5, 2014 at 6:17

jterrace · Accepted Answer · 2014-02-05 06:18:25Z

2

You would use the split method:

>>> 'red/blue'.split('/')
['red', 'blue']
>>> "brother's".split("'")
['brother', 's']

answered Feb 5, 2014 at 6:18

jterrace

67.5k23 gold badges164 silver badges208 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Mozbi Over a year ago

Thanks. How about if I had something like "brothers'" with the single quote after "brothers" and I want it to be ['brother', '\'s']?

mhlester · Accepted Answer · 2014-02-05 06:31:41Z

1

What you're looking for is called split, and it's called on the str object. For instance:

>>> brotherstring = "brother's"
>>> brotherstring.split("'")
['brother', 's']
>>> redbluestring = "red/blue"
>>> redbluestring.split("/")
['red', 'blue']

There are a few variants on split, such as rsplit, partition, etc that all do different things. Read the documentation to find the one that works best for your purpose.

edited Feb 5, 2014 at 6:31

answered Feb 5, 2014 at 6:17

mhlester

23.3k10 gold badges55 silver badges76 bronze badges

Comments

Tanveer Alam · Accepted Answer · 2014-02-05 13:41:35Z

1

Try this.

>>> strr =  "brother's"
>>> strr.replace("'","\\'").split("\\")
['brother', "'s"]

>>> strrr = "red/blue"
>>> strrr.split('/')
['red', 'blue']

answered Feb 5, 2014 at 13:41

Tanveer Alam

5,2754 gold badges24 silver badges44 bronze badges

2 Comments

VISQL Over a year ago

This is a great answer. It shows how to preserve punctuation, in the case that your punctuation is not your delimiter. Can reconstruct the original later, or clean further if the apostrophe is really unwanted.

Tanveer Alam Over a year ago

@VISQL Thanks for the appreciation.

Collectives™ on Stack Overflow

Python tokenizing strings

3 Answers 3

1 Comment

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related