Extract bot name from user-agent string-Python

Question

I want to extract the botname with its version from user-agent strings. I tried using split function. But since the way of displaying user-agent string is different from one crawler to the other what is the best way to get my expected out put?(Please consider that i need a general solution)

Input(user-agent strings)

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)
msnbot/2.0b (+http://search.msn.com/msnbot.htm)

Expected output

Googlebot/2.1
AhrefsBot/4.0
msnbot/2.0b

Take a look: stackoverflow.com/questions/927552/…

alecxe
– alecxe

2013-09-03 08:52:03 +00:00
Commented Sep 3, 2013 at 8:52 — alecxe
– alecxe, Commented Sep 3, 2013 at 8:52

falsetru · Accepted Answer · 2013-09-03 08:50:19Z

4

Try following:

import re

lines = [
    'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)',
    'Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)',
    'msnbot/2.0b (+http://search.msn.com/msnbot.htm)'
]

botname = re.compile('\w+bot/[.\w]+', flags=re.IGNORECASE)
for line in lines:
    matched = botname.search(line)
    if matched:
        print(matched.group())

prints

Googlebot/2.1
AhrefsBot/4.0
msnbot/2.0b

assumed that bot agent names contain bot/.

answered Sep 3, 2013 at 8:50

falsetru

371k69 gold badges769 silver badges659 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Nilani Algiriyage Over a year ago

Thanks falsetru!...:)This is what I expected!

Collectives™ on Stack Overflow

Extract bot name from user-agent string-Python

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related