Regex Parse Email Python

Question

So here's a sample of an e-mail I want to parse, to extract ONLY its body.

RECEIVED: 2012-11 20 09:59:24
SUBJECT: Get Boddy
--- Original Sender: Mark Twain. ---

----- Original Message -----
From: Boby Indo
To: Obum Hunter 
At: 11/20  9:59:22

***NEW ISSUE SUPPORTED THROUGH UNIVERSALITY   vs 104-13 on AY 3s JAN   
10+BB {MYXV ABC 4116    SM  MYXV YA 102-15 <DO>} | 2010/11 4.0s             4.0s
6+ BB {MYXV ABC 4132    NS  MYXV YT 102-22 <DO>} | 2010 4.5s                4.5s
ABO 2006-OP1 M1     00442PAG5     19-24      p5 
***SECOND SUPPORTED TRHOUGH INVERSALITY GEVINGS                      
10+BB  {NXTW VXA 4061   SL  MYXV YA 103-22 <DO>} | 11 wala 3.5s             3.5s
10+BB  {NXTW VXA 12-47  SP  MYXV YA 106-20 <DO>} | 22 wala 4.0s             4.0s

------------------------------------------------------------
© Copyright 2012 The Ridgly Group, Inc. All rights reserved. See
http://www.examply.html for important information disclosure.

Here's what I expect:

***NEW ISSUE SUPPORTED THROUGH UNIVERSALITY   vs 104-13 on AY 3s JAN   
10+BB {MYXV ABC 4116    SM  MYXV YA 102-15 <DO>} | 2010/11 4.0s             4.0s
6+ BB {MYXV ABC 4132    NS  MYXV YT 102-22 <DO>} | 2010 4.5s                4.5s
ABO 2006-OP1 M1     00442PAG5     19-24      p5 
***SECOND SUPPORTED TRHOUGH INVERSALITY GEVINGS                      
10+BB  {NXTW VXA 4061   SL  MYXV YA 103-22 <DO>} | 11 wala 3.5s             3.5s
10+BB  {NXTW VXA 12-47  SP  MYXV YA 106-20 <DO>} | 22 wala 4.0s             4.0s

It would be nice it the ***lines could be eliminated as well.

And here's what I got so far (?P<header>[\S+\s]+At:.*)\n+(?P<body>[\S+\s]([\d\.\d]+[a-z]?$)) .This doesn't seem to do a good job, as it grabs the dash-lines after the last 4.0s and get stuck at at the non-ascii character ©. Thanks!

PS: I think the best approach would be to cutoff the header and the tail of the e-mail with groups. So what's left would be the body. Because the header and tail always stay the same, but the body changes on different e-mails. The solution needs not be to specific for an e-mail.

Have you tried docs.python.org/2/library/email.parser.html#module-email.parser? — georg
– georg, Commented Dec 22, 2012 at 8:50
If this is some sort of line-oriented record format, why don't you just extract the lines which are in this format? Any line with two | characters looks like a starting point, although you might want to add more constraints in order to avoid stray false positives. — tripleee
– tripleee, Commented Dec 22, 2012 at 8:51
I looked into it but it may not help here since I have a header and a footer, and just want to extract the body. They don't have footer function. I think. — Mike
– Mike, Commented Dec 22, 2012 at 8:52
@tripleee: I think I'm on the right track here. I just getting stuck with the last dash-line and that non-ascii character. My regex grabs the header fine. — Mike
– Mike, Commented Dec 22, 2012 at 8:57
The sample doesn't look like a valid RFC822 message anyway. Could be copy/paste error, of course, but an RFC822 parser is not going to be much help in getting at details within the body. — tripleee
– tripleee, Commented Dec 22, 2012 at 9:02

score 1 · Accepted Answer · 2012-12-25 14:14:20Z

1

See if this works for you, the lines that you want start with digits followed by a plus sign:

^[0-9]*\+.*$

This will match the expected output:

\*{3}[^\*]*(?:(?=\*{3})|(?=^-*$))

^ Matches the beginning of the string.

[0-9] Matches any single character in the range 0-9.

* Matches 0 or more of the preceeding token. This is a greedy match, and will match as many characters as possible before satisfying the next token.

\+ Matches a + character.

. Matches any character.

$ Matches the end of the string.

#!/usr/bin/env python
#-*- coding:utf-8 -*-
import re
with open("/path/to/file", "r") as fileInput:
    listLines = [   line.strip()
                    for line in fileInput.readlines()
                    if re.match("^[0-9]*\+.*$", line)
                    ] 


for line in listLines:
    print line

>>> 10+BB {MYXV ABC 4116    SM  MYXV YA 102-15 <DO>} | 2010/11 4.0s             4.0s
>>> 6+ BB {MYXV ABC 4132    NS  MYXV YT 102-22 <DO>} | 2010 4.5s                4.5s
>>> 10+BB  {NXTW VXA 4061   SL  MYXV YA 103-22 <DO>} | 11 wala 3.5s             3.5s
>>> 10+BB  {NXTW VXA 12-47  SP  MYXV YA 106-20 <DO>} | 22 wala 4.0s             4.0s

Updated to meet new requirements:

#!/usr/bin/env python
#-*- coding:utf-8 -*-
import re
with open("/path/to/file", "r") as fileInput:
    regex = re.compile(r"\*{3}[^\*]*?(?:(?=^-*$)|(?=\*))", re.MULTILINE)

    listMsg = [ [   line.strip()
                    for line in message.split("\n")
                    if not line.startswith("*") and line.strip()
                    ]
                for message in regex.findall(fileInput.read())
                ]

>>> 10+BB {MYXV ABC 4116    SM  MYXV YA 102-15 <DO>} | 2010/11 4.0s             4.0s
>>> 6+ BB {MYXV ABC 4132    NS  MYXV YT 102-22 <DO>} | 2010 4.5s                4.5s
>>> ABO 2006-OP1 M1     00442PAG5     19-24      p5
>>> 10+BB  {NXTW VXA 4061   SL  MYXV YA 103-22 <DO>} | 11 wala 3.5s             3.5s
>>> 10+BB  {NXTW VXA 12-47  SP  MYXV YA 106-20 <DO>} | 22 wala 4.0s             4.0s

Updated to extract the whole body of the email:

#!/usr/bin/env python
#-*- coding:utf-8 -*-
import re
with open("/path/to/file", "r") as fileInput:
    regex = re.compile(r"(?<=^At:)([^\n\r]*)(.*?)(?=^-*-$)", re.MULTILINE|re.DOTALL)

    print regex.search(fileInput.read()).groups()[1]

>>> ACE 2006-OP1 ZZ 111111111 19-24 Z5 ZZW 2012-0P1 SD 222222222 77-00 150
>>> ***NEW ISSUE SUPPORTED THROUGH UNIVERSALITY   vs 104-13 on AY 3s JAN   
>>> 10+BB {MYXV ABC 4116    SM  MYXV YA 102-15 <DO>} | 2010/11 4.0s             4.0s
>>> 6+ BB {MYXV ABC 4132    NS  MYXV YT 102-22 <DO>} | 2010 4.5s                4.5s
>>> ABO 2006-OP1 M1     00442PAG5     19-24      p5 
>>> ***SECOND SUPPORTED TRHOUGH INVERSALITY GEVINGS                      
>>> 10+BB  {NXTW VXA 4061   SL  MYXV YA 103-22 <DO>} | 11 wala 3.5s             3.5s
>>> 10+BB  {NXTW VXA 12-47  SP  MYXV YA 106-20 <DO>} | 22 wala 4.0s             4.0s

edited Dec 25, 2012 at 14:14

answered Dec 22, 2012 at 9:01

user1006989

Sign up to request clarification or add additional context in comments.

14 Comments

Mike Over a year ago

I'm not sure I understand what you meant. Will you put down your complete regex solution, please?

user1006989 Over a year ago

@Mike checkout my answer, I updated it with a definition for the patterns

Mike Over a year ago

X.Jacobs: Did you run that code? Here's what I'm getting: ['+', '+', '+', '+', '+', '+', '+', '+', '+', '+'] after running you code. Thanks!

user1006989 Over a year ago

@Mike flawless in my machine, python 2.7, please consider upvoting/accepting responses you might find useful

Mike Over a year ago

X.Jacob: Sorry, I was still figuring out why I was having trouble. I just realized I had omitted one line in my version of text. Please consider tweaking your code accordingly if you don't mind. Thanks!

|

Mike · Accepted Answer · 2012-12-22 11:57:02Z

1

>>> s="""RECEIVED: 2012-11 20 09:59:24
... SUBJECT: Get Boddy
... --- Original Sender: Mark Twain. ---
... 
... ----- Original Message -----
... From: Boby Indo
... To: Obum Hunter 
... At: 11/20  9:59:22
... 
... ***NEW ISSUE SUPPORTED THROUGH UNIVERSALITY   vs 104-13 on AY 3s JAN   
... 10+BB {MYXV ABC 4116    SM  MYXV YA 102-15 <DO>} | 2010/11 4.0s             4.0s
... 6+ BB {MYXV ABC 4132    NS  MYXV YT 102-22 <DO>} | 2010 4.5s                4.5s
... ABO 2006-OP1 M1     00442PAG5     19-24      p5 
... ***SECOND SUPPORTED TRHOUGH INVERSALITY GEVINGS
... 10+BB  {NXTW VXA 4061   SL  MYXV YA 103-22 <DO>} | 11 wala 3.5s             3.5s
... 10+BB  {NXTW VXA 12-47  SP  MYXV YA 106-20 <DO>} | 22 wala 4.0s             4.0s
... 
... ------------------------------------------------------------
... © Copyright 2012 The Ridgly Group, Inc. All rights reserved. See
... http://www.examply.html for important information disclosure."""
>>> r=r'(?P<header>\*\*\*[^\n]*)\n(?P<body>[\s\S]*?\n)\n'
>>> for match in re.finditer(r, s):
...     print match.group('body')
... 
10+BB {MYXV ABC 4116    SM  MYXV YA 102-15 <DO>} | 2010/11 4.0s             4.0s
6+ BB {MYXV ABC 4132    NS  MYXV YT 102-22 <DO>} | 2010 4.5s                4.5s

10+BB  {NXTW VXA 4061   SL  MYXV YA 103-22 <DO>} | 11 wala 3.5s             3.5s
10+BB  {NXTW VXA 12-47  SP  MYXV YA 106-20 <DO>} | 22 wala 4.0s             4.0s

edited Dec 22, 2012 at 11:57

Mike

1252 silver badges10 bronze badges

answered Dec 22, 2012 at 9:01

Nolen Royalty

18.7k4 gold badges43 silver badges51 bronze badges

3 Comments

Mike Over a year ago

Interesting! Never used finditer() before. Let me run it and and found out. I'm running python 2.7. Thanks in advance!

Mike Over a year ago

What version of python are you using? Here's the error I'm getting : SyntaxError: Non-ASCII character '\xc2' which is by the way the same error I was getting. Thanks!

Mike Over a year ago

For some reason your solution is not general. i.e if I add another footer (© Copyright 2012...), it will then grab that footer. Add a couple of sections of footer separated by dash-lines if you will, and run it again to see what I'm talking about. Thanks!

Collectives™ on Stack Overflow

Regex Parse Email Python

2 Answers 2

14 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

14 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related