11

I am trying to read a pdf using python and the content has many newline (crlf) characters. I tried removing them using below code:

from tika import parser

filename = 'myfile.pdf'
raw = parser.from_file(filename)
content = raw['content']
content = content.replace("\r\n", "")
print(content)

But the output remains unchanged. I tried using double backslashes also which didn't fix the issue. can someone please advise?

4
  • 2
    What sort of data structure is "content"? Post a sample of it to help us help you? Commented Feb 19, 2019 at 7:34
  • This example is not reproducible without knowing what content contains. Commented Feb 19, 2019 at 7:35
  • You can't just read a literal PDF file and make text replacements like this. You need a Python library which can parse PDF content. Commented Feb 19, 2019 at 7:37
  • 1
    content is a string. I checked it using type(content). @TimBiegeleisen I use the text after parsing the file from tika as you can see in code. Commented Feb 19, 2019 at 7:40

8 Answers 8

16
content = content.replace("\\r\\n", "")

You need to double escape them.

Sign up to request clarification or add additional context in comments.

6 Comments

You don't need to escape \r\n, because they are already valid character literals.
To be sure for all cases (windows or unix ascii string) please use content = content.replace("\r", "").replace("\n", "")
In case someone is looking at this in the future ... I had to reference stackoverflow.com/questions/47178459/… to really overcome my issue because python converts windows crlf behind the scenes according to the answer in that post
Most upvoted answer has no idea what it is talking about. "You need to double escape them." Well, alrighty then. Meanwhile, the actual answer that addresses this topic by actually importing tika was ignored. </sigh>
@CecilCurry ok, but if those characters are literally in there (not encoded), then you'd need to replace those characters? So, I do know what I am talking about ?
|
4

I don't have access to your pdf file, so I processed one on my system. I also don't know if you need to remove all new lines or just double new lines. The code below remove double new lines, which makes the output more readable.

Please let me know if this works for your current needs.

from tika import parser

filename = 'myfile.pdf'

# Parse the PDF
parsedPDF = parser.from_file(filename)

# Extract the text content from the parsed PDF
pdf = parsedPDF["content"]

# Convert double newlines into single newlines
pdf = pdf.replace('\n\n', '\n')

#####################################
# Do something with the PDF
#####################################
print (pdf)

Comments

3

You can also just use

text = '''
As she said these words her foot slipped, and in another moment, splash! she
was up to her chin in salt water. Her first idea was that she had somehow
fallen into the sea, “and in that case I can go back by railway,”
she said to herself.”'''

text = ' '.join(text.splitlines())

print(text)
# As she said these words her foot slipped, and in another moment, splash! she was up to her chin in salt water. Her first idea was that she had somehow fallen into the sea, “and in that case I can go back by railway,” she said to herself.”

Comments

2

If you are having issues with different forms of line break, try the str.splitlines() function and then re-join the result using the string you're after. Like this:

content = "".join(l for l in content.splitlines() if l)

Then, you just have to change the value within the quotes to what you need to join on. This will allow you to detect all of the line boundaries found here. Be aware though that str.splitlines() returns a list not an iterator. So, for large strings, this will blow out your memory usage. In those cases, you are better off using the file stream or io.StringIO and read line by line.

Comments

1
print(open('myfile.txt').read().replace('\n', ''))

1 Comment

So what is this meant to do? How does this answer the question? Please edit your answer and explain the answer. Additionally please read How to Answer
1

When you write something like t.replace("\r\n", "") python will look for a carriage-return followed by a new-line.

Python will not replace carriage returns by themselves or replace new-line characters by themselves.

Consider the following:

t = "abc abracadabra abc"
t.replace("abc", "x")
  • Will t.replace("abc", "x") replace every occurrence of the letter a with the letter x? No

  • Will t.replace("abc", "x") replace every occurrence of the letter b with the letter x? No

  • Will t.replace("abc", "x") replace every occurrence of the letter c with the letter x? No

What will t.replace("abc", "x") do?

t.replace("abc", "x") will replace the entire string "abc" with the letter "x"

Consider the following:

test_input = "\r\nAPPLE\rORANGE\nKIWI\n\rPOMEGRANATE\r\nCHERRY\r\nSTRAWBERRY"

t = test_input
for _ in range(0, 3):
    t = t.replace("\r\n", "")
    print(repr(t))

result2 = "".join(test_input.split("\r\n"))
print(repr(result2))

The output sent to the console is as follows:

'APPLE\rORANGE\nKIWI\n\rPOMEGRANATECHERRYSTRAWBERRY'
'APPLE\rORANGE\nKIWI\n\rPOMEGRANATECHERRYSTRAWBERRY'
'APPLE\rORANGE\nKIWI\n\rPOMEGRANATECHERRYSTRAWBERRY'
'APPLE\rORANGE\nKIWI\n\rPOMEGRANATECHERRYSTRAWBERRY'

Note that:

  • str.replace() replaces every occurrence of the target string, not just the left-most occurrence.
  • str.replace() replaces the target string, but not every character of the target string.

If you want to delete all new-line and carriage returns, something like the following will get the job done:

in_string = "\r\n-APPLE-\r-ORANGE-\n-KIWI-\n\r-POMEGRANATE-\r\n-CHERRY-\r\n-STRAWBERRY-"

out_string = "".join(filter(lambda ch: ch not in "\n\r", in_string))

print(repr(out_string))
# prints -APPLE--ORANGE--KIWI--POMEGRANATE--CHERRY--STRAWBERRY-

Comments

0
#write a file 
enter code here
write_File=open("sample.txt","w")
write_File.write("line1\nline2\nline3\nline4\nline5\nline6\n")
write_File.close()

#open a file without new line of the characters
open_file=open("sample.txt","r")
open_new_File=open_file.read()
replace_string=open_new_File.replace("\n",." ")
print(replace_string,end=" ")
open_file.close()

OUTPUT

line1 line2 line3 line4 line5 line6

1 Comment

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
0

regex will work in this case

r'(\r\n)+ pattern matches one or more occurrences of \r\n and then it replaced with single \r\n

import re

content = '\r\n\r\n\r\n\r\n\r\ntest'
content = re.sub(r'(\r\n)+', r'\r\n', content)  # '\r\ntest'

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.