Is it possible to read FTP files without writing them using Python?

Question

I am trying to read files using Python's ftplib without writing them. Something roughly equivalent to:

def get_page(url):
    try:
        return urllib.urlopen(url).read()
    except:
        return ""

but using FTP.

I tried:

def get_page(path):
    try:
        ftp = FTP('ftp.site.com', 'anonymous', 'passwd')
        return ftp.retrbinary('RETR '+path, open('page').read())
    except:
        return ''

but this doesn't work. The only examples in the docs involve writing files using the ftp.retrbinary('RETR README', open('README', 'wb').write) format. Is it possible to read ftp files without writing first?

Martin Prikryl · Accepted Answer · 2021-06-14 10:17:10Z

70

Well, you have the answer right in front of you: The FTP.retrbinary method accepts as second parameter a reference to a function that is called whenever file content is retrieved from the FTP connection.

Here is a simple example:

#!/usr/bin/env python
from ftplib import FTP

def writeFunc(s):
  print "Read: " + s

ftp = FTP('ftp.kernel.org') 
ftp.login()
ftp.retrbinary('RETR /pub/README_ABOUT_BZ2_FILES', writeFunc)

You should implement writeFunc so that it actually appends the data read to an internal variable, something like this, which uses a callable object:

#!/usr/bin/env python
from ftplib import FTP

class Reader:
  def __init__(self):
    self.data = ""
  def __call__(self,s):
     self.data += s

ftp = FTP('ftp.kernel.org') 
ftp.login()
r = Reader()
ftp.retrbinary('RETR /pub/README_ABOUT_BZ2_FILES', r)

print r.data

Update: I realized that there is a module in the Python standard library that is meant for this kind of things, BytesIO:

#!/usr/bin/env python
from ftplib import FTP
from io import BytesIO

ftp = FTP('ftp.kernel.org') 
ftp.login()
r = BytesIO()
ftp.retrbinary('RETR /pub/README_ABOUT_BZ2_FILES', r.write)

print r.getvalue()

edited Jun 14, 2021 at 10:17

Martin Prikryl

205k64 gold badges560 silver badges1.1k bronze badges

answered Jun 26, 2012 at 14:17

daniel kullmann

14.1k9 gold badges53 silver badges69 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

aensm Over a year ago

Awesome, thanks! I didn't realize the callback could be a user defined function

Tim Richardson Over a year ago

For Python 3, retrbinary requires BytesIO, because it returns bytes, not string. If you want StringIO, try ftp.retrlines()

jimmymcheung Over a year ago

@TimRichardson I tried ftp.retelines(f'RETR {filename}') returns BrokenPipeError, trackback shows there're some problem in file encoding (at self.sock.sendall(line.encode(self.encoding)) in putline). The file I'm trying to get is md5 hash (link: ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed23n0003.xml.gz.md5)

Tim Richardson Over a year ago

@jimmymcheung you have a typo, it is not "retelines"

jimmymcheung Over a year ago

@TimRichardson thanks, but it was the typo here. I checked the script and it was correct (tho now I just use retrbinary and comment out the retrlines script, because I read the documentation more carefully and found retrlines is not read file in "string" mode)

Dysmas · Accepted Answer · 2022-12-29 09:11:17Z

0

With retrlines, if the file retrieved is in text mode, things can be even more simple and you get directly a list, like readlines :

#!/usr/bin/env python
from ftplib import FTP
    
ftp = FTP('ftp.kernel.org') 
ftp.login()
r = []
ftp.retrlines('RETR /pub/README_ABOUT_BZ2_FILES', r.append)

print(r)

for the reverse process (send from BytesIO) see How can I send a StringIO via FTP in python 3?

edited Dec 29, 2022 at 9:11

answered Dec 29, 2022 at 9:00

Dysmas

15111 bronze badges

Collectives™ on Stack Overflow

Is it possible to read FTP files without writing them using Python?

2 Answers 2

5 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related