20

I have a CLI script and want it to read data from a file. It should be able to read it in two ways :

  • cat data.txt | ./my_script.py
  • ./my_script.py data.txt

—a bit like grep, for example.

What I know:

  • sys.argv and optparse let me read any args and options easily.
  • sys.stdin let me read data piped in
  • fileinput make the full process automatic

Unfortunately:

  • using fileinput uses stdin and any args as input. So I can't use options that are not filenames as it tries to open them.
  • sys.stdin.readlines() works fine, but if I don't pipe any data, it hangs until I enter Ctrl + D
  • I don't know how to implement "if nothing in stdin, read from a file in args" because stdin is always True in a boolean context.

I'd like a portable way to do this if possible.

Improve this question
1
  • note: you can determine if stdin exists (ie your script is running in a pipe), as detailed here Commented Feb 2, 2022 at 19:50

6 Answers 6

Reset to default
20

Argparse allows this to be done in a fairly easy manner, and you really should be using it instead of optparse unless you have compatibility issues.

The code would go something like this:

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--input', type = argparse.FileType('r'), default = '-')

Now you have a parser that will parse your command line arguments, use a file if it sees one, or use standard input if it doesn't.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Thanks to you as well. I'm learning a lot today! Anyway, don't you think there is a way to do that with the standard lib? If not, I'm ok with argparse. But optparse does exist...
argparse is pretty tiny, is pure python code and is also nicer to use than optparse. While I wouldn't normally want to add a new dependancy to a project just to read in the command line options, the three factors above have made argparse more than worthwhile in my experience.
@S.Lott I didn't know that, thanks for mentioning. In any case, I recommended argparse for its overall benefits anyway.
argparse is now part of the standard lib; in fact, it is the new recommended argument parsing library. It's also awesome, and worth learning (which doesn't take very long).
Or you can just do: parser.add_argument('--input', dest=input, type = file) since FileType('R') will leave you with an open file afterwards. You can check if args.input and read from stdin if not.
13

Process your non-filename arguments however you'd like, so you wind up with an array of non-option arguments, then pass that array as the parameter to fileinput.input():

import fileinput
for line in fileinput.input(remaining_args):
    process(line)
Improve this answer

3 Comments

I like that, it seems highly effective. Is there anything bad I'm missing?
I believe that this will provide behaviour similar to other Unix commands. In another comment you said you're slightly concerned about the hang-until-input effect when no arguments are given; unless you take steps to notice when no arguments are passed, this will still happen. There's no reason you shouldn't catch this case, though, as passing '-' as a parameter will still read from stdin.
I will use a combination of Ignocio's advice and Andrew's solution.
9

For unix/linux you can detect whether data is being piped in by looking at os.isatty(0)

$ date | python -c "import os;print os.isatty(0)"
False
$ python -c "import os;print os.isatty(0)"
True

I'm not sure there is an equivalent for Windows.

edit Ok, I tried it with python2.6 on windows XP

C:\Python26>echo "hello" | python.exe -c "import os;print os.isatty(0)"  
False

C:\Python26> python.exe -c "import os;print os.isatty(0)"  
True

So maybe it it not all hopeless for windows

Improve this answer

13 Comments

Thanks to both of you. Since I lack of the necessary knowledge to choose between your answer and Ignocio's, what are the drawbacks of each ? Them seem both valid.
@e-satis: What happens if no filename is passed as an argument? When you can answer this question you'll know what you need to do.
For now, this seems to work so I accept it. Thanks again. I'm still interested in knowing the issues with what I'm doind. What is a tty, and why does this work? When will it not?
Note that this is defined as Unix-only in the docs, so you hurt portability to Windows and possibly other operating systems. It also assumes you'll never want to just paste data in, which is a silly assumption.
@e-satis: A tty technically stands for teletype terminal. These days it really just means it's an interactive terminal. The particular concept used by isatty() is non-portable outside Unix, however.
|
4

I'm a noob, so this might not be a good answer, but I'm trying to do the same thing (allow one or more files on the command line, default to STDIN otherwise).

The final combo I put together:

parser = argparse.ArgumentParser()
parser.add_argument("infiles", nargs="*")
args = parser.parse_args()

for line in fileinput.input(args.infiles):
    process(line)

This seems like the only way to get all the desired behavior in one elegant package, without requiring named args. Just like unix commands are used as such:

cat file1 file2
wc -l < file1

Not:

cat --file file1 --file file2

Would appreciate feedback/confirmation from veteran idiomatic Pythonistas to make sure I've got the best answer. Haven't seen this complete solution mentioned anywhere else, just fragments.

Improve this answer

1 Comment

One more thing -- if you don't want to depend on the next guy already knowing that fileinput.input() magically defaults to stdin when it gets an empty list, you can add ', default="-"' to the call to add_argument(). It doesn't change anything, but it makes the logic perfectly explicit.
3

There is no reliable way to detect if sys.stdin is connected to anything, nor is it appropriate do so (e.g., the user wants to paste the data in). Detect the presence of a filename as an argument, and use stdin if none is found.

Improve this answer

9 Comments

Thanks to both of you. Since I lack of the necessary knowledge to choose between your answer and gnibbler's, what are the drawbacks of each ? Them seem both valid.
Oh, just tried this approach but their is a problem: if there is no files and no stdin, I end up reading stdin and it hangs. How can I write an error message to the user to tell him to provide data?
Note well that the hang-until-EOF is inevitable here if someone runs it without a filename -- this is standard behavior in the Unix world. see also grep, cat, etc.. If this isn't acceptable, the only portable way you can avoid it is to use another typical convention wherein providing a filename of '-' means 'read from stdin' (or write to stdout).
Actually, I just fired grep without any input. I just fired grep and sed without any input and they don't hang.
Ok you are right. Cat does wait for input But how does grep and sed do? I give them no args, I guess they check, they see no arg, then they look for stdin, why do they manage to display the message ?
|
3

You can use this function to detect if the input is from a pipeline or not.

sys.stdin.isatty()

It returns false if the input is from pipeline or true otherwise.

Improve this answer

2 Comments

The behavior of your program should not depend significantly on whether stdin is connected to a tty. If no filenames are given, simply read from stdin, whether it is from a tty or a pipe.
This really helped me out. I needed to know whether my data was piped or tty for a particular project and this was the only good solution I could use.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.