Bash command won't execute in Jupyter Notebook

Question

I am trying to execute a Bash command from inside of a Jupyter Notebook that writes a random number to some files. The command I'd like to issue is

echo $(( RANDOM )) &> {output_files[i]}

where output_files is a Python list of output filenames. I can't figure out how to put the Python variable output_files[i] into a Bash command successfully, so instead I create the entire command as a string and try to execute that through the Notebook:

# Create a list of 5 filenames
output_files = [os.path.join(os.getcwd(), 'random-%s.txt' % i) for i in range (5)]

# Write random number to each file
for i in range (5):
    command = "echo $(( RANDOM )) &> " + output_files[i]
    print(command)
    !{command}

When I try this, the files ./random-0.txt etc. are created, but no number is written to them - they are empty. When I run the created commands directly in the terminal, though, it works exactly as expected. The files are created and contain a single random number each. The print(command) line produces

echo $(( RANDOM )) &> /filepath/random-0.txt
0
echo $(( RANDOM )) &> /filepath/random-1.txt
echo $(( RANDOM )) &> /filepath/random-2.txt
echo $(( RANDOM )) &> /filepath/random-3.txt
echo $(( RANDOM )) &> /filepath/random-4.txt

Web searches for this problem give hits for several similar questions on Stack Overflow as well as blogs, but the result of all of them is either "Bash commands don't exist on Windows" or "You can use '!' to execute Bash commands in Jupyter Notebooks!" with no further information about how it works so that I could debug an issue like this.

How do I make this work?

As an aside, the &> redirection is not adding anything useful over plain > and in fact, in the unlikely even that you actually get an error message,. you probably don't want it to overwrite your output file. — tripleee
– tripleee, Commented Feb 18, 2019 at 13:17

Inian · Accepted Answer · 2019-02-18 13:06:43Z

3

Why not do the whole thing in python itself? Use the random module the randint() function to generate your numbers. Define a range of your choice and include it within the (..)

import os
import random
output_files = [os.path.join(os.getcwd(), 'random-%s.txt' % i) for i in range (5)]
for of in output_files:
    with open(of, "w") as text_file:
        text_file.write(str(random.randint(99,1000)))

answered Feb 18, 2019 at 13:06

Inian

87k15 gold badges166 silver badges183 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Darien Marks Over a year ago

I'm comparing two methods, one of which is Bash-based, and I need to keep as much the same as possible. But it doesn't matter why, the question as asked is well-defined, with a well-defined answer. If you can answer it as asked, please do.

Heiko Becker · Accepted Answer · 2019-02-18 12:57:43Z

2

Hello and welcome to Stack Overflow,

As far as I can tell, your issue does not arise form the bash command execution with ! but rather from the fact that you do not "replace" output_files[i] by its actual content. When you run the loop

# Write random number to each file
for i in range (5):
    command = "echo $(( RANDOM )) &> {output_files[i]}"

in each iteration, command will be the literal string "echo $(( RANDOM )) &> {output_files[i]}". As you did in your first line, you should use string substitution to replace a placeholder with the content of output_files[i]:

# Write random number to each file
for i in range (5):
    command = ("echo $(( RANDOM )) &> {%s}"%output_files[i])

To debug such an issue, it is usually a good start to inspect the value of the variable in question. The easiest way of doing so is by adding print (x) lines where x is the variable whose content you want to inspect.

answered Feb 18, 2019 at 12:57

Heiko Becker

6164 silver badges17 bronze badges

4 Comments

Darien Marks Over a year ago

The error you point out was only an error in how I typed the question, it wasn't in the code. I was correcting it while you were typing your answer, I think. print() does confirm that command is being created exactly as the string(s) I want. I tried your suggestion for constructing command, and the result is the same: the five files random-i.txt are created, but they do not contain numbers.

Heiko Becker Over a year ago

Can you maybe update the question with the content of command for each of the loop bodys? Just to be sure.

Heiko Becker Over a year ago

Nevermind, I think the answer by @Inian is a better way to approach this.

Darien Marks Over a year ago

I've edited the question with this info. Also, no, it isn't.

tripleee · Accepted Answer · 2019-02-19 13:31:29Z

2

In reaction to one of the answers here, let me show how to do this with subprocess.check_call().

If you want to use a shell redirection in subprocess, you have to have a shell:

for i in range(5):
    subprocess.check_call('echo "$RANDOM" >random-{}.txt'.format(i), shell=True)

You could superficially try to get rid of the shell=True ... but then echo and $RANDOM are shell features, too.

A pure-Bash solution would be

!for i in {0..4}; do printf "%s\n" "$RANDOM" >random-"$i".txt; done

(As an aside, the numeric context in $((RANDOM)) is not wrong, but entirely superfluous.)

You can interpolate a Python expression into the ! command line with {...}. The following also works, for example:

!{';'.join(['echo $RANDOM >random-{}.txt'.format(i) for i in range(5)])}

In iPython I can produce the results I expect with the construct you tried:

in [15]: for i in range(2):
    ...:     c='echo $RANDOM >random-{}.txt'.format(i)
    ...:     !{c}
    ...:

in [16]: !tail random-*.txt
==> random-0.txt <==
15637

==> random-1.txt <==
32475

If Jupyter is using the iPython kernel, I'm thinking this ought to work for you, too.

Notice also how I used plain old > redirection rather than &> - there really really really is no sane reason to want error messages to be redirected to the files; and if a file can't be created, you have absolutely no idea what failed or why, or even that it failed. (I suspect this is really the reason for your problems, actually. Permission denied? Disk full? We can't know.)

I would suggest you use Python to perform this simple task, though. Running a subprocess to perform something which is relatively easy in Python itself is inefficient and clumsy as well as somewhat brittle.

edited Feb 19, 2019 at 13:31

answered Feb 18, 2019 at 14:14

tripleee

192k37 gold badges318 silver badges367 bronze badges

4 Comments

jwalton Over a year ago

Nice solutions. I have had success with the second two commands, however the first command raises a FileNotFoundError for me.

tripleee Over a year ago

Oh I left old code into the answer, sorry about that; hopefully fixed now.

jwalton Over a year ago

Yes, this works now. However, it's still missing a closing bracket. Should be subprocess.check_call('echo "$RANDOM" >random-{}.txt'.format(i), shell=True)

tripleee Over a year ago

Yeah, thanks for your edit suggestion, it got rejected before I got a chance to accept it.

Collectives™ on Stack Overflow

Bash command won't execute in Jupyter Notebook

3 Answers 3

1 Comment

4 Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

4 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related