How do I provide inline input to an IPython (notebook) shell command?

Question

I want to put together an IPython notebook with some shell commands and their input. In the bash prompt I can use "here-document" syntax:

bash-3.2$ mysql -u root <<END_IPUT
> use mydb;
> show tables;
> END_INPUT

How do I get the same effect in IPython, and specifically in a jupyter notebook? I know how to execute shell commands as IPython as "line magics" or "cell magics", e.g.:

In [7]:  !! ls -tF
Out[7]:  ['Demo-notebook.ipynb',
          'createdb.sql',
          ...

I've looked at IPython as a system shell, which shows how to enable some syntactic niceties. After the following, I can run system commands without prepending ! or !!

# Turn everything in $PATH into an alias; 
# then enable calling aliases without ! or %
%rehashx      
%autocall 2

But none of this helps with providing input to these commands inline: The here-document syntax is invalid in IPython, and results in a python SyntaxError. So how do I do it?

Have you tried the bash cell magic? Start a cell with the line %%bash, and put bash code in the rest of it. — Thomas K
– Thomas K, Commented May 17, 2016 at 17:07
Good suggestion, thanks! I'd already found %%sx, which does the same I think. But the documentation for %bash led me to %%script, which is exactly what I was looking for. In the meantime I also found an extension specifically for sql, which solved my immediate problem even better. But if you want to write an answer expanding on your suggestion, I'll accept it. — alexis
– alexis, Commented May 17, 2016 at 21:28
@Thomas, before you delete my tag for a third time, look at how many people follow each one. If ipython-notebook is such a problem, please take up the process to have the tags merged. — alexis
– alexis, Commented May 22, 2016 at 23:17
I am trying to merge the tags, but I can't do that until the new tag has 80% as many uses as the old. I am periodically sweeping new questions over to the new tag to boost its numbers. Thanks. — Thomas K
– Thomas K, Commented May 23, 2016 at 10:21
I see. Well, please stop deleting my tags. Having both should be good enough for you. (And just add jupyter-notebook to other questions.) And making the tags themselves synonyms, of course. — alexis
– alexis, Commented May 23, 2016 at 10:25

Thomas K · Accepted Answer · 2016-05-18 08:37:47Z

8

The bash cell magic allows you to enter multiple lines of bash. Just start any cell with %%bash:

%%bash
echo "This is bash"
ls

If you're using a heredoc to pipe text into another program, you can also use the script cell magic:

%%script bc
2+3

There may also be other cell magics which address your problem more specifically!

answered May 18, 2016 at 8:37

Thomas K

40.7k7 gold badges88 silver badges89 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Mahmoud Over a year ago

there is few small differences between ! and %%bash when running a command. 1. %%bash doesn't display ASCII colors, while color appear fine using!. 2. %%bash collect the output and flush them on the screen once while ! displays the output one by one as it executes.

alexis · Accepted Answer · 2018-02-26 22:41:54Z

With some more research, and with the help of @ThomasK's tip (see accepted answer), I found a couple of ways to do this.

One solution is the cell magic %sx. It executes the cell contents as a bash script, and captures and returns the output as a list of lines. Sometimes handy, sometimes not.
```
In[1]:  %%sx
        echo Hello, world
        cat -n <<DATA
        this
        and that
        DATA

Out[1]: ['Hello, world', '     1\tthis', '     2\tand that']
```
Change that to %%bash, as @ThomasK suggested, and the output is printed out, not returned. To capture it for further processing, ipython provides the --out parameter. But then I need to print it myself-- it won't show up by default.
```
In[1]:  %%bash --out var
        echo Hello, world
        echo "   Again"

In[2]:  var

Out[2]: 'Hello, world\n   Again\n'
```
%%bash is actually shorthand for %%script bash, billed as working "like the #! line in a script" (see the help with %%script?). Which it does. So here's how to dispense with the here-document syntax and have any program read the cell contents as input. %%script also accepts the --out parameter.
```
        %%script --out var mysql -u root -p XYZ
        USE somedb;
        SELECT * FROM users
             WHERE passwd IS NULL\G
```

This is nice enough, but I ended up not using it because my use case is mysql, and eventually I discovered that there's a great third-party ipython-sql extension that I can use instead:

%load_ext sql
%sql mysql+pymysql://user:passwd@localhost/somedb

This presupposes installing the extension, with pip install ipython-sql. For mysql, I also needed pip install pymysql. After the above setup, I just talk to the database like this

In[1]:  %%sql 
        SELECT * FROM users
            WHERE passwd IS NULL

Out[1]: 1 rows affected.
        ...

The returned table is actually drawn as a notebook table with borders, which is kinda nice. And it is made available as a Pandas dataframe (a smart list), which I can grab from _.

Could I do: %%bash /usr/bin/env ipython in order to run an exported notebook.py at the command line?

Collectives™ on Stack Overflow

How do I provide inline input to an IPython (notebook) shell command?

2 Answers 2

1 Comment

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related