bash commands in parallel

Question

I want to have two youtube-dl processes (or as much as possible )to run in parallel. Please show me how. thanks in advance.

#!/bin/bash
#package: youtube-dl axel

#file that contains youtube links
FILE="/srv/backup/temp/youtube.txt"

#number of lines in FILE
COUNTER=`wc -l $FILE | cut -f1 -d' '`

#download destination
cd /srv/backup/transmission/completed

if [[ -s $FILE ]]; then
  while [ $COUNTER -gt 0 ]; do

    #get video link
    URL=`head -n 1 $FILE`

    #get video name
    NAME=`youtube-dl --get-filename -o "%(title)s.%(ext)s" "$URL" --restrict-filenames`

    #real video url
    vURL=`youtube-dl --get-url $URL`

    #remove first link
    sed -i 1d $FILE

    #download file
    axel -n 10 -o "$NAME" $vURL &

    #update number of lines
    COUNTER=`wc -l $FILE | cut -f1 -d' '`
  done
else
  break
fi

@MattBall what about NAME and vURL initialization? how can I run them in parallel? — ihm
– ihm, Commented Feb 15, 2013 at 18:14

Ole Tange · Accepted Answer · 2013-02-16 01:42:59Z

2

This ought to work with GNU Parallel:

cd /srv/backup/transmission/completed
parallel -j0 'axel -n 10 -o $(youtube-dl --get-filename -o "%(title)s.%(ext)s" "{}" --restrict-filenames) $(youtube-dl --get-url {})' :::: /srv/backup/temp/youtube.txt

Learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

answered Feb 16, 2013 at 1:42

Ole Tange

34.1k9 gold badges93 silver badges111 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Édouard Lopez · Accepted Answer · 2013-02-15 18:05:00Z

1

Solution

You need to run your command in a subshell, i.e. put your command into ( cmd ) &.

Definition

A shell script can itself launch subprocesses. These subshells let the script do parallel processing, in effect executing multiple subtasks simultaneously.

Code

For you it will look like this I guess (I add quote to $vURL) :

( axel -n 10 -o "$NAME" "$vURL" ) &

answered Feb 15, 2013 at 18:05

Édouard Lopez

43.7k30 gold badges134 silver badges190 bronze badges

6 Comments

ihm Over a year ago

what about NAME and vURL initialization? how can I run them in parallel?

Lluís Over a year ago

I think the trick here is not subshell, but & which makes your command run in background

ihm Over a year ago

@tictacbum I mean, the script works fine. but I think I can make NAME and vURL run in parallel as well. I've tried a couple of ways. they didn't work out.

William Over a year ago

Yes, you can, but unless those commands take more than seconds to execute, why make things more complex for so little gain?

Édouard Lopez Over a year ago

@William what is the correct syntax with $() command substitution ?

|

Lluís · Accepted Answer · 2013-02-15 18:39:41Z

0

I don't know if it is the best way, you can define a function and then call it in background

something like this:

#!/bin/bash
#package: youtube-dl axel

#file that contains youtube links
FILE="/srv/backup/temp/youtube.txt"

# define a function
download_video() {
  sleep 3
  echo $1
}

while read -r line; do
  # call it in background, with &
  download_video $line &
done < $FILE

script ends quick but function still runs in background, after 3 seconds it will show echos

also used read and while loop to simplify the file reading

edited Feb 15, 2013 at 18:39

answered Feb 15, 2013 at 18:29

Lluís

1,3372 gold badges14 silver badges38 bronze badges

2 Comments

Ian Stapleton Cordasco Over a year ago

You don't need cat here. I know it isn't essential to the answer, but you should do while read -r line ; do download_video $ line & ; done < $FILE. Or lines=$(cat $FILE) if you insist on using cat

ihm Over a year ago

@tictacbum nice. this structure is better. but it still doesn't have NAME and vURL in parallel. This youtube-dl command takes the longest time. that's why I wanna have both of them in parallel

William · Accepted Answer · 2013-02-15 22:42:15Z

0

Here's my take on it. By avoiding several commands you should see some minor improvement in speed though it might not be noticeable. I did add error checking which can save you time on broken URLs.

#file that contains youtube links
FILE="/srv/backup/temp/youtube.txt"

while read URL ; do
    [ -z "$URL" ] && continue

    #get video name
    if NAME=$(youtube-dl --get-filename -o "%(title)s.%(ext)s" "$URL" --restrict-filenames) ; then
        #real video url
        if vURL=$(youtube-dl --get-url $URL) ; then
            #download file
            axel -n 10 -o "$NAME" $vURL &
        else
            echo "Could not get vURL from $URL"
        fi
    else
        echo "Could not get NAME from $URL"
    fi
done << "$FILE"

By request, here's my proposal for paralleling the vURL and NAME fetching as well as the download. Note: Since the download depends on both vURL and NAME there is no point in creating three processes, two gives you about the best return. Below I've put the NAME fetch in its own process, but if it turned out that vURL was consistently faster, there might be a small payoff in swapping it with the NAME fetch. (That way the while loop in the download process won't waste even a second sleeping.) Note 2: This is fairly crude, and untested, it's just off the cuff and probably needs work. And there's probably a much cooler way in any case. Be afraid...

#!/bin/bash

#file that contains youtube links
FILE="/srv/backup/temp/youtube.txt"

GetName () {  # URL, filename
    if NAME=$(youtube-dl --get-filename -o "%(title)s.%(ext)s" "$1" --restrict-filenames) ; then
        # Create a sourceable file with NAME value
        echo "NAME='$NAME'" > "$2"
    else
        echo "Could not get NAME from $1"
    fi
}

Download () {  # URL, filename
    if vURL=$(youtube-dl --get-url $1) ; then
        # Wait to see if GetName's file appears
        timeout=300  # Wait up to 5 minutes, adjust this if needed
        while (( timeout-- )) ; do
            if [ -f "$2" ] ; then
                source "$2"
                rm "$2"
                #download file
                if axel -n 10 -o "$NAME" "$vURL" ; then
                    echo "Download of $NAME from $1 finished"
                    return 0
                else
                    echo "Download of $NAME from $1 failed"
                fi
            fi
            sleep 1
        done
        echo "Download timed out waiting for file $2" 
    else
        echo "Could not get vURL from $1"
    fi
    return 1
}


filebase="tempfile${$}_"
filecount=0

while read URL ; do
    [ -z "$URL" ] && continue
    filename="$filebase$filecount"
    [ -f "$filename" ] && rm "$filename" # Just in case
    (( filecount++ ))
    ( GetName "$URL" "$filename" ) &
    ( Download "$URL" "$filename" ) &
done << "$FILE"

edited Feb 15, 2013 at 22:42

answered Feb 15, 2013 at 18:54

William

4,9652 gold badges17 silver badges14 bronze badges

3 Comments

ihm Over a year ago

NAME and vURL still run in sequence. is it possible to run them in parallel?

William Over a year ago

You could wrap the part from #get video name to the last fi in a function and run that in the background: ( myfunc "$URL" ) & (Replace instances of $URL with $1 in the function, of course.) Personally, I'd rather get those values and know they are valid before spawning the download, but if they are time consuming it might be worthwhile.

William Over a year ago

OK, I see what you're asking now. You want those two individually paralleled. The short answer is, as far as I know, yes, but it isn't pretty. They can't directly set NAME and vRUL in that case so you'd have to jump through some hoops to retrieve those values.

Collectives™ on Stack Overflow

bash commands in parallel

4 Answers 4

Comments

Solution

Definition

Code

6 Comments

2 Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Solution

Definition

Code

6 Comments

2 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related