Using bash command to copy files from a subfolder to another

Question

I have the following structure:

.
├── dag_1
│   ├── dag
│   │   ├── current
│   │   └── deprecated
│   └── sparkjobs
│       ├── current
│       |    └── spark_3.py
│       └── deprecated
│           └── spark_1.py
│           └── spark_2.py
├── dag_2
│   ├── dag
│   │   ├── current
│   │   └── deprecated
│   └── sparkjobs
│       ├── current
│       |    └── spark_3.py
│       └── deprecated
│           └── spark_1.py
│           └── spark_2.py

I want to create a new folder getting only current spark jobs, my expected output folder is:

.
├── dag_1
|    └── spark_3.py
├── dag_2
     └── spark_3.py

I've tried to use

find /mnt/c/Users/User/Test/ -type f -wholename "sparkjob/current" | xargs -i cp {} /mnt/c/Users/User/Test/output/

Although my script is not writing the files and returns me no error. How can I solve this?

Gilles Quénot · Accepted Answer · 2023-01-06 19:30:22Z

2

Use this, install command take the input file and copy it to another dir structure, creating the whole tree of dirs if necessary as mkdir -p transparently:

(you need to add wildcard * in -wholename to effectively find files)

find . -type f -wholename "*/sparkjob/current/*" -exec bash -c '
    dir=${1#./} dir=${dir%%/*} file=${1##*/}
    install -D "$1" "./$dir/$file"
' bash {} \;

Exemple of what is done:

install -D ./dag_2/sparkjob/current/spark_3.py ./dag_2/spark_3.py
install -D ./dag_1/sparkjob/current/spark_3.py ./dag_1/spark_3.py

The source path is an example, if longer, no issue.

edited Jan 6, 2023 at 19:30

answered Jan 6, 2023 at 15:16

Gilles Quénot

188k43 gold badges232 silver badges229 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

tripleee · Accepted Answer · 2023-01-06 16:15:13Z

2

First you should check what find returns by removing everything after |. You'll see find doesn't find any files. The reasons:

as the name implies, -wholename matches the whole name, so you need */sparkjob/current/*
according to your tree output, the folder is not named sparkjob but sparkjobs.

I'd start with something like this:

find /mnt/c/Users/User/Test/ -type f -wholename "*/sparkjobs/current/*" -print0 | while IFS= read -r -d '' file; do
    echo mv "$file" "$(realpath "$(dirname "$file")"/../..)"
done

I added an echo so you can check all paths and commands are correct.

You may want to trade simplicity for performance. See https://mywiki.wooledge.org/BashFAQ/001 if performance is important (many files or frequent runs).

edited Jan 6, 2023 at 16:15

tripleee

192k37 gold badges318 silver badges367 bronze badges

answered Jan 6, 2023 at 15:18

steffen

17.4k4 gold badges47 silver badges87 bronze badges

2 Comments

Gilles Quénot Over a year ago

This will break on files with spaces

steffen Over a year ago

@GillesQuenot Yes, changed.

glenn jackman · Accepted Answer · 2023-01-06 15:17:55Z

1

You'll want to do:

mkdir ../new_folder
find . -type f \
       -path '*/sparkjobs/current/*' \
       -exec sh -c 'f=$1
                    new=${f/sparkjobs\/current\//}
                    dest="../new_folder/$(dirname "$new")"
                    mkdir -p "$dest"
                    cp -v "$f" "$dest"' sh '{}' \;

‘./dag_1/sparkjobs/current/spark_3.py’ -> ‘../new_folder/./dag_1/spark_3.py’
‘./dag_2/sparkjobs/current/spark_3.py’ -> ‘../new_folder/./dag_2/spark_3.py’

answered Jan 6, 2023 at 15:17

glenn jackman

249k42 gold badges233 silver badges362 bronze badges

Comments

Paul Hodges · Accepted Answer · 2023-01-06 16:50:58Z

1

This looks pretty straightforward.

for d in $old_loc/dag_*
do mkdir -p "$new_loc/${d##*/}"
   cp "$d"/sparkjobs/current/spark_*.py "${d##*/}"
done

answered Jan 6, 2023 at 16:50

Paul Hodges

16k1 gold badge24 silver badges43 bronze badges

Collectives™ on Stack Overflow

Using bash command to copy files from a subfolder to another

4 Answers 4

Exemple of what is done:

Comments

2 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Exemple of what is done:

Comments

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related