0

i trying to make a script to organize a pair of list i have, and process with other programs, but im a little bit stuck now.

I want from a List in Txt process every line first creating a folder to each line in the list and then process due to different scripts i have.

But my problem is is the list i give to the script is like 3-4 elements works great and create there own directory, but if i put a list with +1000 lines, then my script process only a few elements thru the scripts.

EDIT: the process are like 30-35 scripts, different language python,bash,python and golang

Any suggestions?

cat $STORES+NEW.txt | while read NEWSTORES 
do
    cd $STORES && mkdir $NEWSTORES && cd $NEWSTORES && mkdir .Files
    python3 checkstatus.py -n $NEWSTORES
    checkemployes $NEWSTORES -status
    storemanagers -s $NEWSTORES -o $NEWSTORES+managers.txt
    curl -s https://redacted.com/store?=$NEWSTORES | grep -vE "<|^[\*]*[\.]*$NEWSTORES" | sort -u | awk 'NF' > $NEWSTORES+site.txt
     ..
     ..
     ..
     ..
     ..
     ..
    cd ../..
done
Improve this question
9
  • 1
    I can't see any reason why it would only process the first few lines of the file. Commented Dec 29, 2021 at 23:15
  • 1
    Unless checkstatus.py or checkemployes tries to read from standard input. Then they'll read the rest of the input file. Commented Dec 29, 2021 at 23:17
  • Aside: You can create both directories at once with mkdir -p "$NEWSTORES/.Files" Commented Dec 29, 2021 at 23:18
  • Do you have set -e in your bash script? Commented Dec 30, 2021 at 2:30
  • @joop : Complementing the comment by Barmar: You have left out many commands from your loop body, and replaced them by just ... Did you verify that they don't eat your standard input? Commented Dec 30, 2021 at 9:40

1 Answer 1

Reset to default
1

I'm not supposed to give an answer yet but I mistakenly answered my what should be a comment reply. Anyway here a few things I can suggest:

  • Avoid unnecessary use of cat.

  • Open your input file using another FD to prevent commands that read input inside the loop from eating the input: IFS= read -ru 3 NEWSTORES; do ...; done 3< "$STORES+NEW.txt" or { IFS= read -ru "$FD" NEWSTORES; do ...; done; } {FD}< "$STORES+NEW.txt". Also see https://stackoverflow.com/a/28837793/445221.

  • Not completely related but don't use while loop in a pipeline since it will execute in a subshell. In the future if you try to alter a variable and expect it to be saved outside the loop, it won't. You can use lastpipe to avoid it but it's unnecessary most of the time.

  • Place your variable expansions around double quotes to prevent unwanted word splitting and filename expansion.

  • Use -r option unless you want backslashes to escape characters.

  • Specify IFS= before read to prevent stripping of leading and trailing spaces.

  • Using readarray or mapfile makes it more convenient: readarray -t ALL_STORES_DATA < "$STORES+NEW.txt"; for NEWSTORES IN "${ALL_STORES_DATA[@]}"; do ...; done

  • Use lowercase characters on your variables when you don't use them in a global manner to avoid conflict with bash's variables.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.