I need to split a big sorted file into smaller chunks, each file contains a list of sorted person names. Now I want to guarantee that person with same name will not appear in two files, for example,
File1:
.
.
James
James
Kobe
File2:
Kobe
Nash
Nash
.
.
I need to make it to
File1:
.
.
James
James
Kobe
Kobe
File2:
Nash
Nash
.
.
Previously I do this manually using sed. Now I want to write a bash script to automate this, but not familiar with the bash.. Any help how to do it?
You need to compare the last line of the "current" file with the first line of the "next" file. I assume your files are named "File1, File2, ... FileN". This is untested.
n=1
while true; do
current=File$n
next=File$((++n))
if [[ ! -f $next ]]; then
break
fi
last=$(tail -1 "$current")
first=$(head -1 "$next")
while [[ $last == $first ]]; do
echo "$last" >> "$current" # append the name to the end of the current
sed -i 1d "$next" # remove the first line of the next file
first=$(head -1 "$next")
done
done
This may be kind of slow because you may be repeatedly remove a single line from the next file. This might be a bit faster: again, untested.
n=1
while true; do
current=File$n
next=File$((++n))
if [[ ! -f $next ]]; then
break
fi
last=$(tail -1 "$current")
first=$(head -1 "$next")
num=$(awk -v line="$last" -v N=0 '$0 == line {N++; next} {print N; exit}' "$next")
if (( num > 0 )); then
for (( i=1; i<=num; i++ )); do
echo "$last" >> "$current"
done
sed -i "1,$Nd" "$next"
fi
done
