1

I have a file called my_file.txt that contains the following strings:

tasmax_day_ACCESS_historical_r1i1p1f3_gn.nc
tasmax_day_EC-Earth3_historical_r1i1p1f1_gn.nc
tasmax_day_EC-Earth3_historical_r1i1p1f1_gr.nc
tasmax_day_EC-Earth3_historical_r1i1p1f3_gn.nc
tasmax_day_HadGEM-MM_historical_r1i1p1f1_gn.nc
tasmax_day_HadGEM-MM_historical_r1i1p1f1_gr.nc
tasmax_day_HadGEM-MM_historical_r3i1p1f1_gn.nc
tasmax_day_MIROC_historical_r1i1p1f1_gn.nc
tasmax_day_MIROC_historical_r2i1p1f1_gn.nc

I need to perform a unique of the substring starting with tasmax end ending in _historical where for each such substring I would keep only that line containing it that comes first in alphabetical order.

My expected output of my_file.txt is the following:

tasmax_day_ACCESS_historical_r1i1p1f3_gn.nc
tasmax_day_EC-Earth3_historical_r1i1p1f1_gn.nc
tasmax_day_HadGEM-MM_historical_r1i1p1f1_gn.nc
tasmax_day_MIROC_historical_r1i1p1f1_gn.nc

Thanks for any help.

Improve this question

1 Answer 1

Reset to default
4

A simple Awk would suffice. Form a hash-map, keyed by the unique identifier string and print only those lines

awk -F_ '{ key = $1 FS $2 FS $3 $4 } !unique[key]++ ' file

With the delimiter set to _, access the individual works by $1 notation and form the key upto including $4. The expression !unique[key]++ will be non-zero only if the key (formed) from the line is not already seen.

This assumes your tasmax string occurs at $1 and historical at $4 and does not work otherwise.

Or just use the sort tool, asking it to unique (-u) lines by delimiting on _ using fields 1-4. Works on both BSD and GNU sort variants

sort -u -t_ -k1,4 < file
Improve this answer
3
  • I was just thinking this was a job for awk. (or perl if you into that.) Commented Oct 22, 2021 at 9:30
  • 2
    sort seems to be faster Commented Oct 22, 2021 at 10:00
  • don't know about velocity but definitely sort solution much more intuitive :) Commented Oct 24, 2021 at 11:58

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.