Extracting variables from gridded netcdf data into ASCII

Question

I would like to know if anyone has any experience extracting information from gridded climate data and organizing it in a way that would be easier to work with to produce a time series. I am not sure how to use the indices (i and j or lat and lon, respectively) and I would like to only select data from above 60 degrees latitude. When I tried to extract the data to line it up I keep getting the following error:

ValueError: slicing expression exceeds the number of dimensions of the variable"

I am trying to put the data in the following table format:

Date,  lat,    lon,    snc

The header information is:

netcdf snc_day_MPI-ESM-LR_amip_r1i1p1_19790101-20081231 {
dimensions:
    time = UNLIMITED ; // (10958 currently)
    lat = 96 ;
    lon = 192 ;
    bnds = 2 ;
variables:
    double time(time) ;
        time:bounds = "time_bnds" ;
        time:units = "days since 1979-1-1 00:00:00" ;
        time:calendar = "proleptic_gregorian" ;
        time:axis = "T" ;
        time:long_name = "time" ;
        time:standard_name = "time" ;
    double time_bnds(time, bnds) ;
    double lat(lat) ;
        lat:bounds = "lat_bnds" ;
        lat:units = "degrees_north" ;
        lat:axis = "Y" ;
        lat:long_name = "latitude" ;
        lat:standard_name = "latitude" ;
    double lat_bnds(lat, bnds) ;
    double lon(lon) ;
        lon:bounds = "lon_bnds" ;
        lon:units = "degrees_east" ;
        lon:axis = "X" ;
        lon:long_name = "longitude" ;
        lon:standard_name = "longitude" ;
    double lon_bnds(lon, bnds) ;
    float snc(time, lat, lon) ;
        snc:standard_name = "surface_snow_area_fraction" ;
        snc:long_name = "Snow Area Fraction" ;
        snc:units = "%" ;
        snc:cell_methods = "time: mean" ;
        snc:cell_measures = "area: areacella" ;
        snc:history = "2011-05-26T14:06:06Z altered by CMOR: replaced missing value flag (1e+22) with standard missing value (1e+20)." ;
        snc:missing_value = 1.e+20f ;
        snc:_FillValue = 1.e+20f ;
        snc:associated_files = "baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_landIce_fx_MPI-ESM-LR_amip_r0i0p0.nc areacella: areacella_fx_MPI-ESM-LR_amip_r0i0p0.nc" ;

// global attributes:
        :institution = "Max Planck Institute for Meteorology" ;
        :institute_id = "MPI-M" ;
        :experiment_id = "amip" ;
        :source = "MPI-ESM-LR 2011; URL: http://svn.zmaw.de/svn/cosmos/branches/releases/mpi-esm-cmip5/src/mod; atmosphere: ECHAM6 (REV: 4418), T63L47; land: JSBACH (REV: 4418);" ;
        :model_id = "MPI-ESM-LR" ;
        :forcing = "GHG Oz SD Sl Vl LU" ;
        :parent_experiment_id = "N/A" ;
        :parent_experiment_rip = "N/A" ;
        :branch_time = 0. ;
        :contact = "[email protected]" ;
        :history = "Model raw output postprocessing with modelling environment (IMDI) at DKRZ: URL: http://svn-mad.zmaw.de/svn/mad/Model/IMDI/trunk, REV: 3135 2011-05-26T14:06:06Z CMOR rewrote data to comply with CF standards and CMIP5 requirements." ;
        :references = "ECHAM6: n/a; JSBACH: Raddatz et al., 2007. Will the tropical land biosphere dominate the climate-carbon cycle feedback during the twenty first century? Climate Dynamics, 29, 565-574, doi 10.1007/s00382-007-0247-8;" ;
        :initialization_method = 1 ;
        :physics_version = 1 ;
        :tracking_id = "7c36f2e9-7f1a-4bcc-94f5-bf2ab21dd16d" ;
        :product = "output" ;
        :experiment = "AMIP" ;
        :frequency = "day" ;
        :creation_date = "2011-05-26T14:06:06Z" ;
        :Conventions = "CF-1.4" ;
        :project_id = "CMIP5" ;
        :table_id = "Table day (27 April 2011) 86d1558d99b6ed1e7a886ab3fd717b58" ;
        :title = "MPI-ESM-LR model output prepared for CMIP5 AMIP" ;
        :parent_experiment = "N/A" ;
        :modeling_realm = "landIce land" ;
        :realization = 1 ;
        :cmor_version = "2.5.9" ;

Thank you for your response. I guess this is the problem is that I am trying to figure out how to arrange the data where I can just use the snc information to produce a time series for one particular area shown with lat and lon. There is no station id or indexing information so how would I arrange this information from gridded data? — user3275006
– user3275006, Commented Aug 11, 2014 at 10:52

Bart · Accepted Answer · 2016-11-20 18:00:23Z

2

Agree with N1B4 on use of ncks. Also note that ncks accepts coordinate values (indicated by decimal points) not just indices (indicated by indices). Hence, e.g., latitude values greater than 60 degrees can be obtained by

ncks -d lat,60., in.nc

without knowing the underlying coordinate grid exactly.

edited Nov 20, 2016 at 18:00

Bart

10.4k5 gold badges54 silver badges83 bronze badges

answered Sep 3, 2014 at 2:31

Charlie Zender

6,45217 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user3275006 Over a year ago

Thank you very much! From yours and the above help it has led me to find out all the great (and efficient) ways I can use NCO for my netCDF data. However, due to the large amount of data from these files, as well as missing data, is there a way I can omit the data that has missing values so that the output only selects data where applicable? I don't know if this is something I should open a new question for. Thank you again!

Charlie Zender Over a year ago

There is no elegant way to omit the missing values. However, if space is your concern, then output the file as netCDF4 format with compression and you may greatly reduce the output filesize. E.g., try ncks -4 -L 1 -d lat,60., in.nc out.nc

Charlie Zender Over a year ago

Oops, I see you want ASCII output, so netCDF4 is not helpful. Instead pipe the output of ncks into a UNIXy filter that removes lines containing the missing value. Such a filter can be written in Perl, awk, etc. Since missing values are printed as an underscore by default, it may work simply to echo all lines that do not contain an underscore, e.g., ncks -d lat,60., in.nc | awk my_script

N1B4 · Accepted Answer · 2014-08-13 16:09:33Z

1

ncks from the NCO package is ideal for this type of procedure. You can easily extract a time series of snc for latitudes greater than 60 degrees. You need to first identify the indices in the dimension lat that contain values greater than 60.

ncks -v lat infile.nc | more

This will print the values of lat (the variable, not the dimension). Without having the file, let's just say values greater than 60 are 80-96. Remember that in Python (and NCO), the first index is 0, so the range of 80-96 is 79-95 in zero-index world.

Now we can use ncks to extract values of snc within the lat range of 79-95.

ncks -v snc -d lat,79,95 infile.nc -O outfile.nc

You can add as many dimension slices as you want using ncks. If you want a specific time and lon range, simply add it on! For example,

ncks -v snc -d lat,79,95 -d lon,0,25 -d time,0,5 infile.nc -O outfile.nc

answered Aug 13, 2014 at 16:09

N1B4

3,4631 gold badge23 silver badges25 bronze badges

1 Comment

user3275006 Over a year ago

I see that it is doing a great job selecting the correct ranges. I think this may be exactly what I need but I will need to spend some time with it or look further into the nco discussion forums perhaps. Thank you very much for your input and for introducing the user-friendly nco option.

Collectives™ on Stack Overflow

Extracting variables from gridded netcdf data into ASCII

2 Answers 2

3 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related