Convert ASCII File to netCDF using Python (selective columns from ASCII file)

Question

Firstly I am new to this forum so please excuse as to any mistake I am doing in posting here. I would be glad if you could point me my mistakes out and I will make sure I don't repeat them when I post any thing else.

Task: Converting ASCII Data Files (UCAC 4 Star Catalog) to netCDF Format using Python. Only output some fixed number of columns from ASCII file to netCDF file.

Problem: Traceback (most recent call last): File "D:\Work 1\FINAL\New Try\txt2nc.py", line 51, in vic_runoff[ra,spd,:,:] = output; File "netCDF4.pyx", line 2821, in netCDF4.Variable.setitem (netCDF4.c:35204) File "C:\Python27\lib\site-packages\netCDF4_utils.py", line 187, in _StartCountStride ee = range(start,stop,step) File "C:\Python27\lib\site-packages\numpy\ma\core.py", line 4102, in int raise MaskError('Cannot convert masked element to a Python int.') MaskError: Cannot convert masked element to a Python int.**

Thanks in advance. Any help is appreciated!

    from __future__ import division
    from netCDF4 import Dataset
    import numpy as np
    import os

    PATH = 'D:\\Internship Work 1\\Alok Data\\ASCII'
    LL = np.loadtxt('%s\\4uc001.txt' %PATH, delimiter='|', usecols = 
     (0,1,2,3), skiprows=0);
    LL = LL[:,:]

    # NC file setup
    root_grp = Dataset('%s\\4uc001.nc' %PATH, 'w', format='NETCDF4')
    root_grp.description = 'Star Catalog UCAC Data'

    # dimensions
    ra = root_grp.createDimension('ra', 32)
    spd = root_grp.createDimension('spd', 80)
    magm = root_grp.createDimension('magm', 96)
    maga = root_grp.createDimension('maga', 120)


    # variables
    ra = root_grp.createVariable('ra', np.byte, ('ra',))
    spd = root_grp.createVariable('spd', np.byte, ('spd',))
    magm = root_grp.createVariable('magm', np.byte, ('magm'),)
    maga = root_grp.createVariable('maga', np.byte, ('maga'),)
    vic_runoff = root_grp.createVariable('vic_runoff', np.byte, ('ra',
     'spd', 'magm', 'maga',))

    ra.units = 'mas'
    spd.units = 'mas'
    magm.units = 'millimag'
    maga.units = 'millimag'

    for ra in enumerate(ra):
      tempstore = np.zeros((206,4),int)


    output_filename = 'D:\\Internship Work 1\\Alok Data\\ASCII\\4uc001.txt'

    output = np.loadtxt(output_filename,delimiter='|',usecols = (0,1,2,3))

    tempstore[:,:] = output # ensembles x months
    vic_runoff[ra,spd,:,:] = tempstore[:,:] # write all ensembles to netcdf

    print('work done')

jhamman · Accepted Answer · 2016-02-09 16:23:28Z

2

A few comments that will hopefully help you:

In vic_runoff[ra,spd,:,:], spd is a netCDF4 variable. It can not be used as an indexer. Try vic_runoff[ra,0,:,:] = tempstore[:,:] to see if that fixes your problem.

Also, you should consider using pandas.read_csv or read_table to read your ASCII files. There's a SO post around here documenting the performance improvement over numpy.loadtxt

Lastly, if you went the pandas route, you may consider using xarray for the writing to netCDF. xarray easily allows you to convert from pandas to a xarray.Dataset then you're whole netCDF file is written with a quick Dataset.to_netcdf() call.

answered Feb 9, 2016 at 16:23

jhamman

6,5441 gold badge26 silver badges44 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Alok Dadlani Over a year ago

Thanks for your help! I will surely try it out let you know

Alok Dadlani Over a year ago

Thank You very much! I tried using pandas and xarray and its working now, I will post the solution here soon for others!

Collectives™ on Stack Overflow

Convert ASCII File to netCDF using Python (selective columns from ASCII file)

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related