I need to modify a very large geoTif file (over 200GB), however when I save it my code crashes as it loads the file in memory. The code looks like this.
import xarray as xr
# open the file
ds = xr.open_rasterio('file.tif')
# modify here
.....
# this part uses too much memory and gives me the problem
ds.rio.to_raster('output.tif')
Is there any way to save the file as a tif, without loading it all in memory?
I had a similar problem. My geotif is not as large as yours (over 20GB), my ram capacity is 16 GB, and when I run the code. I frequently get error message Process finished with exit code 137 (interrupted by signal 9:SIGKILL).
The solution for me:
Instead of xarray, use rioxarry to read and write tif, the code is provided here multi-worker parts. This works very well for me, i don't recieve any error message and speed up my running.
for example:
import rioxarray as rio
ds = rio.open_rasterio('file.tif', chunks=True,lock=False)
# your modification
.....
# multithreaded
ds.rio.to_raster('output.tif' , tiled=True, lock=threading.Lock())
The chunks = True makes the function automatically split your data into the sensible chunks, and lock = False allow multiple thread, thus speed up your processing. And when you modify your tif, it's better for you to use dask.array, I followed the workflow in this question.
# modify here? we need to see all your code to diagnose the issue, ideally as a minimal reproducible example.