How to save xarray data to csv

[shalu....@gmail.com]

Hello All,

I have used xarray to merge several netcdf files into one file and then I subset the data of my point of interest using lat/long. Now I want to save this array data (time,lat,long) into csv file but I am getting an error with my code:

dsmerged = xarray.open_mfdataset('F:/NTU_PDF__Work/1_Codes/1_Python/testing/netcdf/*.nc')

#save to netcdf file
dsmerged.to_netcdf('combine.nc')

# Extraction of data as per given coordinates
#values (lat/long with their upper and lower bound)

fin = netCDF4.Dataset("combine.nc" ,"r")
# print the all the variables in this file
print (fin.variables)
# print the dimensions of the variable
print (fin.variables["clt"].shape)
#Out: (20075, 90, 144)
# retrieve time step from the variable
clt0 = (fin.variables["clt"])
# extract a subset of the full dataset contained in the file
clt0sub = clt0[10:30,20:30]
# xarray to numpy array
clt1=numpy.array(clt0sub)
# saving data into csv file
with open('combine11.csv', 'wb') as f:
writer = csv.writer(f, delimiter=',')
writer.writerows(enumerate(clt1))

getting this error - TypeError: a bytes-like object is required, not 'str' when I am removing "b" the error disappears but the data saving in wrong format example:-

0,"[[ 99.93312836 99.99977112 100. ..., 98.53624725
99.98111725 99.9799881 ]
[ 99.95301056 99.99489594 99.99998474 ..., 99.9998703 99.99951172
99.97265625]
[ 99.67852783 99.96372986 99.99999237 ..., 99.96694946 99.9842453
99.96450806]
...,
[ 78.29571533 45.00857544 24.39345932 ..., 90.86527252
84.48490143 62.53995895]
[ 42.03381348 46.1696701 22.71044922 ..., 80.88492584
71.15007019 50.95384216]
[ 34.75331879 49.99913025 17.66173935 ..., 57.12231827
62.56645584 40.6435585 ]]"

1,"[[ 100. 100. 100. ..., 99.93876648
99.98928833 100. ]
[ 99.9773941 100. 99.4933548 ..., 97.93031311
97.36623383 97.07974243]
[ 98.5934906 99.72555542 99.44548035 ..., 79.59191132
85.77541351 94.40919495]
...,

suggestions would be appreciated

[Rhodri James]

On 16/04/18 15:55, shalu....@gmail.com wrote:
> Hello All,
>
> I have used xarray to merge several netcdf files into one file and then I subset the data of my point of interest using lat/long. Now I want to save this array data (time,lat,long) into csv file but I am getting an error with my code:

You don't say, but I assume you're using Python 2.x

[snip]

> # xarray to numpy array
> clt1=numpy.array(clt0sub)
> # saving data into csv file
> with open('combine11.csv', 'wb') as f:
> writer = csv.writer(f, delimiter=',')
> writer.writerows(enumerate(clt1))
>
> getting this error - TypeError: a bytes-like object is required, not 'str'

Copy and paste the entire traceback please if you want help. We have
very little chance of working out what produced that error without it.

> when I am removing "b" the error disappears

Which "b"? Don't leave us guessing, we might guess wrong.

> but the data saving in wrong format

Really? It looks to me like you are getting exactly what you asked for.
What format were you expecting? What are you getting that doesn't
belong. I suspect that you don't want the "enumerate", but beyond that
I have no idea what you're after.

--
Rhodri James *-* Kynesim Ltd

[Chris Angelico]

On Tue, Apr 17, 2018 at 3:50 AM, Rhodri James <rho...@kynesim.co.uk> wrote:
> You don't say, but I assume you're using Python 2.x
>
> [snip]
>

>> getting this error - TypeError: a bytes-like object is required, not 'str'

Actually, based on this error, I would suspect Python 3.x. But you're
right that (a) the Python version should be stated for clarity
(there's a LOT of difference between Python 3.3 and Python 3.7), and
(b) the full traceback is very helpful.

ChrisA

[shalu....@gmail.com]

On Tuesday, 17 April 2018 01:56:25 UTC+8, Rhodri James wrote:
> On 16/04/18 15:55, shalu....@gmail.com wrote:
> > Hello All,
> >
> > I have used xarray to merge several netcdf files into one file and then I subset the data of my point of interest using lat/long. Now I want to save this array data (time,lat,long) into csv file but I am getting an error with my code:
>
>
> You don't say, but I assume you're using Python 2.x

Hi James, I am using WinPython Spyder 3.6.

>
> [snip]
>
> > # xarray to numpy array
> > clt1=numpy.array(clt0sub)
> > # saving data into csv file
> > with open('combine11.csv', 'wb') as f:
> > writer = csv.writer(f, delimiter=',')
> > writer.writerows(enumerate(clt1))
> >
> > getting this error - TypeError: a bytes-like object is required, not 'str'
>
> Copy and paste the entire traceback please if you want help. We have
> very little chance of working out what produced that error without it.
>
> > when I am removing "b" the error disappears

here i mean [with open('combine11.csv', 'wb') as f:] wb: writing binaries
if i am using "wb" so i m getting "TypeError: a bytes-like object is required, not 'str'"

if i am removing "b" and using only "w" so this error disappears and when i am writing data into txt/csv so it is just pasting what i am seeing in my console window. I mean i have 20045 time steps but i am getting 100.2...like that as previously mentioned. Not getting full time steps. It is like printscreen of my python console.

My question is how can i save multi-dimentional (3d: time series values, lat, long) data (xarrays) into csv.

Thanks

[shalu....@gmail.com]

On Tuesday, 17 April 2018 02:01:19 UTC+8, Chris Angelico wrote:
> On Tue, Apr 17, 2018 at 3:50 AM, Rhodri James <rho...@kynesim.co.uk> wrote:
> > You don't say, but I assume you're using Python 2.x
> >
> > [snip]
> >
> >> getting this error - TypeError: a bytes-like object is required, not 'str'
>
> Actually, based on this error, I would suspect Python 3.x.

Yes Chris, I am using 3.x only

[Rhodri James]

On 17/04/18 03:25, shalu....@gmail.com wrote:
> My question is how can i save multi-dimentional (3d: time series values, lat, long) data (xarrays) into csv.

What do you want each line of the CSV file to look like? That's the key
question. Once you know that you can arrange your data into a (one
dimensional) list of that form and then write it out.

Since I have exactly no idea what netCDF4.Dataset() does (and less
interest in finding out), I can't offer much advice. The slicing that
you're doing looks like some weird limiting selection, but there must be
some way of iterating through entries, surely?