"missing_value" and Time, Time1, etc problem

[Kenneth schaudt]

Encountering time, time1 in the downloaded files which causes my data processing system to fail.   As seen below, I get a warning about missing_value versus _fillvalue after this command:   ncks -D9 -d time,2026-02-11T00:00:00Z,2026-02-17T23:00:00Z -v water_temp -d lat,20.,35. -d lon,260.,280. -d depth,0. http://tds.hycom.org/thredds/dodsC/FMRC_ESPC-D-V02_ts3z/FMRC_ESPC-D-V02_ts3z_best.ncd -O ../data/FMRC_ESPC-D-V02_ts3z_best_0m.nc4

Anyone know what value "_FillValue" should be set to?   Would this fix the time, time1 issues that occur occasionally?

 WARNING nco_mss_val_get() reports that variable time has attribute "missing_value" but not "_FillValue". To comply with netCDF conventions, NCO ignores values that equal the _FillValue attribute when performing arithmetic. Confusingly, values equal to the missing_value should also be neglected. However, it is tedious and (possibly) computationally expensive to check each value against multiple missing values during arithmetic on large variables. So NCO thinks that processing variables with a "missing_value" attribute and no "_FillValue" attribute may produce undesired arithmetic results (i.e., where values that were intended to be neglected were not, in fact, neglected). We suggest you rename all "missing_value" attributes to "_FillValue" or include both "missing_value" and "_FillValue" attributes (with the _same values_) for all variables that have either attribute. Because it is long, this message is only printed once per operator even though multiple variables may have the same attribute configuration

[Ken Schaudt]

Don't know why multiple times occur now and again with the velocity fields.  Having two different time variables and different numbers of U and V records plays havoc with the R's metR ReadnetCDF routine that I used to input and initially process the "best" netCDF files.  

My inelegant solution was to stop using ReadnetCDF, match U and V components by date and retain only complete records as shown below..  

library(ncdf4) 

library(data.table)

# 1. Open the NetCDF file 

nc_file <- nc_open("FMRC_ESPC-D-V02_uv3z_best_0m.nc4")

# 2. Extract Coordinate Dimensions
lat    <- as.vector(ncvar_get(nc_file, "lat"))
lon    <- as.vector(ncvar_get(nc_file, "lon"))
depths <- as.vector(ncvar_get(nc_file, "depth"))
time_raw  <- ncvar_get(nc_file, "time")
time1_raw <- ncvar_get(nc_file, "time1")

# 3. Identify index for depth = 0
depth_idx <- which(depths == 0)
if (length(depth_idx) == 0) {
  stop("Depth 0m not found in the dataset.")
}

# 4. Extract Time Units and Origin for conversion
# Format is usually "hours since YYYY-MM-DD HH:MM:SS"
time_unit_attr <- ncatt_get(nc_file, "time", "units")$value
time_origin    <- gsub("hours since ", "", time_unit_attr)

time1_unit_attr <- ncatt_get(nc_file, "time1", "units")$value
time1_origin    <- gsub("hours since ", "", time1_unit_attr)

# 5. Extract Variables (Handling 3D vs 4D structures)
# ncdf4 often drops dimensions of length 1 (like depth in this file)
u_array <- ncvar_get(nc_file, "water_u")
v_array <- ncvar_get(nc_file, "water_v")

nc_close(nc_file)

# 6. Slice to Depth = 0 if necessary
# If the array is 4D [lon, lat, depth, time], we take only the 0m slice
if (length(dim(u_array)) == 4) {
  u_array <- u_array[ , , depth_idx, ]
  v_array <- v_array[ , , depth_idx, ]
}

# 7. Assign Dimension Names for Melting
# After slicing (or if it was already 3D), the shape is [lon, lat, time]
dimnames(u_array) <- list(lon = lon, lat = lat, time = time_raw)
dimnames(v_array) <- list(lon = lon, lat = lat, time1 = time1_raw)

# 8. Convert to Data Tables (na.rm = TRUE removes land masks to save memory)
dt_u <- as.data.table(reshape2::melt(u_array, value.name = "water_u", na.rm = TRUE))
dt_v <- as.data.table(reshape2::melt(v_array, value.name = "water_v", na.rm = TRUE))

# 9. Convert Numeric Time to POSIXct (Date-Time)
# Multiply by 3600 to convert "hours since" to "seconds since" for R
dt_u[, time := as.POSIXct(time * 3600, origin = time_origin, tz = "UTC")]
dt_v[, time1 := as.POSIXct(time1 * 3600, origin = time_origin, tz = "UTC")]

# 10. Merge and Finalize
# Rename time1 to time temporarily to align the tables on shared keys
setnames(dt_v, "time1", "time")
ocean_dt <- merge(dt_u, dt_v, by = c("lon", "lat", "time"), all = TRUE)

# Add constant columns and restore time1
ocean_dt[, depth := 0]
ocean_dt[, time1 := time]

# Reorder columns
setcolorder(ocean_dt, c("depth", "lat", "lon", "time", "time1", "water_u", "water_v"))

# Retain complete records 
SurfaceUV = na.omit(ocean_dt)