Error in micro_era5 when scenario = 2

[Zihan Zhou]

Hi Mike,

I'm currently working with the micro_era5 function and have encountered two issues I would like to ask for your advice on:

1. When running the code, I receive the following warning message:

In microclima::microclimaforNMR(lat = longlat[2], long = longlat[1], : Data sequence in UTC/GMT. Some or all dates using system timezone are in CST

Despite the warning, the code successfully generates the micro output. However, I wonder if I should explicitly modify the microclimaforNMR function or the date inputs to ensure that all data are consistently in UTC timezone. Would that help avoid potential issues?

2. When I set scenario = 0, the code runs normally. But when I set scenario = 2, I encounter the following error:

ERROR: RAINFALL has the wrong number of inputs Error in `colnames<-`(`*tmp*`, value = metout.names) : cannot set 'colnames' for an object with less than two dimensions

Along with additional warnings such as:

ERROR: RAINFALL has the wrong number of inputs

Error in microclimate(micro): object 'a' not found

In RAINFALL * RAIN_diff:
   longer object length is not a multiple of shorter object length

In RAINhr[midnights + 1] <- RAINFALL/3 : replacement length not a multiple of target length

Could this be due to a mismatch between the data lengths, perhaps because the Terra scenario 2 data covers a full year, while the ERA5 data I use as input is only monthly? If so, do you have any suggestions?

Thank you very much for your time and guidance.

Best regards,
Zihan Zhou

[Zihan Zhou]

There are my codes used for calculating microclimate:

cal.micro <- function(id, year,month, scenario) {

  lon <- desert$Longitude[id]
  lat <- desert$Latitude[id]
  ERR <- 1.5  
  dstart <- paste0("01/", sprintf("%02d", month), "/", year)
  dfinish <- paste0(days_in_month(as.Date(paste0(year, "-", sprintf("%02d", month), "-01"))), "/", sprintf("%02d", month), "/", year)
   
    micro <- try(micro_era5(
      loc <- c(as.numeric(lon), as.numeric(lat)),
      scenario = scenario,
      terra_source = "/public/shareddata/TerraClim/data",
      spatial = paste0("/public/shareddata/era5_singlelevel/beta_CDS/", year, "/", sprintf("%02d", month)),
      dstart = dstart,
      dfinish = dfinish,
      REFL = 0.35,
      Usrhyt = 0.01,
      RUF = 0.0003,
      CMH2O = 0.5,
      ERR = ERR
  ))
   
  if (class(micro) == "try-error") {
      cat("Model failed for month", month, "with ERR =", ERR, "\n")
      return(NA)
  } else {
      while (min(micro$metout[, 1]) == 0 & ERR <= 3) {
        cat("Model crashed for month", month, ", trying a higher error tolerance (ERR =", ERR + 0.25, ") \n")
        ERR <- ERR + 0.25
        micro <- try(zhou_micro_era5(
          loc = c(lon, lat),
          scenario = scenario,
          terra_source = "/public/shareddata/TerraClim/data",
          spatial = paste0("/public/shareddata/era5_singlelevel/beta_CDS/", year, "/", sprintf("%02d", month)),
          dstart = dstart,
          dfinish = dfinish,
          REFL = 0.35,
          Usrhyt = 0.01,
          RUF = 0.0003,
          CMH2O = 0.5,
          ERR = ERR
        ))
      }
     
      if (min(micro$metout[, 1]) == 0 & ERR > 3) {
        cat("Model failed for month", month, "with ERR > 3 \n")
        ERR <- ">3"
      }
     
      micro <- micro[c("dates", "nyears", "minshade", "maxshade", "REFL", "DEP", "KS", "BB", "PE", "metout", "shadmet", "soil", "shadsoil", "soilmoist", "shadmoist", "humid", "shadhumid", "soilpot", "shadpot", "tcond", "shadtcond", "RAINFALL", "elev", "longlat")]
      micro$ERR <- ERR
     
      saveRDS(micro, paste0(result_path, scenario, "/", year, "/", "micro_", scenario, "_", year, "_", sprintf("%02d", month), "_", id, ".rds"))
    }
 
}

[NicheMapR]

Hi Zihan,

That message about time zones is just a warning to alert you about what time zone the results are in so no need to worry about it. And yes, if you don't have the ERA5 data in yearly chunks that may be the cause of the problem. I know that the ERA5 data gets downloaded in months now via mcera5. Kris Wild in my group has been combining them into years - I'll ask him to post the code he's using to do that.

All the best,

Mike

[David Klinges]

Hi all,

Dave Klinges here, maintainer of mcera5 -- this functionality is already built into mcera5, which will ultimately provide you annual ERA5 netCDF files if you follow default arguments. The reason for the default behavior of downloading monthly chunks was to circumvent the ECMWF CDS API data volume limits, which became more restrictive when they moved to the new server location. But you can specify mcera5::build_era5_request(..., by_month = FALSE) to build queries by year. In any case, even if you used build_era5_request(by_month = TRUE), then the default behavior of the next function is request_era5(..., combine = TRUE), which combines monthly queries into annual files for easy use in eg with NicheMapR. So you'll end up with an annual file.

I made the by_month behavior default because I was getting a number of mcera5 users who were querying large spatial extents of ERA5 and having their queries canceled due to volume. I think this remains a more common use case than small spatial extents.

Best,

Dave

ᐧ

ᐧ

--
You received this message because you are subscribed to the Google Groups "NicheMapR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nichemapr+...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/nichemapr/1d49863a-2d36-4e84-88eb-a95ef6d9a714n%40googlegroups.com.

--

David Klinges, PhD

https://natureinparadise.github.io/

Google Scholar Page

[Zihan Zhou]

Hi Mike and Dave,

Thank you both for your help! I’ll try it out and let you know if I run into any issues.

All the best,
Zihan

[kristoffer wild]

Zihan (and others),

If you have already downloaded ERA5 data with the month option 'on'. Here is a function that will combine your data by year after it is requested. To give you an idea of how my ERA5 files are set up, I have also attached a screenshot of my file structure. At the end of the function, you will find an explanation and example of how to use it. I hope this helps.

Kris 

#############
### Combine ERA5 month to year function
# 1. Define the function
combine_era5_years <- function(root, years, out_dir) {
  # ensure output folder exists
  dir.create(out_dir, showWarnings = FALSE, recursive = TRUE)
 
  for (yr in years) {
    message("Combining year", yr)
   
    # 2. List the 12 monthly .nc files for this year
    monthly_nc <- list.files(path = root,
                             pattern = paste0("^ERA5_", yr, "_[0-9]{1,2}\\.nc$"),
                             full.names = TRUE)
    if (length(monthly_nc) == 0) {
      warning("no files found for", yr)
      next
    }
   
    # 3. Order by month number (so 10,11,12 don’t come first)

    # had issues with this
    months <- as.integer(sub(".*_([0-9]{1,2})\\.nc$", "\\1", basename(monthly_nc)))
    monthly_nc <- monthly_nc[order(months)]
   
    # 4. Combine into one annual file
    out_file <- file.path(out_dir, paste0("ERA5_", yr, ".nc"))
    combine_netcdf(filenames     = monthly_nc,
                   combined_name = out_file)
    message("✅  wrote: ", out_file)
  }
 
  invisible(NULL)
}


#################
# FUNCTION explanation & example use
# root = where your monthly ERA5_YYYY_M.nc files live  
# years = vector of years (or year-ranges) to loop over  
# out_dir = where you want your ERA5_YYYY.nc files to go  

# run example
combine_era5_years(root = "/Users/kristofferwild/Desktop/ERA5_folder",
                                      years = 2018:2022,
                                      out_dir = "/Users/kristofferwild/Desktop/ERA5_folder/combined")

[Zihan Zhou]

Hi Kris,

Thanks a lot for sending over the function and the screenshot of your file structure—it’s super helpful! The explanation at the end made it really easy to follow, and this method is exactly what I needed. Appreciate you taking the time to share it.

All the best,
Zihan

[Urban Dajcman]

Hi everyone,

i am encountering a similar error. I donwnloaded the monthly era5 data, merged them using Kris's function and the regular micro_era5 runs are sensible (validated on some weather station data) although i do keep getting some rainfall warnings. They also give good predictions with ectotherm (deb=1).
Now, when i try to run the scenario = 2 or 4 things get out of hand, both TREF and TALOC are way too high, im talking about 70 degrees in the summer (values are almost double of what one would expect). Please find attached the regular scenario=0 test runs with observed temp data i used to validate the micro_era5.

If anyone has any advice im very much open to suggestions. I think i still might have some issues with compatibility between the emrged era5 data and the terracliamte scenarios data.

Here is my code to generate the micro objects in a loop, due to issues with snowfall and model crashing at low ERR values im permiting quite a high ERR but most locations work with the default value.

Cheers

Urban


minshade <- 0
maxshade <- 90
Thcond <- 2.5
SpecHeat <- 870
Density <- 2.56
BulkDensity <- 2.55
windfac <- 1
REFL <- 0.2
cap <- 0
SLE <- 0.95
warm <- 0
Usrhyt <- 0.01
clearsky <- 0
ERR <- 1.5
DEP <- c(0, 3, 5, 10, 15, 20, 30, 50, 100, 200)
scenario <- 2

dstart <- "01/01/1990"
dfinish <- "31/12/2010"

getwd()

# Loop over each location
for (i in 1:nrow(locations_df)) {
  ERR <- 1.5
  # Extract location details
  name <- locations_df$Location[i]
  lat <- locations_df$Latitude[i]
  lon <- locations_df$Longitude[i]
 
  # Define the save path for the location
  save_path <- file.path(base_path, name)
 
  # Create the directory if it doesn't exist
  if (!dir.exists(save_path)) {
    dir.create(save_path, recursive = TRUE)
  }
 
  # Change working directory to the new folder
  setwd(save_path)
 
  # Get DEM data
  dem <- microclima::get_dem(r = NA, lat = lat, lon = lon, resolution = 30, zmin = -20, xdims = 100, ydims = 100)
 
  # Convert coordinates to a SpatVector and transform to match DEM CRS
  xy <- terra::vect(data.frame(x = lon, y = lat), geom = c("x", "y"), crs = "EPSG:4326")
  xy <- terra::project(xy, terra::crs(dem))
 
  # Extract elevation using the same method as slope
  elev <- terra::extract(dem, xy, method = "simple")[,2]
 
  # Compute slope and extract value at the location
  slope <- terra::extract(terra::terrain(dem, v = "slope", unit = "degrees"), xy)[,2]
 
  # Compute aspect and extract value at the location
  aspect <- terra::extract(terra::terrain(dem, v = "aspect", unit = "degrees"), xy)[,2]
 
  # Compute horizon angles
  ha36 <- numeric(36)
  for (j in 0:35) {
    har <- microclima::horizonangle(dem, j * 10, terra::res(dem)[1])
    ha36[j + 1] <- atan(terra::extract(har, xy)[,2]) * (180/pi)
  }
 
  # Interpolate horizon angles and constrain to [0,90]
  hori <- spline(x = ha36, n = 24, method = 'periodic')$y
  hori[hori < 0] <- 0
  hori[hori > 90] <- 90
 
  # Run micro_era5
  micro <- micro_era5(
    SLE = SLE,
    warm = warm,
    soilgrids = 0,

    dstart = dstart,
    dfinish = dfinish,

    Usrhyt = Usrhyt,
    DEP = DEP,
    REFL = REFL,
    minshade = minshade,
    maxshade = maxshade,
    loc = c(lon, lat),
    runshade = 1,
    run.gads = 1,
    snowmodel = 1,
    BulkDensity = BulkDensity,
    cap = cap,
    runmoist = 1,
    scenario = scenario,
    Density = Density,
    Thcond = Thcond,
    SpecHeat = SpecHeat,
    slope = slope,
    aspect = aspect,
    hori = hori,
    dem = dem,
    windfac = windfac,
    ERR = ERR,
    save = 1,
    spatial = '../../../Micro_era5/Urban_downloads_era5/ERA5_yearly/ERA5',
    cad.effects = FALSE
  )
  while(min(micro$metout[,1]) == 0 & ERR <= 3.5){
    cat("model crashed, trying a higher error tolerance \n")
    ERR <- ERR + 0.5
    micro <- micro_era5(
      SLE = SLE,
      warm = warm,
      soilgrids = 0,

      dstart = dstart,
      dfinish = dfinish,

      Usrhyt = Usrhyt,
      DEP = DEP,
      REFL = REFL,
      minshade = minshade,
      maxshade = maxshade,
      loc = c(lon, lat),
      runshade = 1,
      run.gads = 1,
      snowmodel = 1,
      BulkDensity = BulkDensity,
      cap = cap,
      runmoist = 1,
      scenario = scenario,
      Density = Density,
      Thcond = Thcond,
      SpecHeat = SpecHeat,
      slope = slope,
      aspect = aspect,
      hori = hori,
      dem = dem,
      windfac = windfac,
      ERR = ERR,
      save = 2,
      spatial = '../../../Micro_era5/Urban_downloads_era5/ERA5_yearly/ERA5',
      cad.effects = FALSE
    )
    gc()
  }
  setwd(base_path)  
  if(!(min(micro$metout[,1]) == 0 & ERR <= 3)){
   
    # Return to main working directory to save results
   
   
    # Save the output file in the newly created directory
    save(micro, file = file.path(base_path, paste0("micro_era5_2_", name, ".RData")))
   
    # Print progress
    message("Finished processing ", name, ". Results saved in: ", base_path)
  }else{
    message("Failed processing ", name)
  }
}

torek, 6. maj 2025 ob 10:01:29 UTC+2 je oseba zhou...@gmail.com napisala:

[Urban Dajcman]

Hi everyone,

i have also noticed some odd behavior when running the scenario=2 micro_era5 models. When i run them with scenarios=0 they perform well and i have validated them to some weather station data.

When running the scenario=2 or4 the predicitons for temperature are way too high (70 degrees in the summer) for Slovenia.

I have downloaded the era5 data using the most recent version of mcera5 by month and merged them using the function posted by Kris.

Im very much open to suggestions regarding what could be the possible reasons. I do have to allow fairly high ERR in some cases but the problem persists even when ERR is at the default.

Cheers

Urban

Here is my code:

minshade <- 0
maxshade <- 90
Thcond <- 2.5
SpecHeat <- 870
Density <- 2.56
BulkDensity <- 2.55
windfac <- 1
REFL <- 0.2
cap <- 0
SLE <- 0.95
warm <- 0
Usrhyt <- 0.01
clearsky <- 0
ERR <- 1.5
DEP <- c(0, 3, 5, 10, 15, 20, 30, 50, 100, 200)

scenario <- 4

    dstart = dstart,
    dfinish = dfinish,

      dstart = dstart,
      dfinish = dfinish,

    save(micro, file = file.path(base_path, paste0("micro_era5_4_", name, ".RData")))

   
    # Print progress
    message("Finished processing ", name, ". Results saved in: ", base_path)
  }else{
    message("Failed processing ", name)
  }
}

torek, 6. maj 2025 ob 10:01:29 UTC+2 je oseba zhou...@gmail.com napisala:

Hi Kris,

[Urban Dajcman]

Hi everyone

I have recently been trying to run some micro_era5 scripts with scenario =2/4 aswell and have noticed some odd behavior, probbably due to my inexperience with the climate change models. My code works and preforms well for the regular scenario=0 but when running with climate change scenarios im getting temperature predicitons that are way too high, like in the 70 degrees TAREF.

I donloawded the era5 data in monthly chunks and merged them using Kris's function and that seems to work well for the no climate change runs, getting some rainfall warnings though.

Please find attached the image of the validation i did for three locations using ther regular scenario=0 as well as my code for the scenario=2 micro_era5 models.

Im very much open to suggestions on how to resolve this problem, could it be something related to a mismatch between the era5 data and the terraclimate data used for climate change predictions? Or i might have messed something up just in terms of the era5 data.

Cheers

Urban

My code, the high ERR is due to models often crashing in my study areas, most locations work with the default though, I am aware this could also be a reason:

    dstart = dstart,
    dfinish = dfinish,

      dstart = dstart,
      dfinish = dfinish,

Hi Kris,

[NicheMapR]

Hi Urban,

Your messages ended up in the 'pending' folder for some reason, so I released them all. Re the issue, I just tried the simulation below and it looked fine. Can you send me an example location where you're getting the exceedingly high differences?

All the best,

Mike

minshade <- 0
maxshade <- 90
Thcond <- 2.5
SpecHeat <- 870
Density <- 2.56
BulkDensity <- 2.55
windfac <- 1
REFL <- 0.2
cap <- 0
SLE <- 0.95
warm <- 0
Usrhyt <- 0.01
clearsky <- 0
ERR <- 1.5
DEP <- c(0, 3, 5, 10, 15, 20, 30, 50, 100, 200)

scenario <- 2

dstart <- "01/01/1990"
dfinish <- "31/12/1990"

  ERR <- 1.5
  # Extract location details

  lat <- -31
  lon <- 145

    scenario = 0,

    Density = Density,
    Thcond = Thcond,
    SpecHeat = SpecHeat,
    slope = slope,
    aspect = aspect,
    hori = hori,
    dem = dem,
    windfac = windfac,
    ERR = ERR,

    save = 0,
    spatial = 'z:/ERA5_Australia_time/ERA5',
    cad.effects = FALSE
  )
 
metout <- as.data.frame(micro$metout)

plot(micro$dates, metout$TAREF, type = 'l')

micro2 <- micro_era5(

  SLE = SLE,
  warm = warm,
  soilgrids = 0,
  dstart = dstart,
  dfinish = dfinish,
  Usrhyt = Usrhyt,
  DEP = DEP,
  REFL = REFL,
  minshade = minshade,
  maxshade = maxshade,
  loc = c(lon, lat),
  runshade = 1,
  run.gads = 1,
  snowmodel = 1,
  BulkDensity = BulkDensity,
  cap = cap,
  runmoist = 1,

  scenario = 2,

  Density = Density,
  Thcond = Thcond,
  SpecHeat = SpecHeat,
  slope = slope,
  aspect = aspect,
  hori = hori,
  dem = dem,
  windfac = windfac,
  ERR = ERR,

  save = 0,
  spatial = 'z:/ERA5_Australia_time/ERA5',
  cad.effects = FALSE
)

metout2 <- as.data.frame(micro2$metout)

plot(micro$dates, metout$TAREF, type = 'l')
points(micro2$dates, metout2$TAREF, type = 'l', col = 'red')

plot(micro$dates, metout$TALOC, type = 'l')
points(micro2$dates, metout2$TALOC, type = 'l', col = 'red')

max(metout2$TAREF - metout$TAREF)
max(metout2$TALOC - metout$TALOC)

[Urban]

Hi Mike,

I was debbuging my issue for the last two days and realized that the loop i made for micro_era5 9 only buggs if i run it as a loop over several locations.

The code works if i run a single location then restart R, then run another single location and so on. If i run single locations without restarting r the second one is off. This makes me.think that the terra climate scenarios data gets stored somewhere in r and interferes with calculations down the line,  the loop worked without scenarios.

Im still using the same script, most locations work with err 1.5 or 2. I didnt have errors for specific locations so any of the ones in the code i provided previously should generate a reproducible example, if you run two in a row (loop over two).

I guess the issue is somewhere in the loop there is a faulty cleanup or something on my local machine is off.

Thanks for the advice and checking the messages, im not sure why they are pending in the g group.

Cheers

Urban

V sre., 16. jul. 2025, 01:59 je oseba NicheMapR <nich...@googlegroups.com> napisala:

To view this discussion, visit https://groups.google.com/d/msgid/nichemapr/dbc3f7d5-e677-43b2-81a3-90381aa875fen%40googlegroups.com.