Error in calibrating hydromad cwi model using fitbyOptim on hourly data

[Thelma Baddoo]

Hi group,

I am a new user of hydromad and I am trying to use hydromad cwi on hourly data. I have followed all the instructions in the hydromad tutorial and some suggestions from this hydromad group. However, when I run the fitbyOptim function, I get this error:

Error in lm.fit(x = z2[, -1, drop = FALSE], y = z2[, 1]) : 

  0 (non-NA) cases

I don't know what I have done or am doing wrong. Any help will be greatly appreciated.

I have attached my raw data (2000.csv) and my hydromad R script (hydromad trial script.R) for further details.

Thank you

[Willem]

Hi Thelma,
I have fixed up a few things in your script, but I am still getting an error in the fitting:

Error in if (k != round(k)) { : missing value where TRUE/FALSE needed
My guess is that it has something to do with the fact that you are trying to fit 5 parameters to essentially 77 data points (with a 20 hour warmup). I think sriv might struggle with that.
I would have to dig in the code on github, where exactly the error message comes from, but maybe Joseph G knows this of the top of his head.
I think it is sriv, because running without sriv (not using line 85 in the script) and then running fitByOptim only gives a warning:
Warning messages:
1: In nseStat(aggrQ, doaggr(X), ref = aggrRef, ...) :
  merged time series have no data; incompatible times?
This might have to do with the fact that your last day in the time series is not a complete day? Again not sure, as I normally don't work with hourly data.

I would suggest you try to things:
1. run with a longer time series and first run without sriv
2. run with a daily time series and see if that works

Good luck
Willem 

[Willem]

Posting Thelma's reply to me on the google groups

Dear Professor Willem,

 

I have received your reply to my hydromad hourly data challenges and I greatly appreciate your help. You suggested I try two things namely:

 

1. run with a longer time series and first run without sriv

2. run with a daily time series and see if that works

 

However, the time series data I have is flood events data and is no longer than the end of the flood of that particular year. The next available data is the flood event data of the subsequent year and thus the data is not continuous. Also, I am not in possession of reliable continuous daily time series for my area of study.

 

I therefore  humbly would like to ask if you have any other suggestions to help with the problems I am facing with the hourly data to help me continue with my research.

 

I hope to hear from you soon.

 

Thank you.

 

--

Best regards,

 

Thelma Baddoo

On Monday, 29 January 2018 21:52:26 UTC+11, Thelma Baddoo wrote:

[Willem]

Dear Thelma,

As I wrote in my previous e-mail, those answers were my best guesses. I am afraid you will have to wait for other users, or try other hourly data (maybe data available in the package) to test whether my suggestions were correct

Please post replies to the users group so it benefits other users

Regards

Willem

[zbr...@gmail.com]

Hi Thelma,

I am using the hydromad cwi with hourly data. I am wondering whether you have resolved your problem and how. Thanks very much.

Best regards,

Binru

在 2018年1月29日星期一 UTC上午10:52:26，Thelma Baddoo写道：

[Thelma Baddoo]

Dear Binru,

I have still not been able to resolve my problem with using the hydromad cwi with hourly data. I would be glad to have a sample of your R script if you have been successful with it.

Thank you.

[赵彬如]

Hi Thelma,

I plan to use hydromad cwi model with hourly data and see your question. I am not sure whether this model can be used with hourly data. I am thinking one problem. Is there any differences between using hourly data and daily data. If not,  may be we can change the time of hourly data to daily dates and run the model. Do you think it is feasible ?

Best regards,

Binru 

--
You received this message because you are subscribed to the Google Groups "hydromad users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hydromad-users+unsubscribe@googlegroups.com.
To post to this group, send email to hydromad-users@googlegroups.com.
Visit this group at https://groups.google.com/group/hydromad-users.
For more options, visit https://groups.google.com/d/optout.

[Thelma Baddoo]

Dear Binru,

Let me use my example in this case. I am trying to use hydromad cwi not just for hourly data but for event (flood) modeling and so it will not be feasible for me to change the hourly data to daily data. Mostly, daily data is used for continuous modeling and hourly data for event modeling. I am not sure of the feasibility of changing hourly data to daily data in this case. 

Thank you.

On Monday, 16 April 2018 16:52:16 UTC+8, 赵彬如 wrote:

Hi Thelma,

I plan to use hydromad cwi model with hourly data and see your question. I am not sure whether this model can be used with hourly data. I am thinking one problem. Is there any differences between using hourly data and daily data. If not,  may be we can change the time of hourly data to daily dates and run the model. Do you think it is feasible ?

Best regards,

Binru 

2018-04-16 9:44 GMT+01:00 Thelma Baddoo <mzded...@gmail.com>:

Dear Binru,

I have still not been able to resolve my problem with using the hydromad cwi with hourly data. I would be glad to have a sample of your R script if you have been successful with it.

Thank you.

On Monday, 16 April 2018 16:25:32 UTC+8, zbr...@gmail.com wrote:

Hi Thelma,

I am using the hydromad cwi with hourly data. I am wondering whether you have resolved your problem and how. Thanks very much.

Best regards,

Binru

在 2018年1月29日星期一 UTC上午10:52:26，Thelma Baddoo写道：

Hi group,

I am a new user of hydromad and I am trying to use hydromad cwi on hourly data. I have followed all the instructions in the hydromad tutorial and some suggestions from this hydromad group. However, when I run the fitbyOptim function, I get this error:

Error in lm.fit(x = z2[, -1, drop = FALSE], y = z2[, 1]) : 

  0 (non-NA) cases

I don't know what I have done or am doing wrong. Any help will be greatly appreciated.

I have attached my raw data (2000.csv) and my hydromad R script (hydromad trial script.R) for further details.

Thank you

--
You received this message because you are subscribed to the Google Groups "hydromad users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hydromad-user...@googlegroups.com.
To post to this group, send email to hydroma...@googlegroups.com.

[赵彬如]

Hi Thelma,

Thanks very much. If you have solutions, please let me know. Thanks very much.

Best regards,

Binru

To unsubscribe from this group and stop receiving emails from it, send an email to hydromad-users+unsubscribe@googlegroups.com.
To post to this group, send email to hydromad-users@googlegroups.com.

[Thelma Baddoo]

Dear Binru,

I will when I do.

[Thelma Baddoo]

Dear Professor Willem,

As per your suggestion, I tried the hydromad cwi on hourly data available in the package (Wye data). However, this data is continuous hourly data with over 17,000 data points (rainfall and runoff with no ET).

1. I did a first trial and I got an error due to the lack of ET values. 

2. I therefore added an arbitrary ET and used a little over 200 points with sriv and got the error

Error in if (k != round(k)) { : missing value where TRUE/FALSE needed

3. I went ahead and removed the sriv from the model and got the error

Warning messages:

1: In nseStat(aggrQ, doaggr(X), ref = aggrRef, ...) :

  merged time series have no data; incompatible times?

4. I then decided to use more data points (about 2000) from the same Wye data.

The model run well with no errors. However, when I ask for model output I get another error

Error in summary.hydromad(WyeModFit) : 

  model parameters not fully specified

 

I have attached pictures of the processes and the final script for your viewing.

This proves your suggestions about sriv and the number of data points. If this is the case then I don't know if hydromad cwi is feasible for event modeling because none of my data points is upto 2000.

[Willem Vervoort]

Dear Thelma,

I found one mistake in your  code, but as I don't have any experience with hourly data, I cannot really help. So while I can get the model to fit, it does not yet converge and this is what the your final error (4) relates to.

The mistake in your code is:

1. You cannot use the standard hmadstat() in your fit as this is a combination of the monthly r.squared and the r.sq.sqrt. As you are fitting hourly data, the monthly r.squared of course does not work. So you need to use hmadstat("r.squared") or any other. I made the same mistake earlier but never went on to use hourly data

https://groups.google.com/forum/#!topic/hydromad-users/FnC5VUS2e8U

I can't get fitByOptim() to work, for some reason it gets stuck in fitBySampling().

My guess is that this is related to the ranges of the parameters that you are fitting: tau_s and tau_q are time parameters, normally in days, so you need to make a guess what they would be in hours. As a result the initial guesses are likely to be very far away from the actual values and I think fitByOptim struggles with this

Willem

--

You received this message because you are subscribed to the Google Groups "hydromad users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hydromad-users+unsubscribe@googlegroups.com.
To post to this group, send email to hydromad-users@googlegroups.com.

[Thelma Baddoo]

Dear Professor Willem,

I have established that because the data points of my data are not much, I cannot use hydromad cwi on my flood modelling research. However, I would like to know where in the script code to change the hmadstat() options in case someone else has a longer hourly time series. Also, what would you suggest the best calibration function will be in cases of hourly data.

Since using hydromad cwi on hourly data is challenging, does this mean I cannot use the other modules in hydromad with hourly data? 

Thank you for all the help Professor Willem. I am very grateful.

To unsubscribe from this group and stop receiving emails from it, send an email to hydromad-user...@googlegroups.com.
To post to this group, send email to hydroma...@googlegroups.com.

[Willem Vervoort]

Dear Thelma,

as I indicated earlier, i don't really have any experience running hydromad with hourly data, I generally use it for daily data, so i can't really answer your question.

The cmd module should work if you can define the parameters again to be on an hourly basis (some are once again time parameters).

for hmadstat, just check ?hmadstat, it is in the fitByOptim() definition

it is the "objective"

Willem

To unsubscribe from this group and stop receiving emails from it, send an email to hydromad-users+unsubscribe@googlegroups.com.
To post to this group, send email to hydromad-users@googlegroups.com.

[Thelma Baddoo]

Thank you Professor Willem.

[docw...@gmail.com]

dear Thelma

have you completed your research   by hydromad with hourly data ??

[Thelma Baddoo]

Hello. I have completed my research with hydromad with hourly data. 

--
You received this message because you are subscribed to a topic in the Google Groups "hydromad users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hydromad-users/cnEpNwmH4Dw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hydromad-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hydromad-users/90b34eb9-cf6f-4e7c-9c34-9682b167554fn%40googlegroups.com.

[Thelma Baddoo]

Dear all,

I have an update of the solution to using hydromad for hourly data and event modeling. The previous codes I was using were updated and corrected by Mr. Takuya Iwanaga of the Australian National University. 

The issue with  my hydromad codes for hourly data as mentioned by Professor Willem is that

" The mistake in your code is:

1. You cannot use the standard hmadstat() in your fit as this is a combination of the monthly r.squared and the r.sq.sqrt. As you are fitting hourly data, the monthly r.squared of course does not work. So you need to use hmadstat("r.squared") or any other. "

 

So instead of using the default monthly fit statistics, Mr. Iwanaga helped me with code for daily and hourly fit statistics which work perfectly.

Please find attached the updated codes. And you can put any questions here for us to solve together.

[docw...@gmail.com]

Hello Thelma

 

How should I go about calibrating the model on several events at the same time? (i.e. having a single value for each parameter of all the period of my data)

Best regards