Error in calibrating hydromad cwi model using fitbyOptim on hourly data

72 views
Skip to first unread message

docw...@gmail.com

unread,
Jan 11, 2021, 5:47:57 AM1/11/21
to hydromad users
hello I have a problem in the calibration with the fitByOptim function knowing that I work with the CWI model, the error is shown below:

Error in if (k != round(k)) { : 
missing value where TRUE/FALSE needed

Thank you  


Willem Vervoort

unread,
Jan 11, 2021, 5:15:59 PM1/11/21
to hydromad-users
Hi,
without a simple reproducible example, it is very hard to find out if this is a coding problem or another problem

Regards
Willem

--
You received this message because you are subscribed to the Google Groups "hydromad users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hydromad-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hydromad-users/c3601363-9e91-4d4d-8a69-244f57c39fe3n%40googlegroups.com.

wafa doc

unread,
Jan 12, 2021, 5:00:37 AM1/12/21
to hydroma...@googlegroups.com
Hi Dear Willem
here is my script and my data:


scripte.R
t_10-2000.csv
pq_10-2000.csv

Willem Vervoort

unread,
Jan 12, 2021, 5:24:18 AM1/12/21
to hydromad-users
I think your problem is that you are trying to fit the model on 24 hours with a 20 hour warm-up. It seems to me that is a very small number of data points, I don't think hydromad is set up to do that.
Maybe Joseph can comment on this
Willem

wafa doc

unread,
Jan 12, 2021, 5:40:31 AM1/12/21
to hydroma...@googlegroups.com

my data is discontinuous rainfall events, I have about 20 events like the one I sent you (I tried to run the code with a single event to understand abort), if you can give me an idea of how I can grab them all at once.


Willem Vervoort

unread,
Jan 12, 2021, 5:54:56 AM1/12/21
to hydromad-users
merge them together in a single time series with missing data in between
Willem

Joseph Guillaume

unread,
Jan 12, 2021, 8:48:41 PM1/12/21
to hydromad users
This was a rather difficult case to debug, but there are at least four problems:

1) As previously noted in this mailing list, hydromad does not support using POSIXct time indices. Use as.chron to convert the index to chron instead.
In this case, this is important because SRIV uses as.ts internally, and using POSIXct results in a one second resolution timeseries, with lots of NAs. This causes a whole lot of problems.
The simple fix in this case is:
index(ts24s)<-as.chron(index(ts24s))

2) Your code runs into a second error because of a bug in hydromad where the warmup parameter is being ignored in this particular circumstance (https://github.com/josephguillaume/hydromad/issues/169)
The temporary fix for this is to run
hydromad.options(warmup=20)

3) The code now runs (with warnings) but does not provide results.
The issue here is that the default objective function in hydromad includes a monthly NSE.
When using short time series the objective function needs to be changed, e.g.
hydromad.options(objective=hmadstat("r.squared"))

4) The performance of the optimised parameters is still poor with a warning that the system is computationally singular.
Exactly what is causing this is unclear, but you do have a problem with your data. Your existing code shows that your runoff is 160 times your rainfall:
> with(AinSefra, sum(Q[ok])/sum(P[ok]))
[1] 160.4862


In summary, you'll need these three lines for your code to run, but you'll need to do some more work on the actual data passed to the calibration.

index(ts24s)<-as.chron(index(ts24s))
hydromad.options(warmup=20)
hydromad.options(objective=hmadstat("r.squared"))

wafa doc

unread,
Jan 13, 2021, 10:40:34 AM1/13/21
to hydroma...@googlegroups.com
hi Mr Guillaume, thank you for your help, it's really kind of you. I did what you told me but I came across the problem of 50 wonings that you already spoke with Mr Willem if you can give me a solution on that you or Mr Willem.
the problem is:
> AinSefraFit <- fitByOptim(AinSefraMod, samples = 20,
+ method = "PORT")
  0:    0.80626184:  42.1053  6.31579
There were 48 warnings (use warnings() to see them)
>  warnings()
Messages d'avis :
1: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 4.16193e-25

2: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
3: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 5.72771e-24

4: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
5: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 4.07479e-24

6: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
7: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 2.80154e-24

8: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
9: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 3.18148e-24

10: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
11: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 6.55247e-25

12: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
13: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 2.00333e-24

14: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
15: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 7.36443e-25

16: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
17: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 3.59979e-25

18: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
19: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 1.02822e-24

20: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
21: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 6.33273e-25

22: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
23: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 1.05042e-24

24: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
25: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 2.08535e-24

26: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
27: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 4.43671e-24

28: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
29: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 6.22984e-25

30: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
31: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 1.63612e-24

32: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
33: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 5.78408e-24

34: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
35: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 1.32612e-24

36: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
37: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 3.84414e-24

38: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
39: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 1.72208e-24

40: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
41: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 4.16193e-25

42: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
43: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 1.82662e-25

44: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
45: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 6.61044e-25

46: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations
47: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  Error in solve.default(xz, xQ) :
  le système est numériquement singulier : conditionnement de la réciproque = 4.16193e-25

48: In do_srivfit(DATA, init.model = init.model, order = order,  ... :
  did not converge after 1 iterations

best regards

You received this message because you are subscribed to a topic in the Google Groups "hydromad users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hydromad-users/sGdxOnlS5AU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hydromad-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hydromad-users/CADfpD61BfLFyHKXdyeqOFL2LFYX5sqxNEGQQy9VsaKxSuME%3DRw%40mail.gmail.com.

Joseph Guillaume

unread,
Jan 13, 2021, 4:28:43 PM1/13/21
to hydromad users
Hi Wafa,

How did you address the fourth problem?

Your existing code shows that your runoff is 160 times your rainfall:
> with(AinSefra, sum(Q[ok])/sum(P[ok]))
[1] 160.4862

I'm guessing the unit conversion was incorrect.

Cheers,

Joseph


Barry Croke

unread,
Jan 13, 2021, 6:33:20 PM1/13/21
to hydroma...@googlegroups.com

G’day Wafa,

 

The problems of “singular” warning message is probably due to the use of a short time period being used to calibrate the model. The version of the SRIV algorithm included in Hydromad uses the en bloc method rather than the recursive method. The SRIV method includes a matrix inversion, and with the en bloc method, the matrix can be close to singular (very small determinant) if too short a period is used for calibration. There is no fixed guide on how long a period is needed to ensure the determinant is large enough to avoid the singularity warning as this depends on the flow and effective rainfall time series. However, generally we use a few years to calibrate a model. Calibrating on individual events may be problematic using the SRIV en bloc approach. Options are to

  1. Use the approach that Willem suggested (i.e. combine the events into a contiguous time series) and treat as a single calibration period. Flow data can have missing values, but rainfall and temperature/potential evaporation should not. Note that while you could fill the time between events with zero rainfall and potential evaporation, this can lead to errors in the antecedent conditions leading into each event.
  2. Use a different calibration method, noting there are a few included in Hydromad

 

Barry

BERDANE, Kheira

unread,
Jan 13, 2021, 6:45:11 PM1/13/21
to hydroma...@googlegroups.com
Hi Mr barry
I have not a rainfall data between events how i should do?

wafa doc

unread,
Jan 13, 2021, 7:15:48 PM1/13/21
to hydroma...@googlegroups.com
I'm sorry Mr Barry I sent the question from my second email; Thank you for your help

best regards 

Reply all
Reply to author
Forward
0 new messages