vDSP_biquadm_SetCoefficientsDouble says to immediately change the coefficients in use. At and around the point where you called it:
The most recent two inputs (at the end of signal1) were 15 and 16.
The most recent two outputs (at the end of signal1Out) were 14.77 and 15.7703.
The next input (at the start of signal2) is 15.
Then, computing with the new coefficients (1, 2, 1, .89766, and .527187), the next output is:
1 * 15 + 2 * 16 + 1 * 15 - .89766 * 15.7703 - .527187 * 14.77 = 40.0571.
This is as you observed.
In contrast, vDSP_biquadm_SetTargetsDouble says to “gradually” change the coefficients toward target values. In the intended use, what would happen here is that the next value would be computed with the old coefficients and would yield the same values as above, and then the coefficients would be changed slightly, and then the next value would be computed with the changed coefficients, and this would continue as the coefficients approached their target values.
However, you called vDSP_biquadm_SetTargetsDouble with 2 for the threshold value. This value says “when the changing coefficients are within this threshold of the target value, call it close enough and jump them the rest of the way.” A problem here is that the old and new coefficients already differ by less than 2 (e.g., 1-.6389 = .3611). vDSP_biquadm was not implemented with this in mind (and it could probably be considered a bug); it expects changing the coefficients to take several iterations at least. What resulted was a code path where vDSP_biquadm immediately used the new coefficients but used them with partially calculated sums already in progress. Thus, it used an unintended mix of old and new coefficients.
I am not sure why the resulting values look like something you expected. They are not intended to have those values by design of vDSP_biquadm.
—edp (Eric Postpischil)