Hello Jamie,
That is the way it is expected when playing Rockband.
Some audio/video setups will show more or less delay.
The calibration that you perform is applied to the sounds coming to the soundtrack, but not to the sounds coming from the drums played during the fills hence the difference in timing.
An option that might work for you is to lower the volume of "Instruments" in the game and use the sounds from your drum module. This is great when you play alone, but if somebody else is playing with you, let's say on a guitar, then it won't work because the guitar sounds will not be heard.