There's a minimum average speed, distance/time, as noted above. There will normally be intermediate controls with their own cut-off times. As long as you make the times, it doesn't matter how your speed varies along the way. So going slow up a hill and fast down, or slow into the wind and fast with a tailwind, are expected.
If you look up past results for the ride in question, you can find out how fast people typically ride the route.
If you anticipate actually needing all of the allotted time, it can be a real problem, due to the wind/hills, stopped time, etc. It helps to have some leeway in there.