Another way to do this is to remove the color aesthetic from the ggplot() call and move it to geom_point() instead:
ggplot(mtcars,aes(x=disp,y=mpg)) +
geom_point(aes(color = as.factor(cyl))) +
geom_smooth(method = lm)
Placing the color aesthetic in the ggplot() call means that it will be expected to be present in *all* subsequent layers, which forces one to use group = 1 as an aesthetic in geom_smooth() to override the expectation of separate lines by levels of the color aesthetic.
Both methods are instructive and worth understanding: e.g., if I had multiple layers in a ggplot, most of which used color (say, for example, colored boxplots and points) then I'd probably put the color aesthetic in ggplot() and use the group = 1 device for geom_smooth() as Ben showed. OTOH, if I only wanted color in one or two layers of a multi-layer ggplot, I would declare them as aesthetics in the geom(s) of interest and not in ggplot(), as in my code above.
Dennis