Darwall's addressing of the harm/benefit objection proposes that according to Act-Utilitarianism, all other things equal, it is not worse to cause harm than it is to provide a benefit. I wish Darwall would have devoted more space to this principe because it seems, upon further thought, to go against our moral common sense. Or at least mine. It seems rational to believe that doing harm to one is worse than doing good to another, even if they are supposedly of the same measure. It seems, according to common sense, that inflicting pain/bad/harm/etc. to someone is worse than doing an equal amount of pleasure/good/happiness/etc. to another simply due to the nature that it is bad you are inflicting bad. For example, suppose I were to punch one baby but give another baby a room full of teddy bears. And suppose these acts would inflict equal amounts of harm and good, respectively, given there was indeed some way we would evaluate this (hedons, possibly?). According to AU I have done nothing wrong. However, it is against our commonly-held societal norms to punch a baby, and I would expect that people would not think my actions justified even after I informed them of all of the happiness that the teddy bears provided the other baby. It seems that, if I were to extrapolate this a bit, I could go around distributing teddy bears to some babies while hitting others and be a morally neutral person.
I am aware that I have done a poor job of expressing this in writing (and perhaps I am interpreting Darwall's writing in another way than is meant), but to me this just seems to go against our ingrained moral code: that doing bad is bad in and of itself. There just seems to be something extra hurtful about inflicting harm.