If someone told you that, they didn't tell you the whole story.
First, I should note that carefully performance tuning code for 6g/8g may be silly at this point. They don't do as many optimizations as a lot of other compilers, so you can get measurable speedups by inlining functions or restructuring loops/conditionals in somewhat arbitrary ways. This may change (I heard rumors of inlining being on its way), and once gccgo is working you'll probably be able to get better performance without odd tweaks. So, unless you are just doing it for the sheer fun of optimization, you're better off writing your code the way that makes the most sense.
Anyhow, passing pointers vs passing values. One problem is that in many cases taking the address of something results in it being allocated on the heap. As it turns out, memory management in Go isn't too fast yet, so this can be a major slowdown. However, if you're passing the same thing around a lot, this is amortized. The advantage of passing pointers is that you potentially have less copying to do, but copying can be pretty darn fast, and with values it can be easier to safely parallelize. I think the relative performance depends very much on your specific case. Avoid allocation where you can, but use what construct makes sense, and tweak things if your benchmarks or generated code
give you a good cause to.