Displacer beasts are explicitly not invisible; they possess a light-bending glamour. Invisibility may grant a 50% miss chance (just as the displacer beast possesses), but that does not mean that you cannot see the displacer beast. Truesight Goggles (Ultimate Equipment, p. 227) are 184,800gp, which you shouldn't be able to afford until around 13th level or so.
You are correct that Truesight Goggles or gold dust would have been ridiculous to carry on your person at any given time, excepting if you absolutely knew that you might be fighting a displacer beast, that you explicitly needed gold dust to handle it, and that you were conveniently provided with a reasonable way to acquire some.
Long story short, your GM took advantage of you and blamed it on you. This is the mark of poor GMing. While I wouldn't necessarily ditch this fellow completely, I would talk to him about how this was not a fair or fun way to present a challenge to you, nor was it consistent with established rules for the foe in question. If he repeats this behavior, I would stop playing with him.