The difference is two-fold.
Pets are part of the process going back to Man's invention of animal husbandry while evolving from a hunter-gatherer species, in which humans take responsibility for the proper treatment, health, welfare and acts of specific animals. Many human governments impose fines and imprisonment for pet owners who fall down on that job. Most pets are incapable of adjusting to the wild; I'm among millions of people who have trapped feral cats near my house for neutering/spaying to reduce the suffering caused by uncontrolled breeding of former pets outside homes.
Game animals are generally predated on not just by humans, but other wild species (this includes microbes and parasites - any offshore fisherman can, while cleaning fish such as drum, find some pretty nasty wildlife growing in the fish). The existence of game is tenuous for many reasons outside hunting and includes instances of slow starvation and death inflicted by other game animals of the same species during dominance behavior.
Hunting and fishing are indeed painful to the animals who are shot or hooked, but hunters mostly use guns in calibers which cause a prompt, relatively painless death (my older brother was a trapper of nutria and muskrat on weekends, so I know not all hunting is that humane, but trapping is also a very small part of hunting).