Yahoo's OAuth implementation appears to be a bit non-standard.
I was able to get it to work with GTMOAuth with a few small changes.
Set the scope and the display name to nil, like
windowController = [[[GTMOAuthWindowController alloc] initWithScope:nil
language:nil
requestTokenURL:requestURL
authorizeTokenURL:authorizeURL
accessTokenURL:accessURL
authentication:auth
appServiceName:kYahooKeychainItemName
resourceBundle:nil] autorelease];
[auth setDisplayName:nil];
Also, be sure that the callback matches the URL registered with Yahoo:
OAuth 1 debugging is unfortunately a bit of a black art. I have added a "Debugging Tips" section to the Introduction wiki page for the project.