Hi Alex,
1. When you generate the genome, you need to use --sjdbGTFfile <annotation.gtf> at the genome generation step. Also, please check that the chromosome names are the same in gencode.gtf and ENSEMBL whole_genome.fa files. Gencode uses "chr" in chromosome names, while ENSEMBL does not, so you have to be careful.
2. If your genome generation with --sjdb* option works, STAR will be mapping to transcriptome and genome simultaneously, and will select the best alignment. You do not need to "map to transcriptome first, then map to the genome".
3. If you really want to map to transcriptome, you would need to generate a genome with sequences of all annotated transcripts, each transcript will be a separate "chromosome". Note, that alignments in the .sam file will be given in the "transcriptome" coordinates. The spliced reads within the transcriptome, mostly non-canonical, will correspond to unannotated (novel) junctions.
Cheers
Alex