You're trying to do too many steps as one. You need to do the following (at a very high level):
1) Install 10g to same version as you have on the old server (look up cloning an Oracle Home)
2) Ensure you have created a pfile from spfile and copied your *.ora files from the DBS & Network/Admin directories on the old server to the same locations on the new.
==>>> BE sure to comment out in the PFILE any deprecated parameters (look at the errors you got) AND rename the SPFILE to something like a ".orig" extension so Oracle does not try to startup with the spfile.
A) Copy the original data files directly to the new server (with the Source DB shut down or in "Backup" mode) *OR*
B) Do the backup on the old server/copy to new server
3) USING the 10G Executables If 2B, then restore & recover db, then open resetlogs.
4) USING the 11G Executables, Perform the upgrade using dbua or if you know all the steps it does you can do them manually.
I've done upgrades over 100 times starting all the way back from 7.3 to 11.2.
Jim
Certified Oracle DBA
Oracle Certified Specialist
Oracle Trainer