You're problem is in the mapping file. A convention won't overwrite a
mapping if there is one, this makes it easy in a convention to set
defaults if a value hasn't been provided by the mapping but without
having to ensure you're not overwriting an existing mapping.
In you're mapping itself you're telling it the custom type for your
property is your "EnumConvention" hence the convention won't be able
to change that. Secondly, conventions are automatically run against
all the mappings in the project so you don't have to tell it to apply
the convention.
So in you're mapping it should just read:
Map(x => x.QueueType, "queueTypeID"); //Custom convention will set
the CustomType
Or,
Map(x => x.QueueType, "queueTypeID").CustomType(typeof
(QueueTypeEnum)); //Set explicitly, convention won't change this