Anytime that a retrieve (or deploy) is taking a long time, it's almost certainly due to server-side queueing. You can confirm that by
enabling debug logging for Metadata Retrieval and Salesforce DX, reproducing the behavior, and looking at the resulting
idea.log which will almost certainly show that the retrieval request has been submitted and IC is just polling the server for status. Moreover, that status will likely be reported as QUEUED, PENDING, or similar.
If that's in fact the case, you will need to log a support case with Salesforce, specifying your org and instance IDs so they can figure out why things are queued so badly.
If you don't see what I've described, feel free to attach or
email the
idea.log for review.
Regards,
Scott Wells