Hello!
It should be safe to delete entries in the cache directory. They are used when fetching directories as artifacts (where it zips them, etc). It would be better to shut down the GoCD server and then delete them. You could move/rename the directory for a little while, if you're uncomfortable with that.
The pipeline run history is in the database, and not in the artifacts directory or in console.log. So, by you removing the console.log file, it means that you won't be able to access console logs, but the history (say, at a stage level) remains and you can reach. I suggest removing or renaming one of those files or directories first to understand what happens.
Performance usually doesn't depend on the number of artifacts, but probably on other aspects such as the size of the database, the number of pipeline runs, etc. Disk IO, memory, CPU, number of API and user requests, etc. can also have an effect. You could also look at the output of /go/api/support to see what is taking time, when you're trying to open the console log.
Cheers,
Aravind