OK, let's take a step back.
You have both Apache and Shiny Server configured to listen on port 80. That is certainly not going to work. I think the first question is whether you actually need Apache for anything. The simplest would be to shut down Apache (or configure it to stop listening on port 80, and reload) and restart Shiny Server, and I think that should start working.
If you do need to have Apache listening on port 80 (for example if you have other web stuff you are serving using other URL paths on the same host/port) then what you want to do is not serve up a <Directory>, but rather, proxy that traffic through to Shiny Server. To do this, change Shiny Server back to listening on port 3838 and remove the server_name directive, then set
ProxyPass and
ProxyPassReverse directives in your Apache config. Probably something like this:
If that doesn't work, first try viewing port 3838 from inside the firewall to see if the Shiny Server instance is even working. If that works, then check the Apache logs to see why the proxying is failing.