Don't know if it's the "supported" way to do it, but I simply removed
the "embedded" version of rack that ships with puppet-dashboard:
cd <path-to-dashboard>
git clone git://github.com/puppetlabs/puppet-dashboard.git
rm -r vendor/gems/rack-1.0.1
sed -i -e 's,~> 1.0.1,~> 1.2.2,' vendor/rails/actionpack/lib/action_controller.rb
cp <my-config-path>/config.ru .
sudo service httpd start (or however you start apache/nginx).
Is there a recommended way to resolve this conflict?
~David Klann