It's not built in, but there's /diag/eval snippet that does it:
wget -O- -q --user=Administrator --password=asdads --post-data="KV = ns_config:get_kv_list(), KV2 = lists:keydelete(vbucket_map_history, 1, KV), ns_config:cas_config(KV, KV2), KV2."
http://lh:9000/diag/evalroot@beta:~/src/altoros/moxi/ns_server# wget -O- --user=Administrator --password=asdasd --post-data='{ok, Config} = ns_bucket:get_bucket("default"), Map = [lists:sublist(Chain ++ [undefined,undefined], 3) || Chain <- proplists:get_value(map, Config)], ns_bucket:update_bucket_props("default", [{map, Map}, {num_replicas, 2}]).'
http://lh:9000/diag/eval
make sure to replace lh:9000 with your host/port and admin name and password. Bucket name is default. You can replace it as well. Magic constant 3 is number of replicas + 1 and magic constant 2 is number of replicas.
After that snippet is run, wait 10 seconds (that's janitor runs period). On UI in server's page you'll see Rebalance button enabled. Hit rebalance and it'll actually populate those extra replicas.
Use it on your own risk. I've found it to work and we've used it once in production to decrease number of replicas and that worked well.