Hi Janet,
You're right, in Dataverse, a dataset cannot be restricted. Only files can be restricted. As you say, when a dataset is published the metadata is always available.
Can you please open an issue at
https://github.com/IQSS/dataverse/issues to indicate that this documentation should be revisited? If anyone out there would like to take a crack at editing that page, it can be found in the source tree at doc/sphinx-guides/source/admin/harvestserver.rst
As you are hoping, the data files themselves are not available for public download when they are restricted. The URLs to those restricted files are available in the metadata, but you would need an API token with the proper permissions to download them.
On a related note, there's a relatively new JVM option called `dataverse.files.hide-schema-dot-org-download-urls` that is related. It's documented at
http://guides.dataverse.org/en/4.12/installation/config.html#dataverse-files-hide-schema-dot-org-download-urls . It was introduced over concern that search engines might cause a lot of load on the server by downloading lots of files at once.
I hope this helps,
Phil