Switch unversions but doesn't delete added directories?

[Nan]

Hi folks --

I have a repository where I copied the trunk to a branch, then added
some directories to trunk and committed them.

Now when I "svn sw" back to the branch, the directories become
unversioned but are not deleted (which in turn causes problems when
switching back to trunk, not to mention problems with compilation and
such).

Is this expected behavior? Have I done something wrong? Is there a
workaround? Google only brings up questions from people asking about
directories that were unversioned before the switch.

SVN client is v1.6.16 on OS X. Server is v1.6.12 on an unidentified
*nix system.

Thanks!
Nan