tl;dr;
Yes the flyweight directory is safe to delete. Make sure to shutdown your server to avoid any issues later. Depending on the number size of materials, your server may take several minutes to clone these again at startup.
The long version
For some centralized SCMs like subversion and TFS, it's possible to do an update-check on a remote repository URL. For distributed SCMs like git and mercurial, the only way to check if new there are any new commits (and see the contents of the commits) is to clone the repository. These clones live in the flyweight directory. The server currently does not purge the contents of the flyweight directory when a material is removed.