Hmmm... I am able to open a directory using:
CreateFile(name, GENERIC_READ, 0, 0, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, 0)
A second call to CreateFile with the same parameters results in ERROR_SHARING_VIOLATION. Therefore the ShareAccess parameter is important for directories too.
Now Dokany does not seem to call OpenDirectory() in this case because the Win32 CreateFile does not specify FILE_DIRECTORY_FILE when opening directories. But Microsoft has now documented NtCreateFile, etc. and many applications use them. So I can see a case where we get a FILE_DIRECTORY_FILE flag with restrictive ShareAccess, which is then routed to OpenDirectory().
The real fix here may be to simply remove OpenDirectory(). I believe it confuses matters and makes life harder for file system implementors. I mean we have to handle opening directories in CreateFile anyway! So now we have to handle opening directories in two methods (CreateFile, OpenDirectory) and in OpenDirectory() we do not even get all the information to do access checks, etc.
This ties into the discussion of adding the CreateOptions parameter to CreateFile. If the CreateOptions parameter was there, there would be little need for OpenDirectory (or CreateDirectory).