First, thank you for all the hard work you've done to create excellent apps for Android users. :) I hear your frustration, so bear with me as I offer some general background before digging into your specific question.
Before KitKat, public APIs only supported a single primary external storage device, which on some devices can be emulated and backed by internal storage. Primary external storage devices continue to work the way they always have since API level 1; if you request WRITE_EXTERNAL_STORAGE you have full read/write access across that entire device.
We've come a long way since API level 1, and looking back we found that some apps would scatter their files across external storage. If an app was uninstalled, these files would be orphaned, and users had to manually scavenge through all these files to free up disk space. This was a bad user experience, and something we wanted to avoid repeating.
So when we added APIs for secondary external storage devices in KitKat, we carefully limited write access to directories belonging to the running app. This gives us the best of both worlds: apps can take advantage of the extra storage space, and users have better visibility into and control over how apps are using their storage space. (These package-specific directories are surfaced to apps through Context.getExternalFilesDirs(), etc.)
Before KitKat, some OEMs chose to leave external storage devices unprotected, where apps could write if they guessed a special filesystem path. These were never part of any platform public API; they were an internal implementation detail. We've always maintained that internal details like this can (and will) change without warning, and that apps should not depend on these hidden behaviors.
Now, on to your specific question: there are many ways that your two apps could coordinate to share/manage files between themselves. One way would be to use a ContentProvider to offer access from the first app to the second. We've encouraged the use of ContentProviders like this because they support granular access controls through temporary Uri permission grants. FileProvider is a good place to start, since it already handles the details related to sharing files, including support for deleting:
FileProvider may not be a perfect fit, but it's a good starting point for writing a custom provider. Using a provider creates a cleaner contract between the two apps, offering an abstraction layer around where/how the files are actually stored, and a place to surface metadata such as download progress, etc.
If you'd like to keep things simpler, another approach would be to define an internal protocol between your two apps, where the second app could request that the first app delete a specific file, perhaps through a broadcast that you protect with a permission.
Hopefully the details above shed more light on why these behaviors exist, and hopefully one of the strategies described above can help improve the interaction between your two apps. :)
Justin Tipton here. Creator of iSyncr and Rocket Player. Two top ten music applications with several million active users. One application allows syncing your iTunes music to your phone's SD card over USB or WiFi. The other, Rocket Player, allows for a great listening experience, including the ability to edit music tags and delete songs.
If I understand you correctly, under KitKat, iSyncr should sync all music over WiFi that the user wants on the SD card to "<SDCard>/Android/data/com.jrtstudio.iSyncr".
Then, when someone uses my Rocket Player app to delete a song, I should toast and say "The Android team doesn't want you to be able to delete this song", because Rocket Player can only access "<SDCard>/Android/data/com.jrtstudio.RocketPlayer", and not "<SDCard>/Android/data/com.jrtstudio.iSyncr".
I'm I understanding this correctly? Apps that sync data wirelessly are confined to a single directory on the SD card that no other apps can access without APIs being built into app that synced the files?
I hope I'm wrong. I really really do.
(P.S., Users are playing files that the media scanner doesn't pick up. Formats not natively supported Android. Do not suggest the MediaStore as a work around for music applications. It isn't.)