Well you can, but the device will certainly crash ;-)
You can enforcing VNDK rules as long that API/ABI and VNDK's name space are respected, there should be OK (well I believe).
My assumption is that you have issues with *.bp files somewhere or missing something when declaring VNDK_SP_LIBRARIES section in device directory ;-)
*** in your source environments
1. Check if you don't miss any *.bp files, or any "name space" typo issues in the cc_binary sections?
2. Check if the "shared_libs" sections are correctly declared in your *.bp files
(take a look at /system/bt/vendor_lis as example)
*** in /device/<platform> directory
3. Check if all your vndk-sp libraries are built. (Android.mk)
4. If you've declared BOARD_VNDK_VERSION then you have to copy the LL_NDK libraries by your own!
(take a look at /device/google/marlin/vndk/Android.mk)
5. Check if all your libraries are declared in VNDK_SP_LIBRARIES somewhere
Hope this help
Chinda