We spent six years in the Med (mostly Greece and Turkey) and getting on for three in the Caribbean so far.
I think you will get any number of different views but for us I would say "The sailing is better in the Caribbean and the culture, ease of integration and food are better in the Med". We had some of the same musings as yourselves before we crossed but in retrospect we are really glad we made the break from Turkey, hard though it was at the time.
We arrived early in 2015 in St Martin and then went to the BVIs. It was busier and more American than we had expected there and we were somewhat disappointed. Landing at one end of the chain though is good and as we worked southwards things improved markedly. As is the case in most places it is hard to build relationships if you are only spending a few days in each place but we found good welcomes in Saba, the French islands (we could speak a bit of French which really helped) and especially Dominica. Mustique was not for us, St Lucia was patchy and St Vincent dodgy. You generally have to be a bit more street wise everywhere than in Greece and Turkey but we have never lost anything or had any big security issue. I have to say though that a few amphitheatres and colosseums wouldn't go amiss in the Caribbean!
We've spent our hurricane seasons in Grenada and that has given us a chance to really get to know some of the indigenous population both expats and Grenadians. There's a buzzing social scene there if that is what you want. The Hash House Harriers meet on Saturdays. These are a great way to get to meet the local people as well as cruisers, ex pats and other sailors. We haven't found much 'western' culture - concerts, art exhibitions or museums but then you wouldn't really expect that. It's different - colourful, vibrant and easy going. Jump ups, beach parties, reggae (especially Bob Marley in the touristy places) and lots of rap?? You can help with reading with the children, join a steel band or sponsor a child through his education. Most islands are pretty poor once you get away from the tourist fringe, some are very poor. Most people are very welcoming and warm. The rum is excellent.
I was interested that you said you might stay 'a few years'. I think that for me would be the clincher. Like us you would probably spend the first year working your way down the main chain which is OK but maybe not worth the effort of crossing for. After that though you'll know people, know the places you like, have heard of other more out of the way places to visit and have an idea of where to go next. There's the ABCs (Aruba, Bonaire and Curacao), Central America, the Turks and Caicos, the Bahamas and the east of coast of America to explore - each one very different from the others.
The other thing to consider is the ocean passage itself which you certainly won't get if you stay in the Med. For me that's absolutely an experience not to be missed. But for other people that's a boring/frightening/full of hassle experience. It's certainly another factor to consider if you are thinking of spending time in the Caribbean.