It can be an issue indeed. My hypothesis is that it comes from the Matrigel composition since you will have batch to batch variability and some Matrigel vials are more protein concentrated and could be "heavier" and therefore settle down with the organoids during a spin.
If you really struggle and find that you still have a lot of leftover hydrogel around the organoids, I have found that doing a couple of extra washes with PBS helps a little bit. In a nutshell:
1) Try and extract as per protocol (remove conditioned media and put aside).
2) Remove as much hydrogel on top of the organoid pellet by pipetting it out with a P1000 pipette. Leave about 2-3mm hydrogel layer on top of the organoids to be sure not to pipette them out (the organoids will break if taken out with a P1000 tip).
3) Transfer the organoids + leftover hydrogel mixture to a 1.5mL Eppendorf tube with a Pasteur pipette.
4) Top up to 1mL with PBS.
5) Very gently mix the organoids and PBS together (~10 times) with a Pasteur pipette. It will help to break the hydrogel layer.
6) Spin 5min/RT/400g
7) Remove supernatant carefully
8) Repeat steps 4-7 one more time. I wouldn't do more than one time in case this is too disruptive for the organoids: no real data here, just a feeling ;)
I hope this helps!