There is an existing gluster role in the edx/configuration repo for setting up a mirrored gluster configuration on servers, but unfortunately that won’t fully work with the platform as a replacement for S3 as there are direct s3 calls in several places in the platform. For example, the ORA2 file upload feature and the grade reports feature are more or less directly tied to S3 (or other S3 compatible store).
However there are a few places you can use a shared local file system if you don’t need those features. For example forum image uploads and our SGA XBlock’s uploads can use any Django storage provider to store files, including a regular file system. This is done in the devstack settings for instance and you could replace that path with one to your gluster mount point. You would want to override this setting in aws.py instead of devstack.py for a production setup.
As an aside, what we use gluster for at MITx Residential and why we wrote the ansible role to install it, is to store our courseware static content instead of the default MongoDB to avoid the painful performance price of GridFS, but this is rife with peril and I wouldn’t generally recommend going there.