Hello Pierig,
Unfortunately, you can't limit the access to your Load Balancer's external IP address in the GCE firewall. However, you can use Google's internal Load Balancer which allows you to accomplish different scenarios, one of which is the traditional 3-tier web services where your tier uses external HTTP(s) or external TCP/IP load balancing and your instances running the application tier or backend databases is deployed behind the internal load balancing.
More information can be found in this Help Center
article.
I hope this helps.
Sincerely,
George