Hello Shlomit.
GitLab doesn’t have the captcha feature yet, but it’s already been suggested http://feedback.gitlab.com/forums/176466-general/suggestions/4493103-captcha-at-login-signup-screen
You can maintain tight control over the number of active users on your GitLab installation, here's documentation about how to enable it: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/raketasks/user_management.md#maintain-tight-control-over-the-number-of-active-users-on-your-gitlab-installation
Regards,
Karen