- Create a Google Group,
- Add the users you want to have access to your extension as group members,
- Then set your extension visibility to private by editing your extension item from the Chrome Web Store developer dashboard
- Select your Google Group in the drop down list (all groups you own should be listed here)
- Save your change
- Once the change is effective give the extension link to your colleague!

Note that, from what I experienced using this with Google Groups on G Suite domains, it can take some time for the permission to propagate when you add a member to the group.
So it is possible that the user you added doesn't have immediate access to the item in the CWS (in that case he will get a 404 error page).
Also ask your colleague to ensure he is logged in in the Chrome Web Store with the account you added to the group (otherwise he will also get a 404 error page)