Setting up a local repo server to host an Android repository using Gerrit

588 views
Skip to first unread message

vishniakou

unread,
Nov 14, 2016, 7:55:29 PM11/14/16
to Repo and Gerrit Discussion
Hello,

I am in the process of setting up a local repo server using Gerrit. We have a need to host a local copy of the Android code base, which pulls data from the Code Aurora server (CAF). We have several developers that would share their changes to this code with each other through a local server.
I realize that there have been many discussions on this topic. I have looked through all of them, but have not found a single comprehensive guide on how to set this up. I am relatively new to git and repo, and have never worked with Gerrit before. I have found that I was not able to follow most of the existing guides end-to-end without facing some kind of errors, especially when it comes to things like working with manifest files, creating and uploading changes, using code review features, etc.

My goal is as follows:

Local server (may or may not be connected to internet - to be determined) - is to hold a copy of the code from the CAF. On CAF, there are regular updates that are coming out. We need to periodically add those updates to the local server, which I expect to be done manually. When this "update" is done, I expect that the existing code base, including the local changes to the code that the local developers are creating, will be merged with the upstream changes from CAF (conflicts can be resolved manually if needed).
Whenever a new release comes out on CAF, the manifest file changes. So the update process on the server should include some method to switch the existing code to the new manifest file.

Local clients (=developers) will only work with the local server, initially receiving the entire code base from the server, then modify it, then push their changes to the server. The clients will never communicate directly with CAF. When the server updates its own code base to get up to date with CAF, clients will have to subsequently get the new changes from the local server by running "repo sync" or similar.

So that's the setup that I am trying to create.
Currently, the following works:
Server:
- I am able to set up the Gerrit server using the 2.13.2 version of Gerrit, internal H2 database, and LDAP authentication
- I am able to create a clone of the CAF repository on the server using the "--mirror" flag (through command line, not through GUI)
- I am able to update the code base on the server from the upstream CAF repository using a new manifest file
Client:
- I am able to connect to the server and initialize a local repository from the local server
- I am able to create a new branch on client inside a specific git repository using repo start command
- I am able to create a change in the code on the client, and check in my change to git
- I am able to push my change onto the server from the client using the git push command. A new branch is successfully created on the server.
- I also seem to be able to get the new changes from the server back onto the client

Currently, the following does not work:
Server:
- The project for the Android code base was never created via the Gerrit GUI, and therefore not shown anywhere on the GUI. I am not sure how to do that properly
- I am also not sure how to prevent Gerrit from showing 400+ git repositories (once the project is properly added) for the Android code base in the GUI, and just show one master repo, and then show the "sub-repositories" if desired
- Any changes that client submits are not shown in the review GUI of the server
Client:
- I am not able to push the changes using "repo upload <repository>" command. However. I am able to push the changes to the server using the "git push" command.
- I am not sure regarding the correct URL to specify in the manifest file for the "review" functionality of Gerrit
- I am not sure how to *properly* receive the upstream changes from local server. What works now is to go through the same procedure again (shown below): revert the manifest.xml changes, run repo init with new manifest file, then again edit the manifest.xml and run repo sync. Is this the best way to do that?

I am hoping the some of the git uberlords here can help me resolve these issues. In turn, I will update this posting to incorporate the suggested changes to help others that are facing the same task as me and maintain the step-by-step feel of this guide.
I am also hoping that you could comment on this setup, and let me know if you see any problems with the current implementation.
I realize that a lot of manual stuff is going on right now. We have a very small team and it is fine for us for now. I just want it fully functional first. That said, I am very open to suggestions for improvements.


Here is a step-by-step procedure that I followed to set up the server and client to implement the configuration discussed above:

The server.
Ubuntu 16.04 LTS
sudo apt-get install git
sudo apt-get install default-jre default-jdk
sudo add-apt-repository ppa:webupd8team/java
sudo apt update
sudo apt install oracle-java8-unlimited-jce-policy
Note: not sure if the “unlimited-jce-policy is necessary”.

Could not get the postgresql to work with the gerrit server
Encountered an issue where every time a user logs in, a new user account is created, with the account ID being increased. This prevented the ability to access the admin account, because there was no way to use “existing user” to log in. Tried to use both ldap and OAUTH login methods, with the same results. Switching to H2 internal database that comes with Gerrit resolved this issue. Others in the IRC channel for gerrit said that mysql is working well for them, although I have not tried to use that.

Create paths on the server:
/home/server/gerrit
/home/server/repo


sudo adduser gerrit2
sudo su gerrit2
Navigate to the directory where the gerrit-*.war file is downloaded
java -jar /home/server/software/gerrit-2.13.2.war init -d /home/server/gerrit
Location of git repositories: /home/server/repo
Select LDAP method of auth
After completing the initialization screen, create bare repositories (otherwise, gerrit won't start)
cd /home/server/repo
git init --bare All-Projects
git init --bare All-Users
Now, should be able to start gerrit and log on to the web interface (actually, might have to wait until ldap server setup is completed below)
/home/server/gerrit/bin/gerrit.sh start
Log on to https://localhost:8443

Set up LDAP server
sudo apt install slapd ldap-utils
sudo dpkg-reconfigure slapd
<set up admin passwd, write it down>, user=admin
Domain = domain1.domain2.com
Check that setup executed correctly:
ldapsearch -x -LLL -H ldap:/// -b dc=domain1,dc=domain2,dc=com dn
Create file entries for creation of groups “groups”, “people”, and “dev”, and account “account1”
ldapadd -x -H ldap:/// -D cn=admin,dc=domain1,dc=domain2,dc=com -W -f ldap_add_people.ldif
ldapadd -x -H ldap:/// -D cn=admin,dc=domain1,dc=domain2,dc=com -W -f ldap_add_groups.ldif
ldapadd -x -H ldap:/// -D cn=admin,dc=domain1,dc=domain2,dc=com -W -f ldap_add_group_dev.ldif
ldapadd -x -H ldap:/// -D cn=admin,dc=domain1,dc=domain2,dc=com -W -f ldap_add_user.ldif
Can run the search command again here to check that the entries have been properly updated
Go to user setup page inside Gerrit webUI and set up the ssh access credentials for the new user account

Set up the initial repo revision
Idea: mirror the repository from the upstream using the “mirror” switch
mkdir -p /home/server/repo/android_test1
cd /home/server/repo/android_test1
sudo apt-get install repo
repo --trace init -u git://codeaurora.org/quic/la/platform/manifest.git -b release -m caf_AU_LINUX_ANDROID_<rest_of_filename>_013.xml --repo-url=git://codeaurora.org/tools/repo.git --repo-branch=caf-stable --mirror
repo sync

I noticed that during “repo sync” on the client, Gerrit produced an error during the sync of the linux kernel repository, which is the largest repository that is being used. The error was shown in Gerrit logs as something like “gc: heap memory exceeded”.
To solve this problem, edit the Gerrit config file, and add the following line.
In the [container] section:
heapLimit = 4096M
This increases the available RAM for the Java process, which allows the sync to run to completion on the client


When a new code release is available, these changes need to be integrated into the server's code base
This can be done using the following commands
cd /home/server/repo/android_test1
repo --trace init -u git://codeaurora.org/quic/la/platform/manifest.git -b release -m caf_AU_LINUX_ANDROID_<rest_of_filename>_016.xml --repo-url=git://codeaurora.org/tools/repo.git --repo-branch=caf-stable
Note that now, there is no “--mirror” flag attached to the repo command
repo sync
Now again, can still cd to the kernel/msm-3.18 repository and again issue git branch command to check that the original test branch that was committed from the client is still there.




The client.

The first thing to do on the client is configure the domain name of the server so that it is found.
sudo gedit /etc/hosts
Add the line:
192.168.0.37 domain1.domain2.com
Here, 192.168.0.37 is the local IP address of the server, and domain1.domain2.com is the hostname that we will give to the server. Every client has to make these changes locally, unless an actual DNS server is available on the local network. One more consideration to make here, if the IP address of the server changes, then every client has to update the /etc/hosts file. This includes the situation where the server was first connected to router via wi-fi, and later reconnected via ethernet, which could prompt the router to assign it a new IP.

Add email to user account
ssh -i ~/.ssh/id_rsa  -p 29418 acco...@domain1.domain2.com gerrit set-account --add-email acco...@domain1.domain2.com account1

Create <android_root> folder somewhere on client
cd <android_root>
repo --trace init -u ssh://acco...@domain1.domain2.com:29418/android_test1/platform/manifest.git -b release -m caf_AU_LINUX_ANDROID_<rest_of_filename>_013.xml --repo-url=ssh://acco...@domain1.domain2.com:29418/android_test1/repo.git –repo-branch=caf-stable
Cannot do repo sync yet, because the manifest file is still pointing to the CAF server. Must manually edit the manifest file. Note: the manifest file “manifest.xml” is actually a symlink to a specific caf_AU.....xml file inside the manifests directory.
gedit <android_root>/.repo/manifest.xml
At the top, in the line <remote fetch=...> change the link to:
fetch=”ssh://acco...@domain1.domain2.com:29418/android_test1” review=”ssh://acco...@domain1.domain2.com:29418”
Note: most likely, the review link is wrong, considering that the project "android_test1" was never properly created through the Gerrit GUI to contain other projects
repo --trace sync
Upon further testing, it was confirmed that the sync indeed is pulling data from the local server, and not from the CAF server. Testing done: server stopped in the middle of the “repo sync” command, which produces an error on the client. Also, noticed that the speed is significantly higher than that compared to the CAF server, because ethernet connection is now used. Observed speeds that approach 11MBit/s.


Start a new branch on the client. Not sure if this is necessary if the branch is already present on the server when initial “repo sync” is run on the client.
cd <android_root>
repo start test_branch kernel/msm-3.18
Starts a new branch called “test_branch” inside the kernel/msm-3.18 git repository (this command will take a while)
repo status
Check that the new branch is listed
cd <android_root>/kernel/msm-3.18
git status
Again, check that the new branch is shown by git
Edit some file. In this example, we will edit the Kconfig file
gedit Kconfig     <Make some change inside this file>
git status
At this point, it should say “On branch ...” and show Kconfig file in red, noting that it has changes.
git add Kconfig
git commit -m “Message for commit”
git status
Now, it should say “Nothing to commit”
git log
Should show the commit that was just created. Type “q” to exit
!!!!! repo upload kernel/msm-3.18
This command does not work! Shows error:
error.GitError: kernel/msm-3.18 rev-list ('--abbrev=8', '--abbrev-commit', '--pretty=oneline', '--reverse', '--date-order', u'^refs/remotes/caf/android-external.lnx.2.0.r5-rel', u'refs/heads/test_branch', '--'): fatal: bad revision '^refs/remotes/caf/android-external.lnx.2.0.r5-rel'

git push caf test_branch
This last command succeeds! This is confirmed on the server, since we are able to do the following:
<on the server>
cd /home/server/repo/android_test1/kernel/msm-3.18.git
Note: have to include “.git” here, because it was created using the “--mirror” command, so the files are not explicitly present on the server
git branch
Should show the newly created branch “test_branch” there

If a new release is available from upstream CAF, the changes must first be synced onto the server, prior to pulling those changes on the client.
On the client, first restore the original manifest.xml:
cd <android_root>/.repo/manifests
git diff
See which file was changed. Then, checkout that file to reset the changes:
git checkout caf_AU_LINUX_ANDROID_<rest_of_filename>_013.xml
cd <android_root>
repo --trace sync
cd <android_root>
$ repo --trace init -u ssh://acco...@domain1.domain2.com:29418/android_test1/platform/manifest.git -b release -m caf_AU_LINUX_ANDROID_<rest_of_filename>_016.xml --repo-url=ssh://acco...@domain1.domain2.com:29418/android_test1/repo.git –repo-branch=caf-stable
Note that a new manifest.xml file is now being used
Again, edit the manifest.xml file and make the changes as before:
cd .repo
gedit manifest.xml
Changes:
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
  <remote fetch="ssh://acco...@domain1.domain2.com:29418/android_test1" name="caf" review="ssh://acco...@domain1.domain2.com:29418"/>
repo --trace sync
At the end, it shows:
kernel/msm-3.18/: manifest switched android-external.lnx.2.0.r5-rel...7c22b71efeb3a2213ecc16783739183e9ba325c7
project kernel/msm-3.18/

: cd /home/build/vtest/kernel/msm-3.18
: git rebase --onto 7c22b71efeb3a2213ecc16783739183e9ba325c7 506560057451953966fa42c435fc38fab21f4596^1
First, rewinding head to replay your work on top of it...
Applying: Testing Kconfig change'


This seems correct, since our original commit was preserved and then applied.






And here are the config files to cross-check the setup:

--------------------gerrit.config---------------------------------------------------
[gerrit]
    basePath = /home/server/repo
    serverId = --------removed for security------------------------
    canonicalWebUrl = http://domain1.domain2.com:8443/
[database]
    type = h2
    database = /home/server/gerrit/db/ReviewDB
[auth]
    type = LDAP
[ldap]
    server = ldap://localhost
    accountBase = ou=people,dc=domain1,dc=domain2,dc=com
    groupBase = ou=groups,dc=domain1,dc=domain2,dc=com
[receive]
    enableSignedPush = false
[sendemail]
    smtpServer = localhost
[container]
    user = gerrit2
    javaHome = /usr/lib/jvm/java-8-oracle/jre
        heapLimit = 4096M
[sshd]
    listenAddress = *:29418
[httpd]
    listenUrl = http://*:8443/
[cache]
    directory = cache


These LDAP configuration files are taken from other postings online about people trying to create a similar setup:

------------ldap_add_group_dev.ldif-----------------------------------------
dn: cn=dev,ou=groups,dc=domain1,dc=domain2,dc=com
objectClass: posixGroup
cn: dev
gidNumber: 5000

------------------ldap_add_groups.ldif---------------------------------------
dn: ou=groups,dc=domain1,dc=domain2,dc=com
objectClass: organizationalUnit
ou: groups


-------------------ldap_add_people.ldif----------------------------
dn: ou=people,dc=domain1,dc=domain2,dc=com
objectClass: organizationalUnit
ou: people


------------ldap_add_user.ldif---------------------------------------
dn: uid=account1,ou=people,dc=domain1,dc=domain2,dc=com
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
uid: account1
sn: User
givenName: Gerrit
cn: account1
displayName: Gerrit Review
uidNumber: 10000
gidNumber: 5000
userPassword: <enter the password here>
gecos: Test User
loginShell: /bin/bash
homeDirectory: /home/users/account1
Reply all
Reply to author
Forward
0 new messages