If your development is based on AOSP source code to build a warehouse, then building a repo server and deploying your own repo warehouse is a very necessary task.
In reality, many companies directly manage AOSP source code in a git repository, and there is nothing wrong with doing this. However, if the development team has a large number, there will be some obvious disadvantages:
- The general size of AOSP source code is 20GB, and it reaches more than 40GB after compilation. Each synchronization will be time-consuming.
- The number of AOSP source code files is nearly 70W, and the file index of a single repository will be huge, which is very time-consuming for each retrieval.
- The original code repository of AOSP and chip platform manufacturers (such as qcom, mtk, rockchip, etc.) is generally provided in the repo repository structure, rather than a single git repository. After changing the warehouse structure, manual maintenance is relatively large.
The AOSP source code structure is complex and huge. In addition to its own code module, it also integrates five or six hundred third-party repositories, which will be updated and iterated continuously. AOSP needs to maintain sufficient flexibility in maintaining the entire code repository framework, so it introduced repo scripts to batch manage git repository. The number of git repositories divided by AOSP's source code is as high as six or seven hundred.
If we want to maintain a consistent warehouse structure with the upstream platform warehouse, it is very convenient and fast to synchronize the code, and we can flexibly manage the local extended warehouse, we need to build and deploy our own repo warehouse.
In actual team development practice, the code management framework is generally a combination of gerrit+repo. Gerrit's installation, deployment and permission management are not the content of this document. Please search for network materials to install and deploy. The main contents of this document include:
- Explain the management strategy of the repo warehouse.
- How to create a list warehouse.
- How to batch split git repository.
- The operation and use of repo repository.
The server operating system of this document is ubuntu18.04.
1 How does repo manage repositories?
1.1 How does repo work?
The new operating system installation package already contains the repo command program, which can be installed directly through the following commands:
sudo apt-get install repo
# View the repo versionrepo --version
<repo not installed>
repo launcher version 2.17
(from /usr/local/bin/repo)
git 2.34.1
Python 3.10.4 (main, Apr 2 2022, 09:04:19) [GCC 11.2.0]
OS Linux 5.15.0-39-generic (#42-Ubuntu SMP Thu Jun 9 23:42:32 UTC 2022)
CPU x86_64 (x86_64)
Bug reports: /p/gerrit/issues/entry?template=Repo+tool+issue
After the installation is completed, we can see that the installation is actually a repo booter. It is not a complete management tool yet.
Check the repo file through the cat command, which is actually a python3 script.
which repo /usr/local/bin/repo cat /usr/local/bin/repo ........
Several command sequences generally used by clients when syncing repositories:
# Initialize the warehouserepo init -u .... # Check out the coderepo sync -j16 # Create a local branchrepo start <BRANCH> --all # Execute git commands in batchesrepo forall -c `GIT COMMAND`
Let’s take the AOSP code on the official website of Tsinghua University as an example to operate the repo’s warehouse synchronization.
# Set REPO_URL.export REPO_URL='/git/git-repo/'
mkdir -p ~/aosp
cd aosp
# Initialize the warehouserepo init -u /git/AOSP/platform/manifest
Downloading Repo source from /git/git-repo/
remote: Enumerating objects: 7372, done.
remote: Counting objects: 100% (7372/7372), done.
remote: Compressing objects: 100% (3935/3935), done.
remote: Total 7372 (delta 4765), reused 5577 (delta 3363)
Received object: 100% (7372/7372), 3.27 MiB | 3.58 MiB/s, Finish.
deal with delta middle: 100% (4765/4765), Finish.
... A new version of repo (2.21) is available.
... You should upgrade soon:
cp /home/xxx/Documents/aosp/.repo/repo/repo /usr/local/bin/repo
Downloading manifest from /git/AOSP/platform/manifest
remote: Enumerating objects: 98793, done.
remote: Counting objects: 100% (98793/98793), done.
remote: Compressing objects: 100% (39423/39423), done.
remote: Total 98793 (delta 38918), reused 98123 (delta 38532)
Received object: 100% (98793/98793), 25.23 MiB | 2.40 MiB/s, Finish.
deal with delta middle: 100% (38918/38918), Finish.
Your identity is: xxx <xxx@>
If you want to change this, please re-run 'repo init' with --config-name
Testing colorized output (for 'repo diff', 'repo status'):
black red green yellow blue magenta cyan white
bold dim ul reverse
Enable color display in this user account (y/N)? y
repo has been initialized in /home/xxx/Documents/aosp/
If this is not the directory in which you want to initialize repo, please run:
rm -r /home/yl/Documents/aosp//.repo
and try again.
After the repo init command is executed, it is actually downloaded in a .repo directory (the repo command version is outdated, you can update it according to the prompts, or you don’t need to update it).
Let's take a look at the contents of the repo directory:
repo
├── manifests // List warehouse│ ├── .git -> ..//
│ ├── // Real warehouse manifest file│ ├──
│ └── OWNERS
├── // Checklist repository git configuration information│ ├── branches
│ ├── config
│ ├── description
│ ├── FETCH_HEAD
│ ├── HEAD
│ ├── hooks
│ ├── index
│ ├── info
│ ├── logs
│ ├── objects
│ ├── packed-refs
│ └── refs
├── // List file, content contains files└── repo // Repo tool script ├──
├──
├── git_trace2_event_log.py
├── hooks
├──
├── repo
├── .........
└── Repo init did these things:
- Download the repo tool repository, which contains python scripts. In the .repo/repo directory.
- Download the manifests list repository. Under the directory .repo/manifests, the soft link contains the .repo/directory.
- Downloaded a repository list file containing it.
The content of the repo batch management warehouse is in. Let's take a look at the file content:
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
<remote name="aosp"
fetch=".."
review="/" />
<default revision="master"
remote="aosp"
sync-j="4" />
<manifest-server url="//manifestserver" />
<superproject name="platform/superproject" remote="aosp"/>
<contactinfo bugurl="go/repo-bug" />
<project path="build/make" name="platform/build" groups="pdk" >
<linkfile src="" dest="build/" />
<linkfile src="" dest="build/" />
<linkfile src="core" dest="build/core" />
<linkfile src="" dest="build/" />
<linkfile src="target" dest="build/target" />
<linkfile src="tools" dest="build/tools" />
</project>
<project path="build/bazel" name="platform/build/bazel" groups="pdk" >
<linkfile src="" dest="WORKSPACE" />
<linkfile src="" dest="tools/bazel" />
<linkfile src="" dest="BUILD" />
</project>
<project path="build/bazel_common_rules" name="platform/build/bazel_common_rules" groups="pdk" />
<project path="build/blueprint" name="platform/build/blueprint" groups="pdk,tradefed" />
<project path="build/pesto" name="platform/build/pesto" groups="pdk" />
<project path="build/soong" name="platform/build/soong" groups="pdk,tradefed" >
<linkfile src="" dest="" />
<linkfile src="" dest="" />
</project>
<project path="art" name="platform/art" groups="pdk" />
<project path="bionic" name="platform/bionic" groups="pdk" />
<project path="bootable/recovery" name="platform/bootable/recovery" groups="pdk" />
<project path="bootable/libbootloader" name="platform/bootable/libbootloader" groups="vts,pdk" />
<project path="device/common" name="device/common" groups="pdk-cw-fs,pdk" />
<project path="device/generic/arm64" name="device/generic/arm64" groups="pdk" />
<project path="device/generic/armv7-a-neon" name="device/generic/armv7-a-neon" groups="pdk" />
<project path="device/generic/art" name="device/generic/art" groups="pdk" />
......
<repo-hooks in-project="platform/tools/repohooks" enabled-list="pre-upload" />
<!--
Merge marker to make it easier for git to merge AOSP-only manifest
changes to a custom manifest. Please keep this marker here and
refrain from making changes on or below it.
-->
</manifest> fold
The syntax of this XML file is not the content of this document. A few key points are briefly explained:
remote: name of the remote server repo repo repository
project: Git repository for each project
path: client git repository path
name: server git repository name
revision: branch name or commit id
linkfile: Soft link file, src and dest represent source and destination.
copyfile: file copy, src and dest represent source and destination.
groups: Group.
A summary of the way to manage repo batches:
- The repo init command downloads the tool script repository and inventory repository.
- The repo sync command downloads the remote repository binary file according to the configuration in the manifest file.
- repo start checks out the branch.
Therefore, the core of the management of the repo warehouse is, and the configuration file that plays a key role is the warehouse list file. The corresponding XML list file is available for as many project warehouses.
1.2 What do you need to do to build a repo service?
Next is the key question: What do we need to do to build a repo server?
in short:
- Deploy a common tool repository.
- Deploy your own inventory repository.
- Write a manifest file
- Bulk creation of project sub-repositories and upload source code.
2 Deploy the tool repository
In fact, if your network link can be linked to the external network, the tool repository does not need to be built and deployed.
You only need to specify the URL through the REPO_URL environment variable to automatically pull the repository.
export REPO_URL=/git/git-repo/
Common URLs are:
## google address, need a ladder./git-repo ## Tsinghua University Open Source Software Mirror Station/git/git-repo/ ## You can search for others online by yourself...
But in reality, many companies' external networks are restricted and inaccessible. At this time, you need to consider building your own tool warehouse.
The construction of this warehouse is very simple, it is just an ordinary git warehouse, but the issues involving GPG signatures are relatively complicated. This process is not the content of this document, you can search for documentation on the Internet.
3 Deploy inventory repository
The way to deploy a list repository is no different from that of an ordinary git repository. The key issues are:
- How to manage multiple project warehouses in list warehouses?
- How to write a manifest file?
3.1 How to design a list warehouse and branch?
3.1.1 How many warehouses are needed?
From a server perspective, your server will definitely manage multiple project repositories (such as qcom8953, qcom8916, mtk6219, mtk2226, rk3399, rk3358, etc.). Then the way we manage these repositories may be:
- The entire server only has one warehouse, and the different xml files under each warehouse manage different project warehouses. For example, etc.
- Each project warehouse establishes a corresponding list warehouse, such as, etc., where the list file xml of each project warehouse is placed.
Both methods are OK, depending on your code management strategy. I usually use one repository on the entire server and manage all project repositories through different xml files. The directory structure design of the server-side project warehouse is as follows:
aosp
├── // List warehouse
│ ├──
│ ├──
│ ├──
│ └──
├── qcom/qcom8953 // Project warehouse
├── rockchip/rk3399 // Project warehouse
├── rockchip/rk3358 // Project warehouse
└── mtk/mtk6226 // Project warehouse
3.1.2 How many branches are needed?
The list repository itself is an ordinary git repository, and it also has branches. For example, it can also split different branches to manage different project repository versions.
The branch of the list warehouse has no direct relationship with the branch of the project warehouse. However, in practical operations, we generally establish branches corresponding to the project warehouse in the inventory warehouse to facilitate management of mappings. For example, the qcom8953 project repository has 4 branches, master, develop, test and release. Then ours can also establish four branches to correspond to the project repository.
I'm creating two branches here: master and release. The following will be followed by these two branches.
Let's first look at the common parameters of the repo init command:
repo init --help
Usage: repo init [options] [-u] url
Options:
Manifest options:
-u URL, --manifest-url=URL
manifest repository location
-b REVISION, --manifest-branch=REVISION
manifest branch or revision (use HEAD for default)
-m , --manifest-name=
initial manifest file
repo Version options:
--repo-url=URL repo repository location ($REPO_URL)
--repo-rev=REV repo branch or revision ($REPO_REV)
--no-repo-verify do not verify repo source code
Other options:
--config-name Always prompt for name/e-mailGenerally speaking, the default value of -b is master, and the default value of -m is. If we want to use different default manifest files, we need to use it together when writing the server manifest file and repo init parameters.
I use the default values here. In the aosp source code, it is reflected in these items:
<remote name="aosp"
fetch=".."
review="/" />
<default revision="master" ## Remote default branch remote="aosp" ## The remote default repository is the name="aosp" in the remote tag sync-j="4" />3.2 How to write a manifest file?
When writing a list file, it is definitely impossible to write all of it by hand, and you need to get an original file. This file requires you to get from your source code repo repository is the best.
Here I will use the AOSP source code of qcom as an example to modify my own repository.
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
<！--
remote: Information about the server remote repository。
name： remotemanifestWarehouse name，Generally default isorigin
fetch： storehouseurl。If set to"."(Note that it is a point), That meansrepo init -u urlMedium configuration。
review: Review system website。If the code is submitted，Will enter this review system。
-->
<remote name="origin" fetch="." review="gerrit.your_domain.com" />
<!--
default： thisrepostorehouse的默认属性。
revision： 默认的工程代码storehouse分支，You can modify this to point to different branches。
remote： remotestorehouse的名称。Here<remote>The tag is the corresponding，当然还可以指向其他storehouse的remote。
-->
<default revision="master" remote="origin"/>
<!--
project: 每个storehouse就是一个project
path: The path name after the client is synchronized to the local area。
name: Server sidegitWarehouse name,Pay attention to the path。
-->
<project path="build" name="build" />
<project path="device" name="device" />
<project path="frameworks/base" name="frameworks/base" />
<!--
Important Note:
HerebuildDirectory and three subdirectories below：blueprint, kati, soongis nestedgitstorehouse结构！
This nestedgitstorehouse。There is a very important control file:.gitignore.
There must be one parent directory.gitignoredocument，其中包含了嵌套的子级storehouse路径。
-->
<project path="build" name="build">
<!-- copyfile: Willbuild/core/Copy asMakefile。There is a relative path -->
<copyfile src="core/" dst="Makefile" />
</project>
<project path="build/blueprint" name="build/blueprint" />
<project path="build/kati" name="build/kati" />
<project path="build/soong" name="build/soong">
<!-- linkfile: Create a soft link tobuild/soong/ -->
<linkfile src="" dest="" />
<linkfile src="" dest="" />
</project>
<!-- 更多storehouse信息 ...... -->
</manifest>
3.3 Create a list repository
There are two ways to create a list repository:
- The gerrit system web page creation. This method is suitable for creating a small number of repositories, which is simple and fast.
- The gerrit command is created. This method is suitable for batch creation of large amounts of repositories, especially AOSP repositories.
The first method is relatively simple, just operate on the WEB page, nothing special.
The second command line method is highlighted.
ssh -p 29418 <USERNAME>@<GERRIT_IP_ADDRESS> gerrit create-project aosp/
After the creation is completed, you can also see this repository on the gerrit page. Administrators can create two branches on the page: master, release.
The client operates according to normal git operations:
## Clone the codegit clone .../ ## Submit manifest filegit add git commit -m "add " git push
In this way, we set up a repository and upload a manifest file.
The next step is how to create a project repository.
4 Bulk creation of project sub-warehouses
The source code of the initial engineering warehouse is best from the complete repo warehouse provided by the platform manufacturer with .repo and .git, and then we can see how to port this warehouse to our own internal warehouse.
General operating steps:
- The gerrit command creates a server-side empty repository in batches.
- Preprocess the local repo repository.
- Upload source code in batches.
4.1 Bulk creation of empty server warehouses
Bulk creation of sub-repositories can only be done through the gerrit command line.
ssh -p 29418 <USERNAME>@<GERRIT_IP_ADDRESS> gerrit create-project aosp/qcom8953/
Here we need to know all the repository names of the qcom8953 repository, and then batch creation through scripts. There are two ways:
- Extract the repository name from the xml script.
- Get it directly from the file in .repo/ of the original source repository.
Either way, it is necessary to manually check that the number and name of the final warehouse are consistent.
We finally need a repository list file:. The file contents are as follows:
qcom/qcom8953/art qcom/qcom8953/bionic qcom/qcom8953/bootable/bootloader/edk2 qcom/qcom8953/bootable/recovery qcom/qcom8953/build/blueprint qcom/qcom8953/build/kati qcom/qcom8953/build qcom/qcom8953/build/soong qcom/qcom8953/cts qcom/qcom8953/dalvik qcom/qcom8953/developers/build qcom/qcom8953/developers/demos/JustForUs qcom/qcom8953/developers/samples/android qcom/qcom8953/development qcom/qcom8953/device/common qcom/qcom8953/device/generic/arm64 qcom/qcom8953/device/generic/armv7-a-neon ......
Next, we can use the following command to create a server-side empty repository in batches:
# Parameter description# --empty-commit: Forced parameter, must be added;# -p YOUR-projects: Your gerrit permission repository. Determine based on your gerrit permission name.# Don’t add other parameters again. After adding more problems, there will be many more problems.# Note that the remote repository name needs to be prefixed with path (qcom/qcom8953) to prevent confusion in the server-side repository path.cat | xargs -I {} ssh -p 29418 [email protected] gerrit create-project "{}" --empty-commit -p YOUR-projectsAfter the creation is completed, you will see hundreds of git repositories on the gerrit page.
4.2 Preprocessing local repo code
Before uploading the local repo repository, several preprocessing actions need to be done:
- Delete the .repo directory in the local project.
- Check the local repository.repo// manifest file and delete the source of copyfile and linkfile in the local code according to the file content.
- Modify the git remote url path of each repository locally. Here you need to write your own script to complete it in batches.
git remote set-url origin ssh://[email protected]:29418/xxx
4.3 Upload branches and codes
After the warehouse preprocessing is completed, you need to upload the specified branches and codes of each warehouse to the server warehouse one by one.
Here is a general operation, you also need to write your own script for uploading and synchronization:
git checkout master git push origin master:master git push -- tags
Of course, after uploading, you'd better verify in a third-party test whether there are some problems with gitignore or some nested repositories and need to be repaired manually.
5 Repo warehouse usage brief description
After the repo repository is built, the use operation is not troublesome for the client. You can complete the code submission using commonly used repo commands and git commands.
# Initialize the warehouserepo init -u <remote manifests url> -n <manifests branch> -m <manifest > # Check out the coderepo sync -j16 # Create a local branchrepo start <BRANCH> --all # Execute git commands in batchesrepo forall -c `GIT COMMAND`
Code commit merge for a single repository:
git add xxx git commit -m "xxxx" git push HEAD:refs/for/BRANCH
A small error with no change id may be reported when submitting the first time. Search on Baidu and it is easy to fix.
This is the article about teaching you how to build an Android source code repo repo repo repository. For more related Android source code repo repo repository content, please search for my previous articles or continue browsing the following related articles. I hope everyone will support me in the future!