Gerrit Large-Repository Acceleration and Index Maintenance Guide

Purpose: Provide a practical, production-ready maintenance guide for Git/Gerrit repositories. The goal is to significantly reduce git clone / git fetch latency—particularly the “Finding sources” stage—by maintaining compact pack files, generating bitmap indexes and commit-graph files, and performing Gerrit/Lucene index maintenance.

Preconditions (Environment Summary)

Symptom (Observed Behavior)

A typical client-side command:

git clone "ssh://[email protected]:29418/Chery/T29-T16A/.../co_common" -b baseline-mp1

Observed output:

Cloning into 'co_common'...
remote: Counting objects: 9907586, done
remote: Finding sources:   4% (361750/9907586)

Observation: The Finding sources step is extremely slow (≈1,500 objects/sec), causing a long overall clone time.

Executive Summary (Root Cause and Remediation)

• Root cause: The repository lacks up-to-date bitmap indexes and commit-graph files. Without these, the server must traverse the commit DAG to compute the set of objects to send, which is very expensive for very large repositories.
• Secondary cause: Many small or fragmented pack files (i.e., lack of periodic repack) increase server CPU and I/O load when forming or reusing packs.
• Primary remediation: Periodically run git repack -adf --write-bitmap-index and git commit-graph write --reachable --changed-paths on bare repositories, and rebuild Gerrit Lucene indexes when needed.

Cause Analysis (Detailed)

1. Counting objects

• This phase enumerates object counts using existing pack indexes (.idx) and generally completes quickly. It is affected by disk I/O and index-read performance.

2. Finding sources (the common bottleneck)

• Purpose: Determine the minimal set of objects the server must send to the client by computing differences between server refs and the client’s have set.
• With bitmap indexes and commit-graph, the server can compute the object set using bitset operations and reachability metadata. Without these structures it performs a full DAG traversal which is slow for repositories with millions of objects.

3. Compressing objects / sending pack

• After determining which objects to transfer, the server either reuses existing pack files or computes a new pack (git pack-objects). If pack contents are fragmented, real-time packing consumes CPU and I/O resources, further slowing git clone.

Practical Maintenance Steps (Commands)

Run these in the bare repository root on the Gerrit server. Perform backups and run during a maintenance window.

Single-repo maintenance (recommended initial run)

# Change to the bare repository directory
cd /data/sdc/cockpit-gerrit-8082/git/Chery/T29-T16A/MediaTek_MTK8678/yocto/BITECH/mt8678-yocto/src/kernel/linux/v6.6_mt8678/co_common.git

# Optional: backup current pack files (short-term)
cp -r objects/pack ../pack_backup_$(date +%F)

# Repack: create compact pack(s) and write bitmap index
git repack -adf --write-bitmap-index --threads=16

# Write commit-graph including changed-paths to accelerate path-restricted queries
git commit-graph write --reachable --changed-paths

# Verify files exist
ls -lh objects/pack/*.bitmap || true
ls -lh objects/info/commit-graph* || true

Notes on flags:

• -a: include all loose objects
• -d: remove older packs after successful repack (keep backup first if unsure)
• -f: force
• --write-bitmap-index: produce .bitmap files

Gerrit Lucene index maintenance

Gerrit’s Lucene indexes accelerate UI and server-side query performance (but do not directly change the Git protocol behavior). Rebuild indexes via the Gerrit administrative command:

# Rebuild index for all projects (resource-intensive)
ssh -p 29418 gerrit@localhost gerrit index start --all

# Or rebuild a single project
ssh -p 29418 gerrit@localhost gerrit index start --project "Chery/T29-T16A/.../co_common"

Caveat: Re-indexing is resource hungry and should run during low-traffic windows.

Batch / Automated maintenance script (example)

#!/bin/bash
set -euo pipefail
GERRIT_SITE="/data/sdc/cockpit-gerrit-8082/git"
LOGFILE="/var/log/gerrit_maintenance_$(date +%F).log"
THREADS=12

cd "$GERRIT_SITE"

for repo in $(find . -name "*.git" -type d); do
  echo "$(date +%F_%T) - Optimizing $repo" | tee -a "$LOGFILE"
  cd "$repo" || continue

  # Optional: keep short-term backup
  # cp -r objects/pack ../pack_backup_$(date +%F)

  git repack -adf --write-bitmap-index --threads=${THREADS}
  git commit-graph write --reachable --changed-paths || true

  cd "$GERRIT_SITE"
  # Optional: sleep to avoid saturating I/O
  # sleep 2
done

# Optional: trigger Gerrit indexing (use with caution)
# ssh -p 29418 gerrit@localhost gerrit index start --all

Recommendation: Schedule via systemd timer or cron during maintenance windows. Include logging, error handling, serialized execution (avoid repacking all repos concurrently), and alerting.

Client and Network Optimizations

• Configure client-side parallel unpacking in ~/.gitconfig:

[pack]
    threads = 8

• Limit transferred data when full history is unnecessary:

# Clone a single branch
git clone --single-branch -b baseline-mp1 <repo-url>

# Shallow clone (only latest commits)
git clone --depth 1 --branch baseline-mp1 <repo-url>

• Monitor and diagnose remaining slow cases by checking:

  • Server CPU, iowait, disk queue length (top, iostat -x 1, iotop)
  • Network packet loss, MTU issues
  • Gerrit logs for repack or indexing activity

Concepts at a Glance

Pack / Pack Index (.pack / .idx)

• Packs combine and delta-compress objects to reduce disk and network footprint. An .idx maps object IDs to offsets inside a pack.
• Operational guideline: fewer, larger, and more compact packs tend to be more efficient.

Bitmap Index

• Bitmap indexes store reachable-object sets as bitsets. Bitset operations enable high-performance set arithmetic when computing the objects to send to a client.
• Generating bitmaps (--write-bitmap-index) transforms Finding sources from an expensive graph traversal to fast bitset arithmetic.

Commit-Graph

• The commit-graph stores precomputed parent relationships and optional metadata such as generation numbers and changed-paths information.
• Commit-graph files accelerate reachability checks and path-restricted history queries.

Compressing Objects / Sending Pack

• After computing the object set, the server reuses existing packs or creates a pack to send. Real-time pack creation can be CPU/I/O heavy if packs are fragmented.

Verification & Troubleshooting Checklist

1. Index verification

   • ls objects/pack/*.bitmap → Confirm .bitmap files exist.
   • ls objects/info/commit-graph* → Confirm commit-graph files exist.

2. Re-test clone speed

   • Perform git clone --single-branch -b baseline-mp1 <url> from another machine before/after maintenance and compare durations.

3. Monitor resource usage during maintenance

   • top, htop — CPU and thread usage
   • iostat -x 1, iotop — disk latency and throughput
   • Gerrit logs — errors related to repack or commit-graph

4. Rollback plan

   • If repack causes issues, restore the previous objects/pack from pack_backup_YYYY-MM-DD and restart Gerrit (coordinate with a maintenance window).

Recommended Maintenance Frequency

• Highly active repositories (frequent pushes):

  • commit-graph: update daily or after large push events
  • repack + bitmap: weekly or at regular release/maintenance windows

• Moderately active repositories:

  • commit-graph: weekly
  • repack + bitmap: monthly

• Cold repositories:

  • Quarterly or as needed

Risks and Operational Considerations

• git repack -adf and git commit-graph write are CPU/memory/disk I/O intensive. Always run during low-traffic windows.
• Keep short-term backups of the objects/pack folder before deleting old packs (-d) until you have validated the new packs.
• Coordinate with teams to avoid concurrent pushes during repack operations.
• Ensure Gerrit / JVM memory sizing is adequate when running Lucene re-indexing.

Quick Commands Reference

# Repack and write bitmap
git repack -adf --write-bitmap-index --threads=16

# Write commit-graph with changed-paths
git commit-graph write --reachable --changed-paths

# Verify bitmap
ls objects/pack/*.bitmap

# Trigger Gerrit index rebuild (admin)
ssh -p 29418 gerrit@localhost gerrit index start --all

Next Steps (Recommended Implementation Plan)

1. Run a one-time maintenance on the slowest repository and measure clone time improvement.
2. Implement the automated script with serialized repo processing and logging; deploy as a systemd timer or cron job during maintenance windows.
3. Add monitoring and alerting for failures and resource saturation during maintenance.
4. Optionally provide an Ansible playbook or systemd unit + timer to orchestrate maintenance with safe rollback.