Package files and compression

Packfiles and Compression

Packfiles and compression mechanisms in Git are the core of efficient repository storage. By combining similar objects and applying delta compression algorithms, they significantly reduce disk usage while maintaining data integrity. Understanding these underlying technologies helps optimize the performance of large repositories.

Basic Principles of Packfiles

Git packs loose objects into binary packfiles to save space. The packing process is triggered when the number of loose objects exceeds a certain threshold (default: 500,000) or when git gc is manually executed. Packfiles contain:

1. Header Information: 4-byte signature "PACK" + version number + object count
2. Object Entries: Compressed object data
3. Checksum: SHA-1 checksum of the entire packfile

Command example to view packfile contents:

git verify-pack -v .git/objects/pack/pack-*.idx

Delta Compression

Git uses delta compression algorithms to store similar objects. The implementation consists of:

1. Delta Base Object: Fully stored object
2. Delta Derived Object: Stores only the differences from the base object

Common delta compression strategy:

Original version: fileA (100KB)
Modified version: fileA' (only 5KB changes)
Storage method:
  - Store fileA in full
  - Store fileA' as "based on fileA, modify 5KB data at offset X"

Packfile Indexes

Each .pack file has a corresponding .idx index file with the following structure:

1. Object counts for 256 sectors
2. List of objects sorted by SHA-1
3. CRC32 checksum for each object
4. Offset within the packfile

View index using low-level command:

git show-index < .git/objects/pack/pack-*.idx

Handling Multiple Packfiles

Large repositories may contain multiple packfiles. Git employs the following strategies:

1. Incremental Packs: New packs generated by git repack contain only new objects
2. Geometric Repacking: Merge small packs into larger ones, maintaining geometric growth in pack sizes

Example manual optimization:

git repack -ad --geometric=2

Compression Level Control

Git provides multiple compression configuration parameters:

# .gitconfig example
[pack]
  window = 15       # Context lines for diff comparison
  depth = 50        # Maximum delta compression depth
  threads = 8       # Multithreaded compression
  compression = 9   # zlib compression level (0-9)

Recommended configurations for different scenarios:

• Development environment: compression=6 (balance speed and size)
• Archive repository: compression=9 (maximum compression)

Binary Delta Algorithm

Git uses an improved xdelta algorithm for binary diffs, with key features including:

1. Rolling Hash: Quickly locate similar blocks
2. Delta Instruction Encoding:
   • COPY instruction: Reference source data block
   • ADD instruction: Insert new data

Example delta instruction sequence:

COPY 0-1000
ADD 20 "new content"
COPY 1000-1500

Object Reuse Strategies

Git reuses existing pack objects in the following cases:

1. Push/Fetch: Transfer only missing packfiles
2. Shallow Clone: Record truncated history via shallow file
3. Partial Clone: Fetch objects on demand using filter parameters

Filtered clone example:

git clone --filter=blob:none <repo-url>

Packfile Maintenance Operations

Common maintenance commands and their functions:

Example to force-optimize all objects:

git repack -a -d --window=250 --depth=50

Debugging Packfile Issues

Diagnostic methods for packfile-related issues:

1. Check packfile integrity:

   git fsck --full
   

2. View object storage locations:

   git cat-file --batch-check='%(objectname) %(objecttype) %(rest)' --batch-all-objects
   

3. Measure packfile statistics:

   git count-objects -v
   

Custom Packing Strategies

Example of automated packing strategy via Git hooks:

#!/bin/sh
# .git/hooks/post-commit

# Trigger lightweight packing when object count exceeds threshold
OBJECTS=$(git count-objects | awk '{print $1}')
if [ "$OBJECTS" -gt 1000 ]; then
    git repack -a -d -l --window=10  # Quick repack
fi

Packfiles and Network Transfer

How Git protocols optimize transfers using packfiles:

1. Negotiation Phase: Client and server exchange object lists
2. Packfile Generation: Server dynamically generates packs containing missing objects
3. Thin Packs: Omit some base objects, to be completed by the client

Underlying process for fetching new objects:

git fetch origin main --no-tags -v
# Sample output:
# remote: Counting objects: 75, done.
# remote: Compressing objects: 100% (53/53)
# Receiving objects: 100% (75/75), 15.25 KiB | 1.52 MiB/s