Binary file processing

Binary File Handling

Git treats binary files differently from text files. Binary files cannot be compared line-by-line for differences, resulting in lower storage efficiency. Common binary files include images, videos, compressed archives, executable programs, etc.

Binary File Issues in Git

Git's core design is optimized for text files, using delta compression algorithms to store file differences. Binary files generate entirely new copies with each modification, causing rapid repository size growth. Frequently modified large binary files significantly impact repository cloning and fetching speeds.

// Example: Reading binary files in Node.js
const fs = require('fs');
fs.readFile('example.zip', (err, data) => {
  if (err) throw err;
  console.log(`Read ${data.length} bytes of binary data`);
});

Git LFS Solution

Git Large File Storage (LFS) is the recommended solution for handling large binary files. It replaces actual binary content with pointer files and stores large files on a separate server. How it works:

1. Install Git LFS client
2. Track specified file types
3. Automatically replace with pointer files during push

# Initialize Git LFS
git lfs install

# Track all .psd files
git lfs track "*.psd"

Binary File Diff Handling

Conventional diff tools are ineffective for binary files. Certain file types can use specialized tools:

• Images: Use git difftool with image comparison software
• PDF: pdf-diff tool to generate visual differences
• Office documents: Convert to text using pandoc for comparison

// Example: Binary image comparison (Node.js)
const crypto = require('crypto');
function compareImages(file1, file2) {
  const hash1 = crypto.createHash('sha256').update(fs.readFileSync(file1)).digest('hex');
  const hash2 = crypto.createHash('sha256').update(fs.readFileSync(file2)).digest('hex');
  return hash1 === hash2;
}

Binary File Merge Conflicts

Binary file conflicts cannot be resolved automatically. Typical resolution process:

1. Confirm conflicting file versions
2. Merge manually using external tools
3. Add the resolved file
4. Mark conflict as resolved

# Conflict resolution example
git checkout --ours image.jpg  # Use current branch version
git checkout --theirs image.jpg  # Use merged branch version
# Or after manual editing
git add image.jpg

Performance Optimization Strategies

Optimization methods when handling large binary files:

• Use shallow clones to reduce initial download size
• Partial clones excluding specific binary directories
• Regular garbage collection to compress objects

# Shallow clone example
git clone --depth 1 https://repo.example.com/project.git

# Partial clone example
git clone --filter=blob:none --sparse https://repo.example.com/project.git
cd project
git sparse-checkout set --no-cone "!/assets/binary"

Version Control Best Practices

Git usage recommendations for binary files:

1. Place frequently modified binary files in separate directories
2. Set up separate repositories for static resources
3. Use .gitattributes to specify file handling methods
4. Regularly clean large files from history

# .gitattributes example
*.mp4 filter=lfs diff=lfs merge=lfs -text
*.png filter=lfs diff=lfs merge=lfs -text

Binary File History Cleanup

Steps to completely remove large files from Git history:

1. Use BFG tool or git filter-branch
2. Rewrite history for all affected branches
3. Force push to remote repository
4. Coordinate team members to re-clone

# BFG cleanup example
java -jar bfg.jar --strip-blobs-bigger-than 10M repo.git
git reflog expire --expire=now --all
git gc --prune=now --aggressive

Automated Processing Solutions

CI/CD patterns for handling binary files:

1. Download LFS files during build
2. Automatically compress resources in release pipeline
3. Generate checksums during version release

// Example: LFS handling in GitHub Actions
name: Build
on: push
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          lfs: true
      - run: du -h  # Check file sizes

Cross-Platform Compatibility Issues

Differences in binary file handling between Windows and Unix systems:

• Automatic line ending conversion issues
• File permission preservation differences
• Filename case sensitivity

# Disable automatic line ending conversion
git config --global core.autocrlf false

Binary File Metadata Management

Additional considerations beyond file content:

• EXIF information (images)
• Creation/modification timestamps
• Filesystem permissions

// Example: Reading image metadata (Node.js)
const ExifReader = require('exifreader');
const tags = ExifReader.load(fs.readFileSync('photo.jpg'));
console.log(tags.DateTimeOriginal.description);