Model version control and migration strategy

Schema Version Control and Migration Strategies

MongoDB, as a document database, offers flexibility with its schemaless nature but also introduces complexity in managing data consistency. As business requirements evolve, changes to data structures are inevitable, making smooth migration without service interruption a critical challenge.

Core Concepts of Schema Version Control

Schema version control essentially introduces traditional database schema management concepts into document databases. The core idea is to add a version identifier field to each document, typically named __schemaVersion or _v. When an application reads data, it determines how to handle format differences based on the version number.

// Typical versioned document structure
{
  _id: ObjectId("5f3d8a9b2c1d4e5f6a7b8c9d"),
  __schemaVersion: 2,
  username: "dev_user",
  contact: {
    email: "user@example.com",
    phone: "+8613800138000"
  }
}

Version control strategies generally fall into three categories:

1. Forward Compatibility: New code can handle old data formats.
2. Backward Compatibility: Old code can handle new data formats.
3. Bidirectional Compatibility: Both old and new code can handle each other's data formats.

Implementation of Migration Strategies

Incremental Migration Approach

Adopt a read-write separation migration method, performing data conversion incrementally during reads:

async function getUser(userId) {
  const user = await db.collection('users').findOne({ _id: userId });
  
  switch(user.__schemaVersion) {
    case 1:
      // Conversion from v1 to v2: Migrate standalone email field to contact subdocument
      user.contact = { email: user.email };
      delete user.email;
      user.__schemaVersion = 2;
      await db.collection('users').updateOne(
        { _id: userId },
        { $set: user }
      );
    case 2:
      // No conversion needed for current version
      return user;
    default:
      throw new Error(`Unsupported schema version: ${user.__schemaVersion}`);
  }
}

Batch Migration Approach

For large-scale data migration, use background tasks:

const batchSize = 1000;

async function migrateUsers() {
  let lastId = null;
  do {
    const query = lastId ? { _id: { $gt: lastId }, __schemaVersion: 1 } : { __schemaVersion: 1 };
    const users = await db.collection('users')
      .find(query)
      .sort({ _id: 1 })
      .limit(batchSize)
      .toArray();

    if (users.length === 0) break;

    const bulkOps = users.map(user => ({
      updateOne: {
        filter: { _id: user._id },
        update: {
          $set: {
            contact: { email: user.email },
            __schemaVersion: 2
          },
          $unset: { email: "" }
        }
      }
    }));

    await db.collection('users').bulkWrite(bulkOps);
    lastId = users[users.length - 1]._id;
  } while (true);
}

Multi-Version Coexistence Strategy

In a microservices architecture, different services may run different versions of code, requiring more complex compatibility solutions:

1. API Version Gateway: Route requests to corresponding service versions via a gateway.
2. Data Transformation Layer: Implement version conversion in the data access layer.
3. Event-Driven Architecture: Use message queues to pass standardized event formats.

// Example of a data transformation layer
class UserDataAdapter {
  constructor(version) {
    this.version = version;
  }

  normalize(user) {
    if (this.version === 'v2') {
      return {
        id: user._id,
        name: user.username,
        email: user.contact?.email || user.email
      };
    }
    // Handle other versions...
  }
}

Change Types and Handling Patterns

Different data structure changes require different migration strategies:

Field Renaming

// New and old fields coexistence approach
{
  $rename: { "oldField": "newField" },
  $set: { __schemaVersion: 2 }
}

Field Type Changes

// Aggregation pipeline for string-to-array conversion
db.users.aggregate([
  { $match: { tags: { $type: "string" } } },
  { $set: { 
    tags: { $split: ["$tags", ","] },
    __schemaVersion: 2 
  }},
  { $merge: "users" }
])

Nested Structure Changes

// Flat structure to nested structure conversion
db.users.updateMany(
  { __schemaVersion: 1 },
  [
    {
      $set: {
        address: {
          city: "$city",
          street: "$street"
        },
        __schemaVersion: 2
      }
    },
    { $unset: ["city", "street"] }
  ]
)

Migration Toolchain Options

MongoDB ecosystem offers various migration tools with different focuses:

1. Native Scripts: mongo shell + JavaScript.
2. Professional Tools: MongoDB Connector for BI.
3. ETL Tools: Apache Spark + MongoDB Connector.
4. ORM Integration: Mongoose plugin system.

// Example of a Mongoose migration plugin
const userSchema = new mongoose.Schema({
  username: String,
  email: String
});

userSchema.plugin(function(schema) {
  schema.post('init', function(doc) {
    if (doc.__schemaVersion === 1) {
      doc.contact = { email: doc.email };
      doc.__schemaVersion = 2;
      doc.save();
    }
  });
});

Monitoring and Rollback Mechanisms

A robust migration system requires monitoring metrics:

// Migration status monitoring query
db.users.aggregate([
  {
    $group: {
      _id: "$__schemaVersion",
      count: { $sum: 1 },
      minCreated: { $min: "$_id" },
      maxCreated: { $max: "$_id" }
    }
  }
])

Key points in rollback strategy design:

1. Preserve original data backups.
2. Log migration operations.
3. Implement reverse migration scripts.
4. Establish version toggle switches.

// Example of a rollback script
db.users.updateMany(
  { __schemaVersion: 2 },
  [
    {
      $set: {
        email: "$contact.email",
        __schemaVersion: 1
      }
    },
    { $unset: "contact" }
  ]
)

Performance Optimization Practices

Performance optimization techniques for large-scale migrations:

1. Batch Processing: Use bulkWrite instead of single updates.
2. Read-Write Separation: Read from secondary nodes.
3. Indexing Strategy: Create partial indexes for __schemaVersion.
4. Parallel Control: Optimize shard keys.

// Parallel migration script
const shardKeys = await db.command({ listShards: 1 });
await Promise.all(shardKeys.shards.map(async shard => {
  const shardConn = new Mongo(shard.host);
  await shardConn.db('app').collection('users').updateMany(
    { __schemaVersion: 1 },
    { $set: { __schemaVersion: 2 } }
  );
}));

Team Collaboration Standards

Best practices for team collaboration:

1. Change Log: Maintain a history of data structure changes.
2. Code Review: Require dual confirmation for schema changes.
3. Environment Policy: Allow automatic migration in development, manual triggering in production.
4. Documentation Sync: Update OpenAPI/Swagger documentation with version changes.

# Change Log

## 2023-11-20 - User Model v2
- Change Type: Field reorganization.
- Impact Scope: User service, order service.
- Migration Script: /scripts/migrations/user-v2.js.
- Rollback Plan: /scripts/rollbacks/user-v2.js.