Character encoding processing translates this sentence into English.

Character encoding is a fundamental technology in computer systems used to represent and store text. In Node.js, handling character encoding involves various scenarios, including file reading/writing, network transmission, data conversion, etc. Proper understanding and use of encoding are crucial for development.

Basic Concepts of Character Encoding

ASCII is the earliest character encoding standard, using 7 bits to represent 128 characters. With the evolution of computers, more encoding schemes emerged:

• UTF-8: Variable-length encoding, compatible with ASCII, uses 1-4 bytes to represent characters.
• UTF-16: Uses 2 or 4 bytes to represent characters.
• GBK: Chinese encoding standard, compatible with GB2312.
• ISO-8859-1: Encoding for Western European languages.

In Node.js, the Buffer object is the core class for handling binary data and is key to character encoding conversion. When creating a Buffer, UTF-8 encoding is used by default:

const buf = Buffer.from('Hello World', 'utf8');
console.log(buf); // <Buffer 48 65 6c 6c 6f 20 57 6f 72 6c 64>

Encoding Conversion in Node.js

Node.js provides multiple ways to perform encoding conversion, with the most common being the toString method of the Buffer class:

const buf = Buffer.from([0xe4, 0xbd, 0xa0, 0xe5, 0xa5, 0xbd]);
console.log(buf.toString('utf8')); // Output: "你好"

When handling files with different encodings, the encoding must be explicitly specified:

const fs = require('fs');

// Read a file encoded in GBK
fs.readFile('gbk-file.txt', (err, data) => {
  if (err) throw err;
  const text = data.toString('gbk');
  console.log(text);
});

// Write a file encoded in UTF-16
const content = 'Sample content';
fs.writeFile('utf16-file.txt', content, 'utf16le', (err) => {
  if (err) throw err;
});

Handling Encoding in HTTP Requests

When processing HTTP requests, it is often necessary to handle request bodies with different encodings. Here is an example of handling a POST request:

const http = require('http');
const iconv = require('iconv-lite');

http.createServer((req, res) => {
  if (req.method === 'POST') {
    let body = [];
    req.on('data', (chunk) => {
      body.push(chunk);
    }).on('end', () => {
      body = Buffer.concat(body);
      
      // Assume the client sent data in GBK encoding
      const decodedBody = iconv.decode(body, 'gbk');
      console.log('Received:', decodedBody);
      
      res.end('Processing complete');
    });
  }
}).listen(3000);

Encoding Issues in Database Operations

Incorrect encoding settings when connecting to databases can lead to garbled text. Example for MySQL connection:

const mysql = require('mysql');
const connection = mysql.createConnection({
  host: 'localhost',
  user: 'root',
  password: 'password',
  database: 'test',
  charset: 'utf8mb4' // Supports full UTF-8, including emojis
});

connection.query('SELECT * FROM users', (error, results) => {
  if (error) throw error;
  console.log(results);
});

Solutions to Common Encoding Issues

Issue 1: Garbled File Text

Solution: Know the file encoding and read it correctly.

const fs = require('fs');
const iconv = require('iconv-lite');

// Read a file with unknown encoding and attempt conversion
const data = fs.readFileSync('unknown-encoding.txt');
const tryEncodings = ['utf8', 'gbk', 'big5', 'shift_jis'];

for (const encoding of tryEncodings) {
  try {
    const text = iconv.decode(data, encoding);
    console.log(`Tried encoding ${encoding}:`, text);
    break;
  } catch (e) {
    continue;
  }
}

Issue 2: Garbled HTTP Response

Solution: Set the correct Content-Type header.

const http = require('http');

http.createServer((req, res) => {
  res.writeHead(200, {
    'Content-Type': 'text/html; charset=utf-8'
  });
  res.end('<h1>Chinese Content</h1>');
}).listen(3000);

Advanced Encoding Conversion Techniques

Use the iconv-lite library for complex encoding conversions:

const iconv = require('iconv-lite');

// Convert GBK to UTF-8
const gbkBuffer = Buffer.from([0xc4, 0xe3, 0xba, 0xc3]); // "你好" in GBK
const utf8Text = iconv.decode(gbkBuffer, 'gbk');
console.log(utf8Text); // Output: "你好"

// Convert UTF-8 to GBK
const newGbkBuffer = iconv.encode('Goodbye', 'gbk');
console.log(newGbkBuffer); // <Buffer d4 d9 bc fb>

Handling Base64-encoded strings:

// Convert string to Base64
const text = 'Encoding Example';
const base64 = Buffer.from(text).toString('base64');
console.log(base64); // "RW5jb2RpbmcgRXhhbXBsZQ=="

// Convert Base64 to string
const originalText = Buffer.from(base64, 'base64').toString();
console.log(originalText); // "Encoding Example"

Performance Optimization Tips

When processing large amounts of text data, encoding conversion can become a performance bottleneck. Here are some optimization suggestions:

1. Use streams for large files.
2. Avoid encoding conversions in loops.
3. Use Buffer operations directly for known encoding data.

Example of streaming large file processing:

const fs = require('fs');
const iconv = require('iconv-lite');

// Stream conversion of large file encoding
fs.createReadStream('big-gbk-file.txt')
  .pipe(iconv.decodeStream('gbk'))
  .pipe(iconv.encodeStream('utf8'))
  .pipe(fs.createWriteStream('big-utf8-file.txt'));

Encoding Detection in Practice

Use the jschardet library to detect the encoding of unknown text:

const jschardet = require('jschardet');
const fs = require('fs');

const data = fs.readFileSync('unknown.txt');
const result = jschardet.detect(data);
console.log('Detected encoding:', result); // e.g., { encoding: 'GBK', confidence: 0.99 }

// Convert encoding based on detection result
const iconv = require('iconv-lite');
const text = iconv.decode(data, result.encoding);
console.log('Converted text:', text);

Handling Special Character Scenarios

Handling UTF-8 files with BOM (Byte Order Mark):

const fs = require('fs');
const stripBom = require('strip-bom');

const content = fs.readFileSync('with-bom.txt');
const cleanContent = stripBom(content);
console.log(cleanContent.toString());

Handling URL-encoded strings:

const querystring = require('querystring');

// Decode URL-encoded string
const encodedStr = '%E4%B8%AD%E6%96%87';
const decodedStr = querystring.unescape(encodedStr);
console.log(decodedStr); // "中文"

// Encode string for URL
const str = 'Test';
const encoded = querystring.escape(str);
console.log(encoded); // "%E6%B5%8B%E8%AF%95"

Encoding Handling in Multilingual Environments

For handling mixed-language content, UTF-8 is usually the preferred encoding. Here is an example of processing a multilingual string:

const multiLangText = 'ChineseEnglish日本語한국어';

// Convert to UTF-8 byte sequence
const utf8Buffer = Buffer.from(multiLangText, 'utf8');
console.log(utf8Buffer);

// Calculate byte length in different encodings
console.log('UTF-8 length:', Buffer.byteLength(multiLangText, 'utf8'));
console.log('UTF-16 length:', Buffer.byteLength(multiLangText, 'utf16le'));
console.log('GBK length:', Buffer.byteLength(multiLangText, 'gbk'));

Batch File Encoding Conversion

Batch conversion of file encodings in a directory:

const fs = require('fs');
const path = require('path');
const iconv = require('iconv-lite');

function convertFilesInDir(dir, fromEncoding, toEncoding) {
  fs.readdirSync(dir).forEach(file => {
    const fullPath = path.join(dir, file);
    if (fs.statSync(fullPath).isFile()) {
      try {
        const content = fs.readFileSync(fullPath);
        const converted = iconv.decode(content, fromEncoding);
        fs.writeFileSync(fullPath, iconv.encode(converted, toEncoding));
        console.log(`Converted successfully: ${file}`);
      } catch (e) {
        console.error(`Conversion failed for ${file}:`, e.message);
      }
    }
  });
}

// Example: Convert all files in the current directory from GBK to UTF-8
convertFilesInDir(__dirname, 'gbk', 'utf8');