node.js - NodeJS consistent file hashing across operating systems - Stack Overflow

admin管理员组

文章数量:1024627

I am hashing files in NodeJS using code similar to this:

import { createReadStream } from 'fs';
import { createHash } from 'crypto';
import stream from 'stream/promises';

export async function fileHash(filePath, algorithm = 'sha256') {
  const input = createReadStream(filePath);
  const hash = createHash(algorithm);
  await stream.pipeline(input, hash);
  return hash.digest('hex');
}

However when run on Linux, MacOS and Windows systems this can create different hashes due to different end of lines (EOL).

When I update my code to strip EOL differences:

import { createReadStream } from 'fs';
import { createHash } from 'crypto';
import stream from 'stream/promises';
import { Transform } from 'stream';

export async function fileHash(filePath, algorithm = 'sha256') {
  const normalizeLineEndings = new Transform({
    transform(chunk: Buffer, encoding, callback) {
      const normalized = chunk.toString('utf8').replace(/\r\n/g, '\n').replace(/\r/g, '\n');
      callback(null, Buffer.from(normalized));
    },
  });
  const input = createReadStream(filePath);
  const hash = createHash(algorithm);
  await stream.pipeline(input, normalizeLineEndings, hash);
  return hash.digest('hex');
}

Using this code produces consistent hashes across Operating Systems, but the hashes do not match a hash generated using a system program e.g.

shasum -a 256 myfile.zip

Is there away to generate hashes that match system tools AND produce the same hash cross Operating System?

I am hashing files in NodeJS using code similar to this:

import { createReadStream } from 'fs';
import { createHash } from 'crypto';
import stream from 'stream/promises';

export async function fileHash(filePath, algorithm = 'sha256') {
  const input = createReadStream(filePath);
  const hash = createHash(algorithm);
  await stream.pipeline(input, hash);
  return hash.digest('hex');
}

However when run on Linux, MacOS and Windows systems this can create different hashes due to different end of lines (EOL).

When I update my code to strip EOL differences:

import { createReadStream } from 'fs';
import { createHash } from 'crypto';
import stream from 'stream/promises';
import { Transform } from 'stream';

export async function fileHash(filePath, algorithm = 'sha256') {
  const normalizeLineEndings = new Transform({
    transform(chunk: Buffer, encoding, callback) {
      const normalized = chunk.toString('utf8').replace(/\r\n/g, '\n').replace(/\r/g, '\n');
      callback(null, Buffer.from(normalized));
    },
  });
  const input = createReadStream(filePath);
  const hash = createHash(algorithm);
  await stream.pipeline(input, normalizeLineEndings, hash);
  return hash.digest('hex');
}

Using this code produces consistent hashes across Operating Systems, but the hashes do not match a hash generated using a system program e.g.

shasum -a 256 myfile.zip

Is there away to generate hashes that match system tools AND produce the same hash cross Operating System?

• node.js
• file
• hash
• sha

Share Improve this question Follow asked Nov 19, 2024 at 6:40 Kim TKim T 6,49222 gold badges6262 silver badges8484 bronze badges 2

• 2 System tools don't care about line endings. Your function shouldn't produce different hashes on different platforms for the same file. If the files have different line endings then the hashes are expected to be different because the files are different. – robertklep Commented Nov 19, 2024 at 7:06
• I'm testing by hashing a file in the git repository. So git must be transforming the line endings when running a git checkout – Kim T Commented Nov 19, 2024 at 15:58

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

I was hashing a file committed to a git code repository. During git checkout file line-endings are updated based on the operating system. This meant that the same file in git was producing different hashes. The solution is to not hash files from git. Download the file independently of git.

I am hashing files in NodeJS using code similar to this:

import { createReadStream } from 'fs';
import { createHash } from 'crypto';
import stream from 'stream/promises';

export async function fileHash(filePath, algorithm = 'sha256') {
  const input = createReadStream(filePath);
  const hash = createHash(algorithm);
  await stream.pipeline(input, hash);
  return hash.digest('hex');
}

However when run on Linux, MacOS and Windows systems this can create different hashes due to different end of lines (EOL).

When I update my code to strip EOL differences:

import { createReadStream } from 'fs';
import { createHash } from 'crypto';
import stream from 'stream/promises';
import { Transform } from 'stream';

export async function fileHash(filePath, algorithm = 'sha256') {
  const normalizeLineEndings = new Transform({
    transform(chunk: Buffer, encoding, callback) {
      const normalized = chunk.toString('utf8').replace(/\r\n/g, '\n').replace(/\r/g, '\n');
      callback(null, Buffer.from(normalized));
    },
  });
  const input = createReadStream(filePath);
  const hash = createHash(algorithm);
  await stream.pipeline(input, normalizeLineEndings, hash);
  return hash.digest('hex');
}

Using this code produces consistent hashes across Operating Systems, but the hashes do not match a hash generated using a system program e.g.

shasum -a 256 myfile.zip

Is there away to generate hashes that match system tools AND produce the same hash cross Operating System?

I am hashing files in NodeJS using code similar to this:

import { createReadStream } from 'fs';
import { createHash } from 'crypto';
import stream from 'stream/promises';

export async function fileHash(filePath, algorithm = 'sha256') {
  const input = createReadStream(filePath);
  const hash = createHash(algorithm);
  await stream.pipeline(input, hash);
  return hash.digest('hex');
}

However when run on Linux, MacOS and Windows systems this can create different hashes due to different end of lines (EOL).

When I update my code to strip EOL differences:

import { createReadStream } from 'fs';
import { createHash } from 'crypto';
import stream from 'stream/promises';
import { Transform } from 'stream';

export async function fileHash(filePath, algorithm = 'sha256') {
  const normalizeLineEndings = new Transform({
    transform(chunk: Buffer, encoding, callback) {
      const normalized = chunk.toString('utf8').replace(/\r\n/g, '\n').replace(/\r/g, '\n');
      callback(null, Buffer.from(normalized));
    },
  });
  const input = createReadStream(filePath);
  const hash = createHash(algorithm);
  await stream.pipeline(input, normalizeLineEndings, hash);
  return hash.digest('hex');
}

Using this code produces consistent hashes across Operating Systems, but the hashes do not match a hash generated using a system program e.g.

shasum -a 256 myfile.zip

Is there away to generate hashes that match system tools AND produce the same hash cross Operating System?

• node.js
• file
• hash
• sha

Share Improve this question Follow asked Nov 19, 2024 at 6:40 Kim TKim T 6,49222 gold badges6262 silver badges8484 bronze badges 2

• 2 System tools don't care about line endings. Your function shouldn't produce different hashes on different platforms for the same file. If the files have different line endings then the hashes are expected to be different because the files are different. – robertklep Commented Nov 19, 2024 at 7:06
• I'm testing by hashing a file in the git repository. So git must be transforming the line endings when running a git checkout – Kim T Commented Nov 19, 2024 at 15:58

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

I was hashing a file committed to a git code repository. During git checkout file line-endings are updated based on the operating system. This meant that the same file in git was producing different hashes. The solution is to not hash files from git. Download the file independently of git.

本文标签： nodejsNodeJS consistent file hashing across operating systemsStack Overflow