Streams

A stream is a way of processing data piece by piece, as it arrives, instead of waiting for the entire dataset to be available before doing anything with it. Streams are one of Node.js's core abstractions — used internally by the fs, http, and zlib modules — and they exist to solve a specific, practical problem: handling data that is too large, or too slow to arrive, to comfortably hold in memory all at once.

If you have ever watched a video start playing before the whole file has finished downloading, you have already experienced what streams make possible.

The Problem Streams Solve

Methods like fs.readFile() load an entire file into memory before returning anything. For a small JSON config file, this is fine — the file is a few kilobytes, and loading it entirely is instant and harmless.

For a 4GB video file, this approach breaks down. Loading 4GB into memory at once is wasteful at best and impossible at worst, depending on how much memory is actually available. And there is no need to wait for the entire file to load before starting to do something useful with it — a video player does not need the whole file before it can start playing the first frame.

Streams solve this by processing data in chunks — small pieces, arriving over time — rather than loading everything as a single unit. Memory usage stays low and predictable, and processing can begin before the entire source is even available.

The Four Types of Streams

Node.js defines four categories of streams, each suited to a different direction of data flow.

Readable Streams

A readable stream is a source of data that arrives over time, in chunks.

Reading a File as a Stream

const fs = require('fs');

const readStream = fs.createReadStream('large-file.txt', { encoding: 'utf8' });

readStream.on('data', (chunk) => {
  console.log('Received chunk:', chunk.length, 'characters');
});

readStream.on('end', () => {
  console.log('Finished reading the file');
});

readStream.on('error', (err) => {
  console.error('Error reading file:', err.message);
});

fs.createReadStream() returns a readable stream immediately — it does not wait for the file to load. The 'data' event fires repeatedly, once for each chunk that becomes available. The 'end' event fires once after the entire stream has been consumed. The 'error' event fires if something goes wrong during reading — and unlike a Promise, a stream's error must always be handled with this listener, or the error becomes an uncaught exception.

Chunk Size

By default, Node.js reads file streams in chunks of 64KB (this can be customised with the highWaterMark option). A 4GB file is processed as roughly 65,000 separate chunks, each one small enough to hold in memory comfortably, processed one after another.

const readStream = fs.createReadStream('large-file.txt', {
  encoding: 'utf8',
  highWaterMark: 1024, // 1KB chunks instead of the 64KB default
});

Writable Streams

A writable stream is a destination that data can be sent to incrementally.

const fs = require('fs');

const writeStream = fs.createWriteStream('output.txt');

writeStream.write('First line\n');
writeStream.write('Second line\n');
writeStream.end('Final line\n');

writeStream.on('finish', () => {
  console.log('All data has been written');
});

write() sends a chunk of data to the stream. end() signals that no more data will be written and closes the stream after flushing whatever remains. The 'finish' event fires once everything has actually been written to the underlying destination.

Piping — Connecting Streams Together

The most common and powerful pattern with streams is piping — connecting a readable stream directly to a writable stream, so that data flows from source to destination automatically, in chunks, without manual event handling.

const fs = require('fs');

const readStream = fs.createReadStream('large-video.mp4');
const writeStream = fs.createWriteStream('copy.mp4');

readStream.pipe(writeStream);

This single line does what would otherwise require manually listening for 'data' events on the readable stream and calling write() on the writable stream for each chunk. pipe() handles this coordination automatically, and — critically — it also handles backpressure.

Backpressure

Backpressure is what happens when a readable stream produces data faster than a writable stream can consume it. Without any coordination, the readable stream would keep emitting chunks regardless of whether the writable side has caught up, and those unconsumed chunks would pile up in memory — defeating the entire purpose of using a stream in the first place.

pipe() automatically pauses the readable stream when the writable stream's internal buffer fills up, and resumes it once the writable stream has caught up and signals it is ready for more. This is the single most important reason to use pipe() rather than manually wiring 'data' and write() together — manual wiring requires you to implement backpressure handling yourself, and it is easy to get wrong.

Transform Streams

A transform stream is both readable and writable, where the data coming out is a modified version of the data going in. This makes it possible to insert a processing step directly into a pipe chain.

Compression with zlib

const fs = require('fs');
const zlib = require('zlib');

const readStream = fs.createReadStream('large-file.txt');
const gzip = zlib.createGzip();
const writeStream = fs.createWriteStream('large-file.txt.gz');

readStream.pipe(gzip).pipe(writeStream);

This reads the file, compresses it using gzip, and writes the compressed output — all in a single chain, processing the data in chunks throughout, never loading the entire file into memory. Multiple pipe() calls can be chained together this way, with each stage transforming the data before passing it to the next.

A Custom Transform Stream

You can build your own transform stream when the built-in ones (zlib, crypto ciphers) do not cover what you need:

const { Transform } = require('stream');

const uppercaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    const uppercased = chunk.toString().toUpperCase();
    callback(null, uppercased);
  },
});

process.stdin.pipe(uppercaseTransform).pipe(process.stdout);

This reads from standard input, converts each chunk to uppercase, and writes the result to standard output — a simple but complete example of a custom transformation applied as data flows through.

Streams in the http Module

The req and res objects covered in the previous topic are themselves streams. req is a readable stream — the request body arrives in chunks. res is a writable stream — the response is sent in chunks.

This means file streaming and HTTP can be connected directly, without ever loading a file into memory:

const http = require('http');
const fs = require('fs');

const server = http.createServer((req, res) => {
  if (req.url === '/video') {
    const videoStream = fs.createReadStream('large-video.mp4');
    res.writeHead(200, { 'Content-Type': 'video/mp4' });
    videoStream.pipe(res);
    return;
  }

  res.writeHead(404);
  res.end('Not found');
});

server.listen(3000);

videoStream.pipe(res) streams the video file directly to the HTTP response, chunk by chunk, as it is read from disk. The server never holds the entire video in memory — even for a multi-gigabyte file, memory usage stays low and constant throughout the request. This is precisely how real-world video and large file delivery works in production servers, and it is a direct, practical payoff of understanding streams.

Handling Errors in a Pipe Chain

A limitation of pipe() is that it does not automatically forward errors — if the readable stream emits an 'error' event, it does not automatically propagate to the writable stream or stop the chain cleanly. Each stream in a chain needs its own error handling:

readStream.on('error', (err) => {
  console.error('Read error:', err.message);
});

writeStream.on('error', (err) => {
  console.error('Write error:', err.message);
});

readStream.pipe(writeStream);

stream.pipeline() — the Safer Alternative

Node.js provides stream.pipeline() as a more robust alternative to chained .pipe() calls. It automatically handles errors across the entire chain and ensures all streams are properly closed when the pipeline finishes or fails:

const { pipeline } = require('stream/promises');
const fs = require('fs');
const zlib = require('zlib');

async function compressFile(input, output) {
  try {
    await pipeline(
      fs.createReadStream(input),
      zlib.createGzip(),
      fs.createWriteStream(output)
    );
    console.log('Compression complete');
  } catch (err) {
    console.error('Pipeline failed:', err.message);
  }
}

compressFile('large-file.txt', 'large-file.txt.gz');

stream/promises provides a Promise-based version of pipeline, usable with async/await. This is the recommended modern approach for chaining streams — it solves the error-propagation gap that raw .pipe() chains leave open, and it guarantees proper cleanup of every stream in the chain regardless of where a failure occurs.

When to Use Streams

Streams are not always the right tool. They introduce more complexity than simply loading data into memory, and that complexity is only worth it under specific conditions.

Summary