SoFunction
Updated on 2025-03-04

Analysis of the principle and usage method of NodeJS module Buffer

As an important concept and function in nodejs, Buffer provides developers with the ability to operate binary systems. This article records several issues to deepen the understanding and use of Buffer:

  • Understand the buffer
  • How to apply for off-pile memory
  • How to calculate byte length
  • How to calculate byte length
  • How to convert character encoding
  • Understand shared memory and copy memory

Know Buffer (buffer)

Buffer is the nodejs core API that provides us with the ability to handle binary data flows. The use of Buffer is very similar to that of ES2017's Uint8Array, but due to the characteristics of node, it specifically provides a deeper API.

Uint8Array literally means: 8-bit unsigned integer array. A byte is 8 bit, and the representation of bytes is also composed of two hexadecimal (4 bit) numbers.

const buf = (1);
(buf); // output: <Buffer 00>

How to apply for off-pile memory

Buffer can break the nodejs limit on the memory size in the heap. Nodejs12 provides 4 APIs to apply for off-heap memory:

  • ()
  • (size[, fill[, encoding]])
  • (size)
  • (size)

vs

When applying for memory, it may have stored other data before this memory. If the original data is not cleared, there will be a security risk of data leakage; if the original data is cleared, the speed will be slower. Which method is used depends on the actual situation.

  • : Apply for a specified size of memory, and clear the original data, and the default fill is 0
  • : Apply for specified memory size, but do not clear the original data, the speed is faster

Depending on the provided API, an alloc can be implemented manually:

function pollifyAlloc(size, fill = 0, encoding = "utf8") {
  const buf = (size);
  (fill, 0, size, encoding);
  return buf;
}

vs

The effect can be seen directly from the naming, which is slower. Because when creating a new Buffer instance using , if the memory to be allocated is less than 4KB, it will be cut from a pre-allocated Buffer. This avoids excessive use of the garbage collection mechanism by creating too many independent buffers.

This approach improves performance and memory usage by eliminating the need for tracking and cleaning.

How to calculate byte length

Using Buffer, you can obtain the true bytes of the data. For example, a Chinese character has a character length of 1. However, since it is a Chinese character encoded by utf8, it takes up 3 bytes.

Directly using () can obtain the byte length of the specified encoding of the string:

const str = "Original address of this article: ";

((str, "utf8")); // output: 31
(); // output: 19

You can also directly access the length attribute of the Buffer instance (not recommended):

((str, "utf8").length); // output: 31

How to convert character encoding

Nodejs currently supports encoding formats: ascii, utf8, utf16le, ucs2, base64, latin1, binary, hex. Other encodings need to be completed with the help of a three-party library.

Below is the encoding conversion function of the nodejs platform encapsulated using () and ():

function trans(str, from = "utf8", to = "utf8") {
  const buf = (str, from);
  return (to);
}

// output: 5Y6f5paH5Zyw5Z2AOiB4eG9vNTIxLmNvbQ==
(trans("Original address: ", "utf8", "base64"));

Shared and copied memory

When generating a Buffer instance and operating binary data, be sure to pay attention to whether the interface is based on shared memory or based on copying the underlying memory.

For example, for the from() that generates a Buffer instance, the behavior of nodejs is different for different types of parameters.

For a more vivid explanation, please see the following two codes.

Code 1:

const buf1 = ("buffer");
const buf2 = (buf1); // Copy the buffer data in the parameter to the new instance
buf1[0]++;

(()); // output: cuffer
(()); // output: buffer

Code 2:

const arr = new Uint8Array(1);
arr[0] = 97;

const buf1 = ();
(()); // output: a

arr[0] = 98;
(()); // output: b

In the second piece of code, the parameter type passed in is arrayBuffer. So it's just creating a view, not copying the underlying memory. The memory of buf1 and arr is shared.

During the operation of the Buffer, special attention should be paid to the difference between sharing and copying, and errors are difficult to check.

The above is all the content of this article. I hope it will be helpful to everyone's study and I hope everyone will support me more.