Skip to main content

JavaScript

Typed Array​

From ES6.

  • ArrayBuffer: Buffer, representing a block of memory, whose content can only be manipulated through View.
  • TypedArray: View, Array for storing fixed-type data, such as Uint8Array(8-bit unsigned integer), Float64Array(64-bit IEEE floating point number).
  • DataView: View, unlimited type, can customize which byte, type, and byte order (endian) to access.

ArrayBuffer​

ArrayBuffer represents a fixed-size memory block, also known as byte-array. The main function is to configure physical memory to store raw binary data. Generally, ArrayBuffer is seldom directly manipulated, and in fact, its reference can only be passed to other objects, so that other objects can process/use the data.

There are many ways to create an ArrayBuffer, which can be configured directly,

// allocate 8 bytes, initial value is 0
const buffer = new ArrayBuffer(8)

// Clone the last 4 bytes of 'buffer' to another ArrayBuffer
const bufferCopied = buffer.slice(-4)

// XMLHttpRequest assign resopnseType (XMLHttpRequest v2)
const xhr = new XMLHttpRequest()
xhr.open('GET', '/path/to/image.jpg', true)
xhr.responseType = 'arraybuffer' // set type of reponse to arraybuffer
xhr.onload = function(e) {
  console.log(this.response) // this.response is ArrayBuffer
}
xhr.send()

// Fetch API provide Body#arrayBuffer to convert Request/Response stream body
const response = await fetch('/path/to/image.gif')
const buffer = await response.arrayBuffer() // get ArrayBuffer instance
console.log(buffer.byteLength) // get length of bytes.

You can Filealso FileReaderread the data uploaded by users through and API.

const input = document.querySelector('input')
input.addEventListener('change', handleFiles, false);

function handleFiles (files) { // files -> FileList object, File instance in it
  if (files[0]) {
    const reader = new FileReader()
    reader.onload = function ({ target: { result } }) {
      console.log(result.byteLength) // result is a ArrayBuffer
    }
    reader.readAsArrayBuffer = files[0] // File is a special Blob type
  }
}

TypedArray​

TypedArray is not any type, nor is it a globally accessible constructor, but an abstract concept that corresponds to many different types of Arrays. To be honest, the naming of TypedArray says it all, let me explain the text, starting with Typed.

Types of TypedArray​

The so-called Typed means " limited type ", and the elements in Array are all of the same type. What types are there? TypedArray is born to operate binary, of course, only the bottom layer is based on bytes, and there are almost no types of abstract concepts. We can decide how much each element should read from raw data and how to read bytes according to the requirements.

Currently ES6 defines the following typed array types:

typebytes/elementCorresponding to C language
Int8Array1int8_t
Uint8Array1uint8_t
Uint8ClampedArray1uint8_t
Int16Array2int16_t
Uint16Array2uint16_t
Int32Array4int32_t
Uint32Array4uint32_t
Float32Array4float
Float64Array8double

In fact, TypedArray itself does not store any buffer data, only the reference of the buffer, and we can TypedArray#bufferget . Therefore, the same ArrayBuffer can construct multiple different TypedArrays. It can be regarded as " interpreting binary data in ArrayBuffer from different perspectives ".

The elements of different types of TypedArray correspond to different byte numbers, and this information will be recorded on TypedArray#BYTES_PER_ELEMENTthe property. For example: one element of Uint8Array corresponds to one byte, and Float64Array corresponds to 8 bytes.

Array-like Methods​

The Array in TypedArray can be regarded as a layer of Array API that can access data on top of the data in ArrayBuffer. Providing methods map, filter, reduce , but the Mutator methods that can change the length of the Array like push are not implemented. After all, TypedArray is just a reference to the buffer, the original data still exists after pop.

What's more interesting is that bothTypedArray#subarray and TypedArray#slice return an array slice.

slice return a shallow-copy new Array . The buffer property of the new array points to the newly cut buffer, the byteoffset which is also based on the new buffer, so it will be 0.

subarray is slicing in the same buffer. bufferwill get the same original buffer with calling subarray, byteOffset calculate the offset based on the original buffer.

Consturct a TypedArray​

Direct initialization​

Create a Uint16Array of 4 * 2 bytes whose initial value is 0.

const u16 = new Uint16Array(4)

Build from TypedArray​

We can also create TypedArrays of the same length from other TypedArrays, which will point to the same buffer.

const u8 = new Uint8Array(u16) // length of u8 is 4

This will create a new type array, but the memory block will not change. In our example, because of overflow, u8 automatically filters even bytes (or odd numbers, depending on endianness ), and only displays the data of the remaining 4 bytes, and the memory addresses become discontinuous.

Build from ArrayBuffer​

However, it is also possible TypedArray#bufferto obtain and share the current buffer, and the memory range of the array will be continuous.

// buffer 4 * 2 = 8 bytes, u8_continuous size 8
const u8_continuous = new Uint8Array(u16.buffer)

Of course, ArrayBuffer can directly allocate a block of memory, and use it to construct TypedArray, and lengtheven byteOffsetconstruct by specifying different ranges of the buffer with and .

const buffer = new ArrayBuffer(16)
const i32 = new Int32Array(buffer) // 32 * 2 bytes

// From 4 bytes offset address, slice 7 bytes array.
const i8 = new Iint8Array(buffer, 4, 7)

Overflow​

The overflow handling rule of TypedArray is the same as that of most languages: the overflow high bits are discarded. Let's look at a simple example.

Uint8Array.of(0xff, 0x100)
// Unit8Array [255, 0]

// 255, 8 bits
0b11111111

// 256, 9 bits
0b100000000
//└── sush 1 overflow, discard
0x100 & 0xFF
// 0

Note: Underflow is handled in the same way as overflow.

What is Uint8ClampedArray​

Clamp's original meaning is pliers. In computer science, it usually means to limit data values to a specific range. Uint8ClampedArrayIn , the element value is limited to 0 - 255. In other words, the rules for handling overflow are Uint8Arraydifferent . When overflow, this value will be equal to the maximum value of 255; when underflow, this value will be equal to 0 .

Uint8Array.of(0xff, 0x100, -100)
// Unit8Array [255, 0, 156]
Uint8ClampedArray.of(0xff, 0x100, -100)
// Uint8ClampedArray [255, 255, 0]

What good is that? It is very convenient in image processing. There is a very common example, there is a Uint8Array of 3 bytes to store the RGB color code, we want to increase its gamma factor, if use Uint8Array to store:

// avoid overflow/underflow
u8[i] = Math.max(0, Math.min(255, u8[i] * gamma)) // u8 is a Uint8Array

If so Uint8ClampedArray, you only need to multiply the gammer factor directly, which is very convenient.

pixels[i] *= gamma // pixels is a Uint8ClampedArray

Composite Data Structure​

When it is necessary to process a compound data structure similar to C struct, as shown below

struct employee {
  unsigned int id;               // 4 * 1 bytes
  unsigned char department[4];   // 1 * 4 bytes
  float salary;         // 4 * 1 bytes
};

We can declare the corresponding TypedArray to handle. Simulate if the data structure of C struct.

const buffer = new ArrayBuffer(12)
const idView = new Uint32Array(buffer, 0, 1)
const deptView = new Uint8Array(buffer, 4, 4)
const salaryView = new Float32Array(buffer, 8)
idView[0] = 123
deptView.forEach((_, i) => { deptView[i] = i * i })
salaryView[0] = 10000

DataView​

As the name suggests, DataView is a view built on buffer. Different from general TypedArray, there is no fixed data type when constructing DataView. Instead, when accessing data, you must clearly specify which data type to get from which byte offset.

Borrow the composite data from the previous example to demonstrate DataViewhow to process custom data for each byte.

const dv = new DataView(buffer)
// byte offset 0 read Uint32
// ID -> 123
dv.getUint32(0, true)

// byte offset 8 write Float32
dv.setFloat32(8, 200000, true)
dv.getFloat32(8, true)

Have you noticed that the bytes getter/setter of DataView has an extra boolean parameter at the end? This parameter specifies to use Little-endian to read data, and falsethe to read in Big-endian. Controllable endian is a very important but annoying feature of DataView. Endianness will be introduced in the next section.

Another important feature of DataView is that it will not have buffer overflow . The so-called buffer overflow is " when writing a piece of data into the specified buffer, if the size of the written data exceeds the boundary of the buffer, the overflow value will overwrite the next byte ". This unsafe nature also makes buffer overflow an attack method of many hackers, which has potential security problems. However, assigning a number that exceeds the maximum value of the type through the DataView setter will not overwrite the data of the adjacent memory address. Instead, the boundary will be checked internally, and the overflow will be processed before writing to the memory area, which makes up for the loophole of buffer overflow.

Precautions​

Typed Arrays can almost do memory operations as delicate as C language. However, the freer the API, the more knowledge you need to learn and pay more attention to details. The following are the things you should keep in mind when operating TypedArray:

Endianness (Byte order)​

--- read order -->
| Offset | 0    | 1    | 2    | 3    |
| ------ | ---- | ---- | ---- | ---- |
| Data   | 0x11 | 0x22 | 0x33 | 0x44 |

# Little-endian 0x44332211 PC
# Big-endian 0x11223344 net protocol

Q: How to deal with endian in JavaScript?

If you don't touch the underlying memory operation, you don't need to worry about endianness when writing JavaScript, but when you want to operate TypedArray, it is very important to understand the endianness of data. TypedArrayThe default is to use the endianness of the system, so if you receive a piece of data that is inconsistent with the endianness of your system, TypedArray will not work. The last parameter of the byte getter/setter of the DataView introduced earlier is used to determine which byte order to access data. The default is Big-endian ( false ).

Q: Then how do we know the byte order of the data?

If the data is used by your own internal system, in fact, it is OK if you communicate well, and no one will care about you if you use Mixed-endian. But if it is any data obtained from the outside world, we can use " BOM (byte order mark) " to determine which endianness the data belongs to. BOM is a Unicode magic number, usually placed at the forefront of the text stream. However, not every data will add this header, and sometimes we don’t need BOM information, and we must strip bom before using the data.

Data Structure Alignment​

To get in touch with the underlying memory, it is inevitable to understand how the CPU reads data from the memory and how the bottom layer of the memory is configured.

Generally speaking, modern CPUs are usually designed to read and write data in the memory in units of words (for example, 4 bytes), and data alignment (data alignment) is to place data on memory addresses that are word-size * n times , so that the CPU reads and writes in the most efficient way. So why is it most efficient to align word-size? Suppose there is a C struct as follows:

struct AlignDemo {
  char c;     // 1 byte
  int i;      // 4 bytes
  short s;      // 2 bytes
};

The theoretical memory configuration is as follows, and a total of 7 bytes of memory space is required.

c = char length of byte
s = short length of byte
i = int length of byte

| 0x000           | 0x020           |
| [c] [i] [i] [i] | [i] [s] [s] [ ] |

As mentioned above, the CPU accesses the data in the memory in word-size. There is no problem when trying to read char and short. The CPU only needs to fetch the word chunk once and then offset to get the correct value. However, when wanting to read an int, the CPU needs to first fetch the first data chunk to get the first three bytes of the int, and then fetch the second word chunk and shift the data to get the last byte of the int. Such redundant memory access will cause an additional burden on the CPU.

The solution is Data Structure Padding , that is, when the data cannot be aligned with the word-size, add some padding members.

In our example, this can be done like this:

struct AlignDemo {
  char c;
  char padding_0[3]; // padding
  int i;
  short s;
  char padding_1[2];
};

The memory configuration is as follows:

p = padding length of byte

| 0x000           | 0x020           | 0x040           |
| [c] [p] [p] [p] | [i] [i] [i] [i] | [s] [s] [p] [p] |

Originally only 7 bytes were needed, but after alignment, so many extra bytes were used, are you kidding me?

We can try to change the order of struct members:

struct AlignDemo {
  int i;
  char c;
  short s;
  char padding[0]
};

The corresponding memory configuration changes, only occupying 8 bytes. Brilliant!

| 0x000           | 0x020           |
| [i] [i] [i] [i] | [c] [s] [s] [p] |

Struct alignment (struct alignment) is not a small science in C language. In addition to the alignment of the members in the structure, the structure itself must also be aligned.

Going back to JavaScript, when you are creating different views, the JavaScript engine will actually perform a simple natural alignment check.

const buffer = new ArrayBuffer(6)
new Uint16Array(buffer, 1)
// RangeError: start offset of Uint16Array should be a multiple of 2
new Uint32Array(buffer, 0)
// RangeError: byte length of Uint32Array should be a multiple of 4

So, when we design composite data, we should think about the corresponding C struct alignment and consider the bottom layer of the memory so that the operation of binary data will not have the adverse effect of low performance.