Data serialization library for communication and storage purpose with versioning.

The library is focused on solving two specific issues:

1. The application version mismatch
   Data serialization is a complicated task, especially when you are maintaining a project for a long time. With addition of a new feature, it becomes complicated to keep your data readable between two versions of your same software.

   • You have saved your data using older version of your software, which would pose an issue when loading with a newer version of your software.
   • You have a client server application, where the server has been updated to a newer version, but there still exists client that hasn't updated for a long time. In rare case scenario (like our hotspot games), the server could be an older version whereas the client might have been a newer version.

2. Loading class instances from serializable data
   While loading data either from storage or a communication channel, it is always extra work to map that data to particular instance. We try to solve that with two different approach:

   1. Make the classes Serializable and use Oracle for serialization, identification and creation of class instances.
   2. For certain scalar objects (ex: ComplexNumber), treat them like a primary data type and create helper functions serializeComplexNumber to serialize these objects.

   function serializeComplexNumber(serializer: Serializer, complexNumber: ComplexNumber) {
     if (serializer.isLoading) {
       const real = serializer.double(0);
       const imaginary = serializer.double(0);
       return new ComplexNumber(real, imaginary);
     } else {
       serializer.double(complexNumber.real);
       serializer.double(complexNumber.imaginary);
       return complexNumber;
     }
   }

Side Goal
When it comes to serialization, most of the time, the error arises due to performing data reading and writing with two different codes. We have tried to minimize that by doing both reading and writing with the same code. For example, a typical object serialization would look like:

function serialize(serializer: Serializer) {
  this.name = serializer.string(this.name);
  this.dob = new Date(serializer.int32(this.dob.getTime() / 1000) * 1000);
  this.salary = serializer.double(this.salary);
}

In rare cases, if you need a different logic for reading and writing, the isLoading flag on Serializer should be used.

yarn add @bhoos/serialization

Use the basic function from Serializer for primary data type serialization:

• int8
• int16
• int32
• uint8
• uint16
• uint32
• float
• double
• string
• bool
• string

There are some Serializer functions (may not be implemented for all provided serializers!) to use with composite data. They have the following signature:

• json<T>: T
• obj<T>: T
• array<T>: T
• larray<T>: T (array with a limit)

For composite objects, we need to also define how the primary components should be serialized. For a class with a private composite member payload, the following works:

type Payload = {
  timestamp: number,
  tokens: string[]
}

serialize(serializer: Serializer, oracle: Oracle) {
  // this.payload is of type Payload
  this.payload = serializer.obj(this.payload, (payload) => {
    payload.timestamp = serializer.uint16(payload.timestamp);
    payload.tokens = serializer.array(payload.tokens, (token, serializer) => {
      return serializer.string(token);
    })
  })
}

Properties available in Serializer

• version The version of the serializer typically based on the app.
• isLoading The mode of the serializer. Based on the mode, the data is either read (loading) or written.

To implement the Serializable interface, all you need is to implement the serialize function for a given class. In actual practice, however, you might want to also write an object initialization function to deserialize. So, for a complete serializable-deserializable object, an interface like the one below is necessary:

class MySerializable implements Serializable {
  private id: number;

  serialize(serializer: Serializer, oracle: Oracle): void {
    this.id = serializer.uint32(this.id);
  }

  static create(id: number) {
    const k = new MySerializable();
    k.id = id;
    return k;
  }
}

Multiple versions of classes can be handled as follows:

import { Serializable, Serializer } from '@bhoos/serialization'
class ComplexNumber implements Serializable {
  real: number;
  imaginary: number;

  // Implement the serialize function
  serialize(serializer: Serializer) {
    // Assuming we used floating size number in earlier version
    // and later upgraded to double size numbers
    if (serializer.version < 2 ) {
      serializer.float(this.real);
      serializer.float(this.imaginary);
    } else {
      serializer.double(this.real);
      serailizer.double(this.imaginary);
    }
  }
}

There are three main requirements this library fulfills for any application: serialization, deserialization and identification of an unknown buffer. Additionally, serializables need to be registered with an oracle (that must be a shared schema between the serializing party and the deserializing one).

Here we first initialize an Oracle, then register the ComplexNumber serializable class with it. The register function takes a numerical id, a class and the default constructor for the class as arguments.

const oracle = new Oracle();
const COMPLEX_NUMBER_ID = 0x1000;

oracle.register(COMPLEX_NUMBER_ID, ComplexNumber, () => new ComplexNumber());

sendObject(obj: Serializable, channel: Channel) {
  // initialize a buffer serializer
  const serializer = new BufferSerializer(this.version, 1000);
  this.oracle.serialize(obj, serializer);
  channel.send(serializer.getBuffer()); // getBuffer returns a node Buffer
}

receiveObject(message: ArrayBuffer) {
  // converting ArrayBuffer to Buffer
  const serializer = new BufferSerializer(this.version, Buffer.from(message));
  const obj = this.oracle.serialize(null, serializer);
  return obj;
}

Often in applications, we need to identify what class a given serializable object belongs to (usually at the receiving end to decide what to do with a message). The Oracle.identify() function should be used to obtain the class id.

isComplexNumber(obj: Serializable): boolean {
  return Boolean(Oracle.identify(obj) & COMPLEX_NUMBER_ID);
}

The standard JSON (JSONSerializer) and Buffer (binary) (BufferSerializer) serializers are provided with the library. If any other format are required, they can be created by implementing the Serializer interface.

The JSONSerializer may not be currently functional.