Tiny Web Metaverse Client core concept

This document describes the Tiny Web Metaverse Client core concept needed for
Client or Addons development.

Client overview

Client is the software (Web page) that end-users directly operate in the Tiny
Web Metaverse framework, and it mainly plays the following roles.

- 3D graphics rendering using WebGL
- VR/AR processing using WebXR, such as positional tracking
- Network synchronization of object states with remote clients using
WebSockets via
State server
- Audio and video communication with remote clients using WebRTC
via Stream server
- Input handling from input devices such as mouse, keyboard, touchscreen, VR headset,
and so on

Client core

Client provides the minimum and basic functions for the virtual 3D space.
Advanced functions are implemented extensively by framework users.

For example, Client provides a way to define an avatar, but setting the
model and other settings of the avatarthe is user's responsibility.

Additionally, Client detects input from input devices and shapes it into a form
that is easy to process, but the users implement how to process the input.

For example, the users implement how to move the avatar with which button when
keyboard input is received.

Moreover, Client does not involve in the 2D UI that is overlapped on the 3D
Canvas. The implementation of the 2D UI is the responsibility of the user.

Hereafter, The "Client that provides basic functions" will be referred to as
"Client core" when emphasizing it

Addons

Framework users need to implement application-specific processing outside of
the Client core. However, many of these processes are highly reusable.

For example, the operation of an avatar using a keyboard is a similar process
in many applications. Such processes are published as Addons.

By selectively importing Addons, framework users do not need to implement
highly reusable processes themselves.

If you want to fine-tune the functionality of an Addon, copy the source code
and edit it for use.

If you want to add an addon to Addons, please send a
Pull Request.

Core third-party libraries

$3

Tiny Web Metaverse Client uses the JavaScript 3D graphics library Three.js
to manage objects in 3D space, render 3D space using WebGL, and process VR and
AR using WebXR.

Three.js knowledge is essential for developing the Client core, Addon, and
user apps. If you are new to Three.js, please learn the basics from the
documentation.

The following explanations assume that the reader has a basic understanding
of Three.js.

$3

Tiny Web Metaverse Client uses the JavaScript ECS library
bitECS.

Similarly to Three.js, basic knowledge of bitECS is required to develop a
Client, Addon, or user app. If you are not familiar with bitECS, I recommend
learning the basics from the documentation.

From here on, this document assumes that the reader has basic knowledge of bitECS.

ECS architecture

Tiny Web Metaverse Client adopts
ECS (Entity Component System) architecture.

Entity Component System (ECS) is a software architectural pattern commonly used in game development for representing game world objects. It decomposes game objects into three distinct parts: entities, components, and systems.

- Entities: Entities represent the unique objects in the game world. They serve
as mere identifiers and don't hold any data or behavior. Each entity is
assigned a unique identifier, allowing systems to reference and operate on
them.
- Components: Components are data containers that hold the attributes and
properties of entities. They are essentially data structures that encapsulate
the characteristics of an entity, such as position, velocity, health, sprite,
and other relevant information.
- Systems: Systems are the functional units that act upon entities and
components. They represent the behavior and logic of the game, processing
data from components and modifying entities accordingly. Systems operate on
groups of entities that possess specific components.

ECS in Tiny Web Metaverse

$3

Use addEntity() of bitECS to create a new entity.

``typescript
import { addEntity, IWorld } from "bitecs";

const somewhere = (world: IWorld): void => {
const eid = addEntity(world);
...
};

`

The essence of an entity is an integer, called an Entity ID. It is often
abbreviated as eid in Tiny Web Metaverse.

When an entity is deleted by the removeEntity() of bitECS, the Entity ID is
returned to a pool managed by bitECS. This ID can be reused later.

In Tiny Web Metaverse, we assume that the pool is large enough that Entity
IDs are not immediately reused, for simplicity.

$3

Use defineComponent() of bitECS to define a component and addComponent()
to assign a component to an entity.

`typescript
// src/components/foo.ts

import { defineComponent } from "bitecs";

export const FooComponent = defineComponent();

// Add a component to an entity

import { IWorld } from "bitecs";
import { FooComponent } from "../components/foo";

const somewhere = (world: IWorld, eid: number): void => {
addComponent(world, FooComponent, eid);
...
};

`

You can define component with component data definition.

`typescript
// src/components/foo.ts
import { defineComponent, Types } from "bitecs";

export const FooComponent = defineComponent({
data: Types.f32
});

// Add a component to an entity and
// initialize the component data

import { IWorld } from "bitecs";
import { FooComponent } from "../components/foo";

const somewhere = (world: IWorld, eid: number): void => {
addComponent(world, FooComponent, eid);
FooComponent.data[eid] = 0.0;
...
};
`

bitECS doesn't support non-numeric data types natively. If you want to use
non-numeric data type, use Component Proxy
style. (Our Component Proxy style is not exactly same as the bitECS one but
inspired by it.)We use static get method to reuse Proxy instance to maintain
high performance iteration.

`typescript
// src/components/foo.ts
import { defineComponent } from "bitecs";
import { Foo } from "foo-lib";
import { NULL_EID } from "../common";

export const FooComponent = defineComponent();

export class FooProxy {
private static instance: FooProxy = new FooProxy();
private eid: number;
private map: Map;

private constructor() {
this.eid = NULL_EID;
this.map = new Map();
}

static get(eid: number): Foo {
FooProxy.instance.eid = eid;
return FooProxy.instance;
}

allocate(foo: Foo): void {
this.map.set(this.eid, foo);
}

free(): void {
this.map.delete(this.eid);
}

get foo(): Foo {
return this.map.get(this.eid)!;
}
}
`

When you add a component that has proxy to an entity you also need to get
proxy and initialize the component data.

`typescript
import { IWorld } from "bitecs";
import { Foo } from "foo-lib";
import { FooComponent, FooProxy } from "../components/foo";

const somewhereToAdd = (world: IWorld, eid: number): void => {
addComponent(world, FooComponent, eid);
const proxy = FooProxy.get(eid);
proxy.allocate(new Foo());
...
};
`

TODO: A mechanism is needed to ensure that a component and its data managed by
proxy have the same lifetime. Currently, this requires manual oversight, which
is prone to errors.

Note that only a single proxy instance exists for each component. If you need
to access components for multiple entities, obtain a new proxy after each
entity's operation is complete.

`typescript
// Bad
const proxy1 = FooProxy.get(eid1);
const proxy2 = FooProxy.get(eid2);

// This operation is wrong, proxy1 internally refers to eid2's component
proxy1.foo.operation();

proxy2.foo.operation();

// Good
const proxy1 = FooProxy.get(eid1);
proxy1.foo.operation();
const proxy2 = FooProxy.get(eid2);
proxy2.foo.operation();
`

TODO: This limitation can be an error prone because static type checking
can't detect the problem and it requires manual oversight.

$3

A system in Tiny Web Metaverse Client is just a function that takes IWorld
of bitECS.

`typescript
import { IWorld } from "bitecs";

export const fooSystem = (world: IWorld): void => {
...
};
`

Systems registered to App that is explained later are invoked once an
animation frame.

We highly recommend to use query of bitECS to access specific components.

`typescript
import { defineQuery, IWorld } from "bitecs";
import { FooComponent, FooProxy } from "../components/foo";

const fooQuery = defineQuery([FooComponent]);

export const fooSystem = (world: IWorld): void => {
fooQuery(world).forEach(eid => {
const foo = FooProxy.get(eid).foo;
foo.operation();
});
};
`

$3

Removing an entity is a tricky part in Tiny Web Metaverse Client because
we use Component Proxy style described above. Just removing an entity
removeEntity() of bitECS doesn't release proxy and component data.
It can cause memory leak and also may cause a problem when an entity
is recycled.

To resolve this problem, a special flow has been introduced.

First, write a system for a component that has proxy to release the
component data when the component is removed from an entity.

`typescript
// src/systems/clear_foo.ts

import {
defineQuery,
exitQuery,
IWorld
} from "bitecs";
import {
FooComponent,
FooProxy
} from "../components/foo";

const exitFooQuery = exitQuery(defineQuery([FooComponent]));

export const clearFooSystem = (world: IWorld): void => {
exitFooQuery(world).forEach(eid => {
const proxy = FooProxy.get(eid);
const foo = proxy.foo;
foo.close();
proxy.free();
});
};
`

When you remove the component from an entity, just use removeComponent()
of bitECS. The component data will be released when the releasing system
is called next time.

`typescript
import { IWorld, removeComponent } from "bitecs";
import { FooComponent } from "../components/foo";

const somewhereToRemove = (world: IWorld, eid: number): void => {
removeComponent(world, FooComponent, eid);
...
};
`

If you want to remove an entity, use built-in removeComponentsAndThenEntity()
utility function. It immediately removes all the components associated with
an entity, and then remove the entity after several animation frames. It allows
systems to release component data in that interval.

`typescript
import { IWorld } from "bitecs";
import { removeComponentsAndThenEntity } from "@tiny-web-metaverse/client/src";

const somewhereToRemove = (world: IWorld, eid: number): void => {
removeComponentsAndThenEntity(world, eid);
...
};
`

App

App in Client manages systems and calls registered systems once an animation
frame for each.

Framework user creates an App instance with canvas
and roomId in their user applications. roomId is an identifier for room.
Only clients in the same room can see and communicate each other. In the
constructor built-in entities and systems are created or registered.

App.start() starts an application. This is an example of a minimal user
application. (But nothing is rendered because no entity has been created to
which Three.js objects are assigned. How to assign Three.js objects is
explained later.)

`typescript
import { App } from "@tiny-web-metaverse/client/src";

const roomId = '1234';
const canvas = document.createElement('canvas');

const app = new App({ canvas, roomId });
document.body.appendChild(canvas);

app.start();
`

$3

App.registerSystem() is a method for registering a system to App. A
registered system is called in animation loop at specified timing.

The method takes system and orderPriority as arguments. system is
a function that takes IWorld of bitECS. orderPriority is an integer.

Registered systems are called in the animation loop in the order of the
orderPriority numbers. Note that the order in which systems with the same
orderPriority value are called is not specified.

$3

Systems are generally expected to be executed in the following order.

- Time: Get elapsed and delta time.
- EventHandling: Handling async events detected while ideling.
- Setup: Set up any resource at the beginning of an animation loop.
- BeforeMatricesUpdate: Update transforms (position/rotation/scale).
- MatricesUpdate: App updates scene graph matrices. See "Matrices update"
section for the details.
- BeforeRender: Operate anything that use updated matrices and that don't need
transform update. Or operate anything that should be done right before
rendering.
- Render: App renders the scene with Three.js WebGLRenderer.render()
- AfterRender: Operate anything that should be done right after rendering.
- PostProcess: Apply post-processing visual effects.
- TearDown: Operate anything that should be done at the end of an animation
loop, for example clearing event components.

We highly recommend to use predefined SystemOrder corresponsing to them to
specify systems execution order. The values are just integers so

`typescript
export const SystemOrder = Object.freeze({
Time: 0,
EventHandling: 100,
Setup: 200,
BeforeMatricesUpdate: 300,
MatricesUpdate: 400,
BeforeRender: 500,
Render: 600,
AfterRender: 700,
PostProcess: 800,
TearDown: 900
});
`

This is an example.

`typescript
import { App, SystemOrder } from "@tiny-web-metaverse/client/src";
import { barSystem } from "./systems/bar";
import { fooSystem } from "./systems/foo";

const roomId = '1234';
const canvas = document.createElement('canvas');

const app = new App({ canvas, roomId });
document.body.appendChild(canvas);

app.registerSystem(fooSystem, SystemOrder.BeforeMatricesUpdate);
// "+ 1" is to ensure that barSystem runs after fooSystem
app.registerSystem(barSystem, SystemOrder.BeforeMatricesUpdate + 1);

app.start();
`

$3

App.getSystemOrderPriority() returns the order priority of a registered
system. This function is useful to ensure to execure a system before or
after certain systems including built-in systems.

`typescript
import { interactSystem } from "@tiny-web-metaverse/client/src";

...

const priority = app.getSystemOrderPriority(interactSystem);
app.registerSystem(fooSystem, priority + 1);
`

Coroutine

Systems execution order is predictable in Tiny Web Metaverse Client. It makes
easier to control systems and improves the simplicity and maintainability.

Async/Await must not be used in systems to keep this policy. Instead, consider
to use Coroutine approach with JavaScript
generator function
and yield.
Some async/await operations may not be avoidable, for example calling async
functions in a third-party library. In that case, use built-in toGenerator()
utility function that allows to handle an async function as a generator function.

This is an example.

`typescript
// src/systems/load_foo.ts

import {
addComponent,
defineQuery,
enterQuery,
exitQuery,
IWorld,
removeComponent
} from "bitecs";
import { loadFooAsync } from "foo-lib";
import { toGenerator } from "@tiny-web-metaverse/client/src";
import {
FooComponent,
FooLoader,
FooLoaderProxy,
FooProxy
} from "../components/foo";

function* load(world: IWorld, eid: number): Generator {
const url = FooLoaderProxy.get(eid).url;
const foo = yield* toGenerator(loadFooAsync(url));
addComponent(world, FooComponent, eid);
FooProxy.get(eid).allocate(foo);
}

const loaderQuery = defineQuery([FooLoader]);
const enterLoaderQuery = enterQuery(loaderQuery);
const exitLoaderQuery = exitQuery(loaderQuery);

const generators = new Map();

export const loadFooSystem = (world: IWorld): void => {
enterLoaderQuery(world).forEach(eid => {
generators.set(eid, load(world, eid));
});

loaderQuery(world).forEach(eid => {
let done = false;
try {
if (generators.get(eid).next().done === true) {
done = true;
}
} catch (error) {
console.error(error);
done = true;
}
if (done) {
removeComponent(world, FooLoader, eid);
}
});

exitLoaderQuery(world).forEach(eid => {
generators.delete(eid);
FooLoader.get(eid).free();
});
};
`

Note that any state can change while waiting for the completion of a generator
function. You must not use any data fetched before a generator function starts,
after a generator function has completed.

`typescript
// Bad
const data: number = BarProxy.get(eid).bar.data;
yield* generatorFunction();
operate(data);

// Good
yield* generatorFunction();
const data: number = BarProxy.get(eid).bar.data;
operate(data);
`

Three.js stuffs

There are some limitations and restrictions for Three.js operations to
simplify and optimize.

$3

If you want to assign Three.js Object3Ds to an entity, use built-in
EntityObject3D component and its proxy. EntityObject3DProxy.allocate()
allocates a new Three.js Group, called EntityRootGroup, as root.

You can access root via EntityObject3DProxy.root. You can control
the transform of an entity's Object3D immediately after assigning
EntityObject3D.

`typescript
import { addComponent, IWorld } from "bitecs";
import {
EntityObject3D,
EntityObject3DProxy
} from "@tiny-web-metaverse/client/src";

const setupEntityObject3D = (world: IWorld, eid: number): void => {
addComponent(world, EntityObject3D, eid);
EntityObject3DProxy.get(eid).allocate();
const root = EntityObject3DProxy.get(eid).root;
root.position.set(0.0, 0.0, -2.0);
};
`

Call built-in addObject3D() utility function to add your Three.js Object3D
(eg: Mesh). You can call addObject3D() even before assiging
EntityObject3D component to an entity because the function assigns it
if the component is not assigned yet. Use built-in removeObject3D()
utility function to remove an Three.js Object3D from an entity.

When an Object3D is assigned to an entity, its transform must be identity
(identity matrix). Update the transform via EntityObject3DProxy.root after
assigning.

`typescript
import { IWorld } from "bitecs";
import { Mesh, MeshBasicMaterial, SphereGeometry } from "three";
import { addObject3D } from "@tiny-web-metaverse/client/src";

const addSphereMesh = (world: IWorld, eid: number): void => {
const geometry = new SphereGeometry(1.0);
const material = new MeshBasicMaterial();
const mesh = new Mesh(geometry, material);
addObject3D(world, mesh, eid);
EntityObject3DProxy.get(eid).root.position.set(0.0, 0.0, 2.0);
};
`

Multiple Object3Ds can be assigned to an entity. addObject3D()
and removeObject3D() form the following Three.js objects structure as
optimization. When swapping the root object, they keep the transform
(position/rotation/scale/matrix).

`
The number of assigned Object3Ds: 0

- EntityRootGroup (EntityObject3DProxy.root)

The number of assigned Object3Ds: 1

- Object3D (EntityObject3DProxy.root)

The number of assigned Object3Ds: 2-

- EntityRootGroup (EntityObject3DProxy.root)
- Object3D_A
- Object3D_B
...
`

root object can be swapped so it is a good practice to access
EntityObject3DProxy.root right before using it.

`typescript
// Bad
const root = EntityObject3DProxy.get(eid);
something(); // This function may add or remove Object3D from an entity
root.position.set(0.0, 0.0, 2.0);

// Good
something();
const root = EntityObject3DProxy.get(eid);
root.position.set(0.0, 0.0, 2.0);
`

TODO: Static type check can't detect misoperation like the bad one in the
above example code. Can we introduce a mechanism to avoid the problem?

TODO: Remove this optimization? It can be simpler.

$3

Add InScene built-in component to an entity to add its Three.js Object3Ds to
Three.js scene. A built-in system adds them to the scene. If InScene
component is removed from an entity, the system removes its Object3Ds from the
scene.

`typescript
import { addComponent, IWorld } from "bitecs";
import { Mesh, MeshBasicMaterial, SphereGeometry } from "three";
import { addObject3D, InScene } from "@tiny-web-metaverse/client/src";

const addSphereMesh = (world: IWorld, eid: number): void => {
const geometry = new SphereGeometry(1.0);
const material = new MeshBasicMaterial();
const mesh = new Mesh(geometry, material);
addObject3D(world, mesh, eid);
EntityObject3DProxy.get(eid).root.position.set(0.0, 0.0, 2.0);
addComponent(world, InScene, eid);
};
`

$3

In Tiny Web Metaverse Client, Three.js Scene is expected to have an identity
matrix and Object3Ds assigned to an entity are expected to be not the children
of other Object3Ds assigned to other entities for simplicity and optimization
as

- The scene graph can be kept shallow and scene graph matrices update cost
can be lower because a Three.js Object3D's transform update doesn't affect
many Object3Ds in the scene.
- The local transform of a EntityObject3D's root match its world transform
so even when world transform is needed world transform calculation is not
needed.

If an entity's Object3D wants to be as if a child of other entity's Object3D
you can manipulate the matrix like this.

`typescript
import { entityExists, hasComponent, IWorld } from "bitecs";
import {
EntityObject3D,
EntityObject3DProxy
} from "@tiny-web-metaverse/client/src";

const asIfChild = (world: IWorld, eid: number, parentEid: number): void => {
if (!entityExists(world, parentEid) ||
!hasComponent(world, EntityObject3D, parentEid) {
return;
}

const root = EntityObject3DProxy.get(eid).root;
const parent = EntityObject3DProxy.get(parentEid).root;

root.updateMatrix();
parent.updateMatrix();
root.matrix.premultiply(parent.matrix);
root.matrix.decompose(root.position, root.quaternion, root.scale);
};
`

TODO: Introduce a mechanism that allows entity's Object3D to act as if
a child of other entity's Object3D?

$3

App updates the entire scene graph matrices at SystemOrder.MatricesUpdate in
an animation loop. Systems that update transform(position/rotation/scale) should
run before it. And systemt that need updated matrices and don't update transform
should run after it for efficiency.

$3

Tiny Web Metaverse supports glTF 3D file
format. Use the built-in GltfLoader component to load a glTF file and
adds a loaded object to an entity.

TODO: Rename GltfLoader component to avoid the conflict name with
GLTFLoader in Three.js?

The built-in gltfLoad system downloads and parses a file specified with
GltfLoader component, creates Three.js objects, adds them to an entity,
and adds the built-in GltfRoot component that refers to the root Three.js
object of the loaded glTF objects.

`typescript
import {
addComponent,
addEntity,
IWorld
} from "bitecs";
import { AnimationMixer } from "three";
import {
EntityObject3D,
EntityObject3DProxy,
GltfLoader,
GltfLoaderProxy,
InScene,
MixerAnimation,
MixerAnimationProxy
} from "@tiny-web-metaverse/client/src";

const assetUrl = 'assets/models/foo.gltf';

export const FooPrefab = (world: IWorld): number => {
const eid = addEntity(world);
addComponent(world, InScene, eid);
addComponent(world, MixerAnimation, eid);
MixerAnimationProxy.get(eid).allocate(new AnimationMixer(null));

addComponent(world, EntityObject3D, eid);
EntityObject3DProxy.get(eid).allocate();

addComponent(world, GltfLoader, eid);
GltfLoaderProxy.get(eid).allocate(assetUrl);

return eid;
};
`

If you want to access glTF objects, you can use GltfRoot component and bitECS
query.

`typescript
import {
defineQuery,
enterQuery,
IWorld
} from "bitecs";
import { GltfRoot, GltfRootProxy } from "@tiny-web-metaverse/client/src";

const enterGltfQuery = enterQuery(defineQuery([GltfRoot]));

export const gltfSystem = (world: IWorld): void => {
enterGltfQuery(world).forEach(eid => {
const gltfRoot = GltfRootProxy.get(eid).root;
...
});
};
`

$3

While not yet officially documented, Three.js GLTFLoader offers an
extensibility mechanism through its plugin system.
Registered plugins will be called at specified stages; before, after, or during
parsing glTF content. The plugin system is usually used for handling unlatified,
vendor-specific, or custom glTF extensions.

In Tiny Web Metaverse, you can add such plugins by creating entities and
adding GltfLoaderPluginComponent component to them. The built-in gltfLoad
system registers the plugins to a Three.js GLTFLoader when loading a glTF file.

`typescript
import {
addComponent,
addEntity,
IWorld
} from "bitecs";
import { GLTFParser } from "three";
import {
GltfLoaderPluginComponent,
GltfLoaderPluginProxy
} from "@tiny-web-metaverse/client/src";

class FooPlugin {
private parser: GLTFParser;

constructor(parser: GLTFParser) {
this.parser = parser;
this.name = 'EXT_foo';
}

...
}

class BarPlugin {
private parser: GLTFParser;

constructor(parser: GLTFParser) {
this.parser = parser;
this.name = 'EXT_bar';
}

...
}

const addPlugins = (world: IWorld): void => {
const fooEid = addEntity(world);
addComponent(world, GltfLoaderPluginComponent, fooEid);
GltfLoaderPluginProxy.get(fooEid).allocate((parser: GLTFParser) => {
return new FooPlugin(parser);
});

const barEid = addEntity(world);
addComponent(world, GltfLoaderPluginComponent, barEid);
GltfLoaderPluginProxy.get(barEid).allocate((parser: GLTFParser) => {
return new BarPlugin(parser);
});
};
`

Event handling

As written above, we avoid async/await in systems for simplicity and predictable
execution order.

However, certain types of asynchronous event processing cannot be avoided, and
it is difficult to process them synchronously as-is. For example, input events
from input devices such as keyboard and mouse are generated by user operations,
so the timing of their occurrence cannot be predicted. Therefore, it is
generally common to listen for these asynchronous events and process them when
they occur. These asynchronous events typically occur during idle time.

Tiny Web Metaverse Client uses a trick to make these asynchronous events
processable synchronously within systems.

1. Add asynchronous events that occurred during idle time to a queue.
2. At the beginning of the next animation frame, a system notifies events to
entities that listen to that events by adding an event component to the
entities.
3. A system processes the entities that received the event notification.
4. At the end of the animation frame, a system deletes the event component.

Let's take a look at the details with some specific code examples.

First create an event and its listener components. The component holds events
as its component data.

`typescript
// src/components/foo.ts

export const FooEvent = defineComponent();

export const enum FooEventType {
Enter,
Leave
};

export type FooEventValue = {
type: FooEventType
};

export class FooEventProxy {
private static instance: FooEventProxy = new FooEventProxy();
private eid: number;
private map: Map;

private constructor() {
this.eid = NULL_EID;
this.map = new Map();
}

static get(eid: number): Foo {
FooEventProxy.instance.eid = eid;
return FooEventProxy.instance;
}

allocate(): void {
this.map.set(this.eid, []);
}

free(): void {
this.map.delete(this.eid);
}

get events(): FooEventValue[] {
return this.map.get(this.eid)!;
}
}

export const FooEventListener = defineComponent();
`

And create a system that stores async events happened during the idle time to
a queue and notifies the events to entities that listen to that event by adding
the event component in an animation loop.

Also create a system that removes the event component.

`typescript
// src/systems/foo.ts

import {
addComponent,
defineQuery,
enterQuery,
hasComponent,
IWorld
} from "bitecs";
import {
FooEvent,
FooEventListener,
FooEventProxy,
FooEventType
} from "../components/foo";
import { NullComponent } from "../components/null";

const eventQueue: { type: FooEventType }[] = [];

// enterQuery + NullComponent is a hack for executing only at the first call
const initializeQuery = enterQuery(defineQuery([NullComponent]));

const listenerQuery = defineQuery([FooEventListener]);
const eventQuery = defineQuery([FooEvent]);

export const fooEventNotificationSystem = (world: IWorld): void => {
initializeQuery(world).forEach(() => {
window.addEventListener('enterfoo', () => {
eventQueue.push({ type: FooEventType.Enter });
});

window.addEventListener('leavefoo', () => {
eventQueue.push({ type: FooEventType.Leave });
});
});

for (const e of eventQueue) {
listenerQuery(world).forEach(eid => {
if (!hasComponent(world, FooEvent, eid)) {
addComponent(world, FooEvent, eid);
FooEventProxy.get(eid).allocate();
}
FooEventProxy.get(eid).push({ type: e.type });
});
}

eventQueue.length = 0;
};

export const clearFooEventSystem = (world: IWorld): void => {
eventQuery(world).forEach(eid => {
const proxy = FooEventProxy.get(eid);
proxy.events.length = 0;
proxy.free();
removeComponent(world, FooEvent, eid);
});
};
`

And then write a system that fetches entities that have the event component
and processes with the events.

`typescript
export const fooEventSystem = (world: IWorld): void => {
eventQuery(world).forEach(eid => {
for (const e of FooEventProxy.get(eid).events) {
...
}
});
};
`

Finally, register the systems to App. We recommend SystemOrder.EventHandling
for systems that notify events to entities and SystemOrder.TearDown for
systems that remove event components. And systems that process with events need
to run between them.

`typescript
app.registerSystem(fooEventNotificationSyste, SystemOrder.EventHandling);
app.registerSystem(fooEventSystem, SystemOrder.BeforeMatricesUpdate);
app.registerSystem(clearFooEventSystem, SystemOrder.TearDown);
`

Event handling systems are ready. What you have to do last is to add a listener
component to entities that want to know events occurrence.

`typescript
const eid = addEntity(world);
addComponent(world, EventListener, eid);
`

$3

Different input devices, such as mice, touch panels, and VR controllers, have
a variety of triggers, such as clicks, touches, and button presses. In some
cases, we may want to perform the same processing for triggers from different
devices. For example, we may want to process the left click of a mouse, the
single-finger touch of a touch panel, and the button press of a VR controller
events with the same processing. However, if we write the same processing for
each of these input events, it will lead to duplicate code and poor
maintainability.

Tiny Web Metaverse provides an abstract concept called InputSource to unify
multiple types of triggers. We can reduce duplicate code by writing processing
for InputSource events as follows:

`typescript
import {
defineQuery,
IWorld
} from "bitecs";
import {
FirstSourceInteractionTriggerEvent
} from "@tiny-web-metaverse/client/src";

const triggerQuery = defineQuery([FirstSourceInteractionTriggerEvent]);

export const fooSystem = (world: IWorld): void => {
triggerQuery(world).forEach(eid => {
...
});
};
`

For example, a mouse often has two buttons, left and right. There are multiple
InputSources, such as FirstSource, SecondSource, and so on, to
distinguish between different types of triggers.

Which trigger should be assigned to which InputSource is
application-dependent, so users must specify it in their own applications by
writing systems like the following:

`typeScript
import {
addComponent,
defineQuery,
IWorld
} from "bitecs";
import {
FirstSourceInteractable,
FirstSourceInteractionLeaveEvent,
FirstSourceInteractionTriggerEvent,
MouseButtonEvent,
MouseButtonEventProxy,
MouseButtonEventType,
MouseButtonType,
SecondSourceInteractable,
SecondSourceInteractionLeaveEvent,
SecondSourceInteractionTriggerEvent
} from "@tiny-web-metaverse/client/src";

const firstSourceEventQuery = defineQuery([FirstSourceInteractable, MouseButtonEvent]);
const secondSourceEventQuery = defineQuery([SecondSourceInteractable, MouseButtonEvent]);

export const mouseInteractionTriggerSystem = (world: IWorld) => {
firstSourceEventQuery(world).forEach(eid => {
for (const e of MouseButtonEventProxy.get(eid).events) {
if (e.button !== MouseButtonType.Left) {
continue;
}
if (e.type === MouseButtonEventType.Down) {
addComponent(world, FirstSourceInteractionTriggerEvent, eid);
} else if (e.type === MouseButtonEventType.Up) {
addComponent(world, FirstSourceInteractionLeaveEvent, eid);
}
}
});

secondSourceEventQuery(world).forEach(eid => {
for (const e of MouseButtonEventProxy.get(eid).events) {
if (e.button !== MouseButtonType.Right) {
continue;
}
if (e.type === MouseButtonEventType.Down) {
addComponent(world, SecondSourceInteractionTriggerEvent, eid);
} else if (e.type === MouseButtonEventType.Up) {
addComponent(world, SecondSourceInteractionLeaveEvent, eid);
}
}
});
};
`

Systems that perform the most typical assignments is provided in
Addons. For more details, see
packages/addons/src/systems/*_interaction.ts. The assignment provided
by Addons is as follows:

| Trigger | InputSource |
| ---- | ---- |
| Mouse left click | FirstSource |
| Mouse right click | SecondSource |
| Touch | FirstSource |
| VR First controller | FirstSource |
| VR Second controller | SecondSource |

TODO: Should VR controllers be distinguished by Left/Right controller instead
of First/Second controller?

2D UI

As mentioned above, Client core doesn't take care of 2D UI controls that
overlaps 3D canvas. It is the responsibility of user-application (or addons);

If you want HTML DOM elements
to interact entities or components, the implementation would be similar to
EventHandling described above, like storing events in idle time and processing
with them in animation loop. This is an example of system code.

`typescript
// src/systems/ui_button.ts

import { IWorld } from "bitecs";
import { FooComponent } from "../components/foo";
import { NullComponent } from "../components/null";

const enum ButtonEventType {
Clicked
};

const eventQueue: { type: ButtonEventType }[] = [];

const button = document.createElement('button');
button.innerText = 'Click';
button.style.buttom = '10px';
button.style.left = '50%';
button.style.position = 'absolute';
button.style.transform = 'translate(-50%)';
button.style.zIndex = '1000';

button.addEventListener(() => {
eventQueue.push({ type: ButtonEventType.Clicked });
});

// enterQuery + NullComponent is a hack for executing only at the first call
const initializeQuery = enterQuery(defineQuery([NullComponent]));
const fooQuery = defineQuery([FooComponent]);

export const buttonUISystem = (world: IWorld): void => {
initializeQuery(world).forEach(() => {
document.body.appendChild(button);
});

for (const e of eventQueue) {
fooQuery(world).forEach(eid => {
...
});
}

eventQueue.length = 0;
};
`

Stream server connection

Tiny Web Metaverse Client connects to the Stream server
for voice communication with remote clients.

For starting the voice communication, there are two steps required. First
connect to the Stream server and then join a room.

To connect to the Stream server, create an entity and add the built-in
StreamConnectRequestor component to it. A built-in system will send a
connection request to the Stream server.

To detect the connection, use the built-in ConnectedStreamEventListener
component and observe the built-in StreamMessageType.Connected event.

To join a room, create an entity and add the built-in StreamJoinRequestor
component to it. A built-in system will send a join request to the Stream
server.

To detect the join, use the built-in JoinedStreamEventListener
component and observe the built-in StreamMessageType.Joined event.

This is an example to connect and join the stream server.

`typescript
// src/systems/stream_event.ts

import {
addEntity,
defineQuery,
enterQuery,
IWorld
} from "bitecs";
import {
ConnectedStreamEventListener,
JoinedStreamEventListener,
NullComponent,
StreamConnectRequestor,
StreamEvent,
StreamEventProxy,
StreamMessageType,
StreamJoinRequestor
} from "@tiny-web-metaverse/client/src";

const initialQuery = enterQuery(defineQuery([NullComponent]));
const eventQuery = defineQuery([StreamEvent]);

export const streamEventSystem = (world: IWorld): void => {
initialQuery(world).forEach(() => {
// Send a connection request in the first call.
addComponent(world, StreamConnectRequestor, addEntity(world));
});

eventQuery(world).forEach(eid => {
for (const event of StreamEventProxy.get(eid).events) {
switch (event.type) {
case StreamMessageType.Connected:
// Connected. Send a join request.
addComponent(world, StreamJoinRequestor, addEntity(world));
break;
case StreamMessageType.Joined:
// Joined. Ready to start the voice conversation.
...
break;
}
}
});
};

// src/app.ts

import {
addComponent,
addEntity,
IWorld
} from "bitecs";
import {
ConnectedStreamEventListener,
JoinedStreamEventListener
} from "@tiny-web-metaverse/client/src";

// App initialization

...

// Create an entity for listening stream events to mange stream server
// connection
const eid = addEntity(world);
addComponent(world, ConnectedStreamEventListener, eid);
addComponent(world, JoinedStreamEventListener, eid);

...

`

State server connection

Tiny Web Metaverse Client connects to the State server
and synchronizes the state of entities state with remote clients through the
state server.

App automatically connects to the State server in the constructor.

TODO: Instead of automatically connecting, connect to the state server when
requested similar to the stream server?

Network sync

In Tiny Web Metaverse, the state of Entities can be selectively synchronized
with remote clients in the same room. Here, the state of an Entities refers to
the entity existence and the specified component data.

A synchronized Entity is called a Networked Entity, and its synchronized
components are called Networked Components.

Some built-in network systems perform network synchronization processing.

The network send system periodically checks for changes to the data in the
networked component. Only when changes are detected the system sends the
updated data to the remote client via the server. This helps to prevent network
data flooding.

The network receive system observes the network data sent from the remote
client. When data is received, the system reflects the data to the networked
components. Because data is only sent periodically, the system may interpolate
the data before reflecting it.

Network entities are bound to a client that created them. If a client leaves
the room, the network entities created by the client will be removed.

You need to do the following steps to create networked components and entities.

- Define networked components
- Write and register serializers/deserializers
- Write and register prefabs
- Create networked entity

Let's take a loot at one by one.

$3

There are three networked types.

- Local: Local networked components/entities should be directly controlled only
by local client. Changes made to Local components/entities are sent to other
clients by built-in network send system. On remote clients, Local
components/entities appear as Remote.
- Remote: Remote networked components/entities should be directly controlled
only by a remote client that created them. Updates to remote entities are
received and applied to corresponding components/entities by built-in network
receive system. Remote components/entities appear as Local on a remote client
that created them and as Remote on other clients.
on their creator client and as Remote on other clients.
- Shared: Shared networked components/entities are controlled by any client.
Built-in network send and receive systems make them synced. Shared
components/entities appear as Shared on all clients.

$3

First you need to create a component that indicates another component is
networked. For instance, if you want to make FooComponent component
network-enabled, create NetworkedFooComponent. When both FooComponent and
NetworkedFooComponent are added to an entity, FooComponent becomes
networked. If only FooComponent is added, it remains non-networked.

`typescript
// src/components/foo.ts

import { defineComponent, Types } from "bitecs";

export const FooComponent = defineComponent({
data: Types.f32
});
export const NetworkedFooComponent = defineComponent();

// Refer to the following "Serializer" section about this component
export const FooComponentInterpolation = defineComponent({
target: Types.f32
});
`

$3

Next, you have to write diff checkers, serializers, and deserializers for
networked components.

Diff checker is a function periodically called to check for changes of
networked component data.

Serializer is a function that serializes networked component data to send.
Deserializer is a function that deserializes serialized network component
data and reflects to component data.

You can define two deserializer types, one with interpolation and another
one for without interpolation. The non-interpolation one is used for component
data initialization and the other one is used for others.

In general, interpolation would be like

Normally, interpolation is a process of gradually approaching the target value.
Such processing should be done in a system, and you need to write a system.
In the deserializer, you will need to add a component to drive the system.

`typescript
// src/serializers/position.ts

import {
addComponent,
hasComponent
} from "bitecs";
import {
FooComponent,
FooComponentInterpolation,
NetworkedFooComponent
} from "../components/foo";

const EPSILON = 0.0001;
export type SerializedFoo = { data: number };

const checkFooDiff = (world: IWorld, eid: number, cache: SerializedFoo): boolean => {
if (!hasComponent(world, FooComponent, eid)) {
throw new Error('checkFooDiff requires FooComponent component.');
}
return Math.abs(cache.data - FooComponent.data[eid]) > EPSILON;
};

const serializeFoo = (world: IWorld, eid: number): SerializedFoo => {
if (!hasComponent(world, FooComponent, eid)) {
throw new Error('serializeFoo requires FooComponent component.');
}
return { data: FooComponent.data[eid] };
};

const deserializeFoo = (world: IWorld, eid: number, data: SerializedFoo): void => {
if (!hasComponent(world, FooComponent, eid)) {
throw new Error('deserializeFoo requires FooComponent component.');
}
FooComponent.data[eid] = data.data;
};

const networkDeserializeFoo = (world: IWorld, eid: number, data: SerializedFoo): void => {
if (!hasComponent(world, FooComponent, eid)) {
throw new Error('networkDeserializeFoo requires FooComponent component.');
}
// Add a component to drive a interpolation system.
// Interpolation system implementation is omitted in this example.
addComponent(world, FooComponentInterpolation, eid);
FooComponentInterpolation.target[eid] = data.data;
};

export const fooSerializers = {
deserializer: deserializeFoo,
diffChecker: checkFooDiff,
networkDeserializer: deserializeFoo,
serializer: serializeFoo
};
`

And then you have to register the functions with the built-in
registerSerializers() function to establish a mapping between the functions
and the networked component.

The second argument is a unique key string within the application that
identifies this mapping. This key is used in built-in network systems.

The third and fourth arguments specify a networked component and functions
for mapping. These functions must be contained within an object that provides
the diffChecker, serializer, deserializer, and networkDeserializer.
Deserializer function without interpolation should be passed as
deserializer, while deserializer function with interpolation should be
passed as networkDeserializer.

`typescript
import { registerSerializers } from "@tiny-web-metaverse/client/src";
import { NetworkedFoo } from "../components/networked_position";
import { fooSerializers } from "../serializers/foo";

registerSerializers(world, 'foo', NetworkedFoo, fooSerializers);
`

$3

Next, you have to write a prefab. Prefab is a function that takes bitECS world
and an optional parameter, and creates an entity with preset components. Prefab
may be said an entity template function.

A networked entity is created from a prefab. Networked components for a
networked entity must be set up in a prefab.

`typescript
// src/prefabs/foo.ts

import {
addComponent,
addEntity,
IWorld
} from "bitecs";
import { FooComponent, NetworkedFoo } from "../components/foo";

export const FooPrefab = (world: IWorld, params: { data: number }): number => {
const eid = addEntity(world);

addComponent(world, FooComponent, eid);
FooComponent.data[eid] = params.data;
addComponent(world, NetworkedFoo, eid);

return eid;
};
`

Similar to serializers, prefab have to be registered with the built-in
registerPrefab() function.

The second argument is a unique key string within the application that
identifies this prefab. This key is used when creating a networked entity.

The third argument specifies a prefab to register.

`typescript
import { registerPrefab } from "@tiny-web-metaverse/client/src";
import { NetworkedFoo } from "../components/networked_position";
import { fooSerializers } from "../serializers/foo";

registerPrefab(world, 'foo', FooPrefab);
`

$3

The set up for networked entity creation has been done. The last thing you
have to do is create networked entities.

You can create a networked entity with the built-in createNetworkedEntity().

The second argument specifies the network type with the built-in NetworkedType
enum, Local or Shared. Remote entities creation is fired from a remote
client so they are created in built-in network systems.
createNetworkedEntity() is only for Local or Shared networked entities.

The third argument is a registered prefab key used to create a networked
entity from.

`typescript
import {
createNetworkedEntity,
NetworkedType
} from "@tiny-web-metaverse/client/src";

const eid = createNetworkedEntity(world, NetworkedType.Local, 'foo');
`

Networked entities have the built-in Networked component. And also they have
either Local, Remote, or Shared built-in component corresponding to
network type.

Audio processing

T.B.D.

$3

T.B.D.

$3

T.B.D.

Other utilities

$3

App first creates an entity with the built-in NullComponent component in
the constructor. This first entity itself should not be used for any processing.

However, its Entity ID (which should be zero) can be used to indicate that the
entity does not exist. For instance, you could write a function that searches
for entities with a specific component and returns their Entity ID. If no such
entity is found, it returns zero. There is a built-in variable, NULL_EID, which
represents a nonexistent Entity ID.

`typescript
import {
Component,
defineQuery
IWorld,
removeQuery
} from "bitecs";
import { NULL_EID } from "@tiny-web-metaverse/client/src";
import { FooComponent } from "../components/foo";

const searchAnyEntity = (world: IWorld, c: Component): number => {
const query = defineQuery([c]);
const eids = query(world);
removeQuery(world, query);
return eids.length > 0 ? eids[0] : NULL_EID;
};

const func = (world: IWorld): void => {
const eid = searchAnyEntity(world, FooComponent);
if (eid !== NULL_EID) {
// If found
} else {
// If not found
}
};
`

And built-in NullComponent component should be added only to that world-first
entity. This NullComponent can be used for letting a system process something
only in the first call by using bitECS defineQuery() and enterQuery().

`typescript
import {
defineQuery,
enterQuery,
IWorld
} from "bitecs";
import { NullComponent } from "@tiny-web-metaverse/client/src";

const initializeQuery = enterQuery(defineQuery([NullComponent]));

export const barSystem = (world: IWorld): void => {
initializeQuery(world).forEach(() => {
// This code is only executed once, the first time the system is called.
});
};
`

User app examples

So far, you have learned the Client core concepts necessary to create your own
Client (User app). You can create your own Client on top of the Client Core by
adding Entities, Components, Systems, Prefabs, Serializers, UI, and so on. You
can also import the Readme of addons package or create and publish
addons for others to reuse.

A good starting point is to create a Client that displays avatars.

If you need more practical examples, see the examples package.

$3

This is a very basic example to handle networked avatars.

- An avatar appears as a sphere object in the scene
- Local avatar is moved with the arrow keys
- Remote avatars shown moved by remote clients

`typescript
// src/prefabs/avatar.ts

import {
addComponent,
addEntity,
IWorld
} from "bitecs";
import { Mesh, MeshBasicMaterial, SphereGeometry } from "three";
import {
addObject3D,
Avatar,
InScene,
NetworkedPosition,
NetworkedQuaternion,
NetworkedScale
} from "@tiny-web-metaverse/client/src";

export const AvatarPrefab = (world: IWorld): number => {
const eid = addEntity(world);

addComponent(world, Avatar, eid);
addComponent(world, NetworkedPosition, eid);
addComponent(world, NetworkedQuaternion, eid);
addComponent(world, NetworkedScale, eid);
addComponent(world, InScene, eid);

const avatarObject = new Mesh(
new SphereGeometry(0.25),
new MeshBasicMaterial({ color: 0xaaaacc })
);
addObject3D(world, avatarObject, eid);

return eid;
};

// src/systems/avatar_key_controls.ts

import {
addComponent,
defineQuery,
IWorld,
removeComponent
} from "bitecs";
import {
Avatar,
KeyEvent,
KeyEventProxy,
KeyEventType,
LinearMoveBackward,
LinearMoveForward,
LinearMoveLeft,
LinearMoveRight,
Local
} from "@tiny-web-metaverse/client/src";

const eventQuery = defineQuery([Avatar, KeyEvent, Local]);

export const avatarKeyControlsSystem = (world: IWorld): void => {
eventQuery(world).forEach(eid => {
const speed = 1.0;
for (const e of KeyEventProxy.get(eid).events) {
if (e.code === 37) { // Left
if (e.type === KeyEventType.Down) {
addComponent(world, LinearMoveLeft, eid);
LinearMoveLeft.speed[eid] = speed;
} else if (e.type === KeyEventType.Up) {
removeComponent(world, LinearMoveLeft, eid);
}
} else if (e.code === 38) { // Up
if (e.type === KeyEventType.Down) {
addComponent(world, LinearMoveForward, eid);
LinearMoveForward.speed[eid] = speed;
} else if (e.type === KeyEventType.Up) {
removeComponent(world, LinearMoveForward, eid);
}
} else if (e.code === 39) { // Right
if (e.type === KeyEventType.Down) {
addComponent(world, LinearMoveRight, eid);
LinearMoveRight.speed[eid] = speed;
} else if (e.type === KeyEventType.Up) {
removeComponent(world, LinearMoveRight, eid);
}
} else if (e.code === 40) { // Down
if (e.type === KeyEventType.Down) {
addComponent(world, LinearMoveBackward, eid);
LinearMoveBackward.speed[eid] = speed;
} else if (e.type === KeyEventType.Up) {
removeComponent(world, LinearMoveBackward, eid);
}
}
}
});
};

// src/app.ts

import { addComponent, addEntity } from "bitecs";
import {
App,
AudioDestination,
createNetworkedEntity,
KeyEventListener,
registerPrefab,
SystemOrder
} from "@tiny-web-metaverse/client/src";
import { AvatarPrefab } from "./prefabs/avatar";
import { avatarKeyControlsSystem } from "./systems/avatar_key_controls";

const roomId = '1234';
const canvas = document.createElement('canvas');

const app = new App({ canvas, roomId });
document.body.appendChild(canvas);

app.registerSystem(avatarKeyControls, SystemOrder.BeforeMatricesUpdate);

const world = app.getWorld();

registerPrefab(world, 'avatar', AvatarPrefab);

const avatarEid = createNetworkedEntity(world, NetworkedType.Local, 'avatar');
EntityObject3DProxy.get(avatarEid).root.position.set(0.0, 0.75, 2.0);
addComponent(world, KeyEventListener, avatarEid);
addComponent(world, AudioDestination, avatarEid);

app.start();
`

$3

This example is based on the above example. Instead of implementing a custom
avatar key controls system, it imports controls addons and set them up.

`typescript
// src/app.ts

import { addComponent, addEntity } from "bitecs";
import { avatarKeyControlsSystem } from "@tiny-web-metaverse/addons/src";
import {
App,
AudioDestination,
createNetworkedEntity,
KeyEventListener,
registerPrefab,
SystemOrder
} from "@tiny-web-metaverse/client/src";
import { AvatarPrefab } from "./prefabs/avatar";

const roomId = '1234';
const canvas = document.createElement('canvas');

const app = new App({ canvas, roomId });
document.body.appendChild(canvas);

app.registerSystem(avatarKeyControls, SystemOrder.BeforeMatricesUpdate);

const world = app.getWorld();

registerPrefab(world, 'avatar', AvatarPrefab);

const avatarEid = createNetworkedEntity(world, NetworkedType.Local, 'avatar');
EntityObject3DProxy.get(avatarEid).root.position.set(0.0, 0.75, 2.0);
addComponent(world, KeyEventListener, avatarEid);
addComponent(world, AudioDestination, avatarEid);

app.start();
``

Built-in components

T.B.D.

Network sync internal

T.B.D.