Augmenta OSCV3 protocol

👋

This protocol has been designed with those aspects in mind

Being easier to use in node based, no-code creative software and limited environment

Being more robust to packet lost (no need to have a timeout to clear objects if no leave message was received)

Better human readability

Limitations

There is one drawback compared to OSCv2

By splitting the data over more messages and using more human-readable paths means that this new protocol has more overhead, thus using more bandwidth. The options has been added to mitigate this but keep in mind that this protocol is not aimed at delivering the maximum amount of performance.

Coordinate system

💡

This is the internal coordinate system of Augmenta, the origin and axis are freely tunable when sending data, to match the target coordinate system.

Origin : Bottom-left of detected plane

Axis : left-handed, Y up

System : metric

Protocol

The Augmenta system will send 2 types of bundles to clients:

object updates

scene infos

The object udpates bundle(s) is sent at each frame. It contains information on all the objects in the scene associated with a frame number.

The scene infos bundle contains information about the scene (its size, its position etc.), it is sent at a fixed frequency (default to 2Hz in Augmenta, can be adjusted).

Object updates bundle

Object updates are packed into bundles. Each bundle includes a frame id header as well as the number and list of active objects for this frame at the end.

Bundle splitting

You can choose to receive split object update bundles with a maximum size of 1432 bytes. This can be useful if you encounter network issues due to packets being too big. In this case, multiple object update bundles belonging to the same frame will have:

the same frame id header

the same number of objects and list of objects present in the scene at the end of each bundle

Values configuration

You can choose between relative or normalized arguments. The relative arguments are in meters, relative to the scene origin. The normalized values are always between 0 and 1, metric values can be computed by dividing the positions/dimensions by the scene size.

You can also toggle an option to have split coordinates. If it is enabled, you received one message for x, one message for y and on message for z. Otherwise all 3 arguments will be sent in one message.

Types notation

i -> int, f -> float, s -> string, if not specified, type is float

plain text
# Objects update header
/au/frame/id i                    // frame id (starts at 0, resets at MAX_INT32)

# Objects data
## Mandatory args

### When "split coordinate" option is unchecked
/au/N/uid i                       // unique ID ex : starts at 0, e.g. 42nd object to enter scene has uid=41 (resets at MAX_INT32)
/au/N/pos x y z                   // centroid position 3d coordinates

### if "split coordinate" option is checked
/au/N/uid i                       // unique ID ex : starts at 0, e.g. 42nd object to enter scene has uid=41 (resets at MAX_INT32)
/au/N/pos/x x                     // centroid X coordinate
/au/N/pos/y y                     // centroid Y coordinate
/au/N/pos/z z                     // centroid Z coordinate

## Optional args

/au/N/speed/val f                 // magnitude of the velocity vector in m/s, always an absolute value
/au/N/presence f                  // 0 means the point just arrived or left, 1 means the point is fully in the scene. It can also have any intermediate value depending on the state of the object.
/au/N/state s                     // state of the object ("in" -> object just entered, "out" -> object is leaving, "up" -> object is still in the scene)
/au/N/age i                       // time alive in frame
/au/N/box/rot r                   // bounding box rotation angle in degrees (counter-clockwise around normal axis from floor plane, 0° aligned with Augmenta's X axis)

### If "split coordinate" option is unchecked
/au/N/box/pos x y z                // x y z : bounding box centre coordinates
/au/N/box/size x y z               // x y z : bounding box size alongside x y z axis
/au/N/highest/pos x y z            // position of the highest point
/au/N/speed/dir x y z              // normalized velocity vector

### if "split coordinate" option is checked
/au/N/box/pos/x x
/au/N/box/pos/y y
/au/N/box/pos/z z
/au/N/box/size/x x
/au/N/box/size/y y
/au/N/box/size/z z
/au/N/highest/pos/x x
/au/N/highest/pos/y y
/au/N/highest/pos/z z
/au/N/speed/dir/x x
/au/N/speed/dir/y y
/au/N/speed/dir/z z

## Bundle end
/au/frame/numObj i                // number of objects
/au/frame/objList i...            // list of objects slotID in the scene

where N is the object id (also called slot id), starting at index 1.

More about the slot ID algorithm

The SlotID algorithm puts new incoming objects in the first free "slot" of a an array. When a registered object leaves, it frees the slot. The SlotID algorithm essentially gives an ID for the whole lifetime of an object in a known range. The maximum number of slots can be set by the user.

This is useful in 2 ways:

Number of slots can be limited to accommodate for limited processing environments (that can only handle a fixed number of objects)

It can be used to assign a "role" to each object that will not change during its lifetime

Below is an example of the SlotID algorithm in action.

Here is a scenario example

Empty scene

A new object with UID 5 enters

A new object with UID 6 enters

A new object with UID 7 enters

The object with UID 6 leaves

A new object with UID 8 enters

The object with UID 5 leaves

The object with UID 8 leaves

The object with UID 7 leaves

The result of the SlotID algorithm for the scenario above is displayed in the table below:

Each row is the state of the slots in each frame.

SlotID:	1	2	3	4	5
frame 0	-	-	-	-	-
frame 1	5	-	-	-	-
frame 2	5	6	-	-	-
frame 3	5	6	7	-	-
frame 4	5	-	7	-	-
frame 5	5	8	7	-	-
frame 6	-	8	7	-	-
frame 7	-	-	7	-	-
frame 8	-	-	-	-	-

Scene information bundle

plain text
# Sent at a fixed frequency
/au/scene/maxNumObj i             // 0 for unlimited, >1 for the number of slots available (the max number of slots)
/au/scene/size w h d              // scene size

# Optional scene infos
/au/scene/video/res w h           // resolution in pixels

// Scene transformation
/au/scene/pos x y z               // scene center 3d coordinate
/au/scene/rot rx ry rz            // Euler rotation of the scene

If provided, you can use the scene position and rotation to convert the relative coordinates of the objects to their absolute position in the Augmenta system. This way, changing the scene transform in the Augmenta configuration will also move it in the content.

Example sequence

In this example normalized data are used.

plain text
# server sends a clearing pattern when (re)connecting
{
/au/scene/frame  0
/au/scene/numObj 0
/au/scene/objList
}

// scene infos bundle sent when (re)connecting
{
/au/scene/pos     0 0 0           //scene info
/au/scene/size    10 15 10         //scene info
}

...   // a lot of frames are received

# objects update bundle
{
/au/frame/id    21375

/au/1/uid         13
/au/1/pos         0.21 0.08 0.1
/au/1/highest/pos 0.23 0.09 0.178  // normalized with 10 in sceneDepth
/au/1/speed/dir   -0.74 0.67 0.0
/au/1/speed/val   1.114
/au/1/box/pos     0.21 0.08 0.089
/au/1/box/rot     50.3
/au/1/presence    0.8
/au/1/state       up

/au/2/uid         34
/au/2/pos         0.95 0.2 0.08
/au/2/highest/pos 0.94 0.21 0.16
/au/2/speed/dir   0.52 -0.4 0.0
/au/2/speed/val   0.656
/au/2/box/pos     0.95 0.19 0.08
/au/2/box/rot     110
/au/2/presence    0.0
/au/2/state       out

/au/4/uid         512
/au/4/pos         0.53 0.1 0.4
... // other infos
/au/4/presence    0.1
/au/4/state       in

/au/frame/numObj  4
/au/frame/objList 13 34 512 147 // objects list, so that they can be cleared when not present
}

# second bundle for this update
{
/au/frame/id   21375

/au/5/uid      147
/au/5/pos      0.5 0.5 0.5
...
/au/5/presence 1.0
/au/5/state    up

/au/frame/numObj   4
/au/frame/objList  13 34 512 147
}

💡

Notes for client applications developers

If you can't script object addition or deletion, you can use the presence value as an alpha value of an asset that would represent the user (1.0 when an object is correctly detected, 0.0 when it disappears, alpha value can vary [0,1] depending on the detection quality).

When it is possible to script or code, /frame/objList message can be used to remove all objects that are not part of this list. This is a global state of all currently active objects. This information is transmitted with each frame, so if one of the object is lost, the information is still in the next frame.

Other server side options

It is possible to change the sending addresses. For example, you can change /au/N/pos to /au/N/position.