General event parameters

Each event has the follow standard parameters:

Parameter Type Description
event_name String The name of the event (such as "action.speech")
event_id String
event_sender String The name of the module that sent the event
event_time String A timestamp when the event was created

monitor.system.start

Reported when the system has started (as issued by system.sendStartSignal()).

Parameter Type Description
system String The name of the system

monitor.module.start

Reported by a module when it starts.

Parameter Type Description
system String The name of the system

monitor.module.state

Reported by a module when it changes state (for visualization and logging purposes)

Parameter Type Description
states List A list of state names

action.skill

Event that a skill can call to change to another skill. Reported by a module when it changes state (for visualization and logging purposes)

Parameter Type Description
entry String The entrypoint of the skill

action.skill.list

Event that a skill can call to get a list of all skills available on the system. Returns an monitor.skill.list event with a list.

monitor.skill.list

Event returned from action.skill.list events.

Parameter Type Description
skills ArrayList<Record> A list of skill entrypoints for matched skills.

Note: a skill that has multiple entrypoints (for example a flow with two or more public states) will show up several times in the list

Each skill entrypoint record has the following parameters:

Parameter Type Description
entry String The entry point of the skill
name String The name of the skill (as returned by the getName() method)

action.skill.query

Event that a skill can call to query all other skills that implement the Queryable interface for a text. Returns an monitor.skill.query event with an answer and/or a list of matched skills.

Parameter Type Description
text String The query text

monitor.skill.query

Event returned from action.skill.query events. Contains a QueryResponse object

Parameter Type Description
status String "NO_MATCH" or "MATCH" depending on if any skills match the query
skills ArrayList<Record> The lists of skills matching the query, sorted by confidence score. Including any answers given.
answer (optional) String The answer with the highest confidence value from any queryable skill's query() method.

Each skill in the skills ArrayList has the following parameters:

Parameter Type Description
entry String The entry point of the skill
name String The name of the skill (as returned by the getName() method)
confidence Double The confidence of an answer (0 < confidence <= 1)
sem Record Semantics returned from the query, to be used as an initial state when moving to a new skill
answer String A text answer on the query

Speech synthesis

action.speech

An action that can be sent to a synthesizer to add an utterance to the speech queue

Parameter Type Description
text String The text to speak
start int The number of milliseconds into the utterance that the synthesis
audio String An audio file to play instead of synthesizing
display String A nicely formatted text representation for display purposes
agent String The ID of the agent that is associated with this synthesis. If this is omitted, all synthesizers should start
ifsilent Boolean Only add utterance if the system is silent. (Default is false)
abort Boolean Whether to abort the current speech queue. (Default is false, i.e., append the utterance).
monitorWords Boolean Whether to send monitor.speech.word events before each word (Default is false)

action.speech.stop

An action that can be sent to a synthesizer to stop speaking (and clearing the speech queue)

Parameter Type Description
action String The ID of the action.speech event that started the speech synthesis. If this is omitted, all synthesizers should stop

monitor.speech.start

Monitors that a speech synthesizer has started an utterance

Parameter Type Description
action String The ID of the action.speech event that started the speech synthesis
text String Taken from action.speech
start int Taken from action.speech
agent String Taken from action.speech
length int The length of the utterance (in msec)
prominence int The position in the utterance (in msec) where the prominence is located

monitor.speech.end

Monitors that a speech synthesizer has completed an utterance

Parameter Type Description
action String The ID of the action.speech event that started the speech synthesis
stopped int If the speech synthesis is stopped prematurely, this parameter reports the position in the utterance (in msec) where it stopped
agent String The ID of the agent that is associated with this synthesis

monitor.speech.done

Monitors that a speech synthesizer has completed an utterance and the speech queue is empty

Parameter Type Description
agent String The ID of the agent that is associated with this synthesis

monitor.speech.prominence

Reported by the synthesizer when prominence in the utterance is detected. Can be useful when linked to facial gestures.

Parameter Type Description
action String The ID of the action.speech event that started the speech synthesis

monitor.speech.mark

Reported by the synthesizer just before a \<mark> is being reached in the utterance (part of the SSML specification).

Parameter Type Description
action String The ID of the action.speech event that started the speech synthesis
name String The name of the mark

monitor.speech.word

Reported by the synthesizer just before each word is being spoken (if monitorWords is set to true in the action.speech event).

Parameter Type Description
action String The ID of the action.speech event that started the speech synthesis
word String The word that is about to be spoken
pos int The position of the word in the utterance (0 being the first word)

action.voice

An action that can be sent to a synthesizer to change the voice. Either name, lang or gender (or a combination thereof) must be provided.

Parameter Type Description
name String The name of the voice
lang String The language code for the voice (such as "en-US")
gender String The gender of the voice ("male" or "female")
agent String The ID of the agent that is associated with this synthesis

Speech recognition

action.listen

An action that can be sent to a recognizer to make it start listening.

Parameter Type Description
context String A name filter with the contexts to use. If omitted, the default context is used (as set with action.context.default).
endSilTimeout int The silence timeout (in msec) to detect end-of-speech
noSpeechTimeout int The silence timout (in msec) if no speech is detected
maxSpeechTimeout int The maximum length of the speech (in msec)
nbest int The maximum number of hypotheses to generate

action.listen.stop

An action that can be sent to a recognizer to make it stop listening.

Parameter Type Description
action String The ID of the action.listen event that started the speech recognizer. If this is omitted, all recognizers should stop

monitor.listen.start

Reported by each RecognizerModule when it starts listening

Parameter Type Description
action String The ID of the action.listen event that issued this recognition
sensor String The ID of the recognizer

action.context.load

Makes each RecognizerModule load a context (such as a grammar or semantic interpreter).

Parameter Type Description
context Context The context to load

monitor.context.load

Sent when a context is loaded.

Parameter | Type | Description ---------- | ---- | -------success | Boolean | Whether loading the context succeeded message | String | An error message in case the loading failed action | String | The ID of the action.context.load event

action.context.default

Sets the default context to be used by each RecognizerModule. This is the context that is used if no context is provided in the action.listen event.

Parameter | Type | Description ---------- | ---- | -------context | String | A name filter with the contexts to use as default

sense.speech.start

Sent by the recognizer when it has detected start-of-speech.

Parameter | Type | Description ---------- | ---- | -------action | String | The ID of the action.listen event that issued this recognition sensor | String | The ID of the recognizer

sense.speech.end

Sent by the recognizer when it has detected end-of-speech.

Parameter | Type | Description ---------- | ---- | -------action | String | The ID of the action.listen event that issued this recognition length | int | The length of the utterance (in msec) sensor | String | The ID of the recognizer

sense.speech.rec

Once the recognition is done, one event that matches sense.speech.rec** is reported. See the different variants below.

If speech recognition is successful, sense.speech.rec will be generated (without suffix). Note that this includes cases where no grammar matched the recogniton, in which case the "text" parameter will be set to "\<NOMATCH>".

Parameter Type Description
text String The result in text
length int The length of the utterance (in msec)
sem Record The semantic interpretation
conf float Confidence score (from 0.0 to 1.0)
words List A list of records containing the individual words
nbest List An n-best list of hypotheses (each a Record containing the attributes in this table)
context String The name of the context that matched
action String The ID of the action.listen event that issued this recognition
sensor String The ID of the recognizer

sense.speech.rec.silence

Reported if no speech was detected and the noSpeechTimeout has been reached.

Parameter Type Description
action String The ID of the action.listen event that issued this recognition
sensor String The ID of the recognizer

sense.speech.rec.maxspeech

Reported if the user's utterance is too long, and the maxSpeechTimeout has been reached.

Parameters are the same as for sense.speech.rec.silence.

sense.speech.rec.failed

Reported if the speech recognition failed due to a technical problem.

Parameters are the same as for sense.speech.rec.silence.

sense.speech.partial

Partial speech recognition result that is reported several times during recognition, if partial results is turned on.

Parameters are the same as for sense.speech.rec.

Embodied agents

action.gesture

Makes the agent perform a specific gesture.

Parameter Type Description
name String The name of the gesture
agent String The ID of the agent

monitor.gesture.start

Parameter Type Description
agent String The ID of the agent
action String The ID of the action.gesture event that issued this gesture

monitor.gesture.end

Sent when the gesture ends

Parameter Type Description
agent String The ID of the agent
action String The ID of the action.gesture event that issued this gesture

action.gaze

Makes the agent shift gaze to a certain location in 3D space

Parameter Type Description
location Location The 3D location where the agent should gaze
mode String See below
agent String The ID of the agent
speed String How fast the head should move

The mode can be one of these:

  • default: Uses eyes and neck in a flexible way
  • eyes: Uses eyes only, hold the neck still
  • headpose: Center the eyes and use the neck

Speed can be one of x-slow, slow, medium, fast, x-fast

monitor.gaze

Sent when the gaze is shifted

Parameter Type Description
action String The ID of the action.gaze event that issued this gaze shift
location Location The resulting gaze target location
head:rotation Rotation The resulting head rotation
agent String The ID of the agent

action.face.texture

Makes the agent change the texture of the face.

Parameter Type Description
name String The name of the texture
agent String The ID of the agent

action.lipsync

An action sent from the speech synthesizer to the embodied agent if lipsync is turned on.

Parameter Type Description
action String The ID of the action.speech event that started the speech synthesis
start int Taken from action.speech
phones Transcription The transcription (phones with timings).
agent String The ID of the agent that is associated with this synthesis

action.lipsync.stop

Stop the lipsync.

Parameter Type Description
action String The ID of the action.speech event that started the speech synthesis
agent String The ID of the agent

monitor.lipsync.start

Sent by an embodied agent to tell the synthesizer that lipsync has started. If lipsync is turned on, the synthesizer should not start until this has been received.

Parameter Type Description
action String The ID of the action.speech event that started the speech synthesis
agent String The ID of the agent

Vision

sense.body

Reported when a camera (or another device) senses bodies

Parameter Type Description
bodies Record A record with all bodies (with body ID:s as keys and Body objects as values)
sensor String The ID of the camera or sensor that detected the bodies

sense.item

Reported when an item is detected or moved, such as an object that can be referred to during the interaction.

Parameter Type Description
items Record A record with all items (with item ID:s as keys and Item objects as values)
sensor String The ID of the camera or sensor that detected the items

Situated interaction

action.situation.detect

An action to retrieve information about the situation, i.e. cause all situation-aware modules to send sense.situation events.

sense.situation

Sent by each situation-aware module when it starts, or when action.situation.detect is issued. The event either contains sensors or system agents.

Each parameter contains either a Sensor or SystemAgent record. The name of each parameter corresponds to the ID of the Sensor or SystemAgent.

action.attend

An action to attend to a specific target, picked up by SystemAgentFlow. Either target or location must be provided.

Parameter Type Description
target String The ID of a user or item that is to be attended (or "nobody" if nobody is to be attended)
location Location A location in 3D space to attend to
mode String See action.gaze above
agent String The ID of the system agent
speed String See action.gaze above

action.attend.all

An action to attend to all users, picked up by SystemAgentFlow.

action.attend.asleep

An action to make the agent fall asleep, picked up by SystemAgentFlow.

monitor.attend

Reported when the agent's attention changes, sent by SystemAgentFlow.

sense.user.enter

Reported by SystemAgentModule when a user enters the interaction space of the system agent.

Parameter Type Description
user String The ID of the user agent
head:location Location The location of the user's head in absolute space
head:rotation Rotation The rotation of the user's head in absolute space
sensor String An ID on the form "SID-BID", where SID is the ID of the camera sensor, and BID is the ID of the body reported by the camera sensor
agent String The ID of the system agent

sense.user.leave

Reported by SystemAgentModule when a user leaves the interaction space of the system agent.

Parameter Type Description
user String The ID of the user agent
agent String The ID of the system agent

sense.user.move

Reported by SystemAgentModule when one or more users move.

Each user is represented as a record under a parameter that is the user agent ID. Each such user record has the same parameters as sense.user.enter.

Parameter Type Description
agent String The ID of the system agent

sense.user.attend

Reported by SystemAgentModule when a user's attention shifts.

Parameter Type Description
user String The ID of the user agent
target String The ID of the agent that is being attended
sensor String See sense.user.enter
agent String The ID of the system agent

sense.item.move

Items relevant to the system agent have been moved.

Parameter Type Description
agent String The ID of the system agent
items Record All items that have moved, with the ID of each item as keys
prominent String The ID of the items that was moved most

sense.user.speak

This is a higher-level event that is produced by the SystemAgentFlow. It corresponds to sense.speech.rec. However, even if there several recognizers, only one sense.user.speak event will be generated. Thus, it is very useful for handling multi-party interaction.

The action.listen will result in either

  • sense.user.speak
  • sense.user.silence
  • sense.user.speak.side (only for multi-party interaction)
  • sense.user.speak.multi (only for multi-party interaction)

The parameters are the same as sense.speech.rec above, with the addition of:

Parameter Type Description
user String The ID of the user agent
attsys boolean Whether or not the user was attending the system during the utterance
all List If several users were speaking, all events will be placed here, each with the same parameters as sense.speech.rec above.

sense.user.speak.side

Reported if one of the users was attended to by the system, and some other user answered.

The parameters are the same as for sense.user.speak.

sense.user.speak.multi

Reported if the system was attending all users and several users replied while attending the system, or if no user attended the system, but several of them replied. If only one user replied, or if only one user replied while attending the system, sense.user.speak is generated instead.

The parameters are the same as for sense.user.speak, with the speech data from the different users merged. The separate speech events can be accessed by the parameter "all".

sense.user.silence

Reported if no user replied and the end-of-speech silence threshold was reached.

sense.user.speech.start

Reported at the start of speech for each user. Corresponds to sense.speech.start, but with the user identified.

Parameter Type Description
user String The ID of the user agent
speakers int The numbers of simultaneous speakers (1 when the first user has started speaking)
attsys boolean Whether or not the user was attending the system at the start of the utterance
agent String The ID of the system agent

sense.user.speech.end

Reported at the end of speech for each user. Corresponds to sense.speech.end, but with the user identified.

Parameter Type Description
user String The ID of the user agent
speakers int The numbers of simultaneous speakers (0 when the last user has stopped speaking)
attsys boolean Whether or not the user was attending the system during the utterance
agent String The ID of the system agent

Logging

action.logging.start

Triggers the LoggingModule to start logging.

Parameter Type Description
timestamp String A timestamp to use when naming the log files. Can be omitted, in which case System.currentTimeMillis() will be used as a name.

action.logging.stop

Triggers the LoggingModule to stop logging.