Introduction

In this tutorial, you will learn the basics of how to make a skill that has a graphical web interface that Furhat is aware of and can interact with. The goal in this tutorial is to create a simple Skill where Furhat shows two cats to the user on a graphical interface, looks at the cats, and comments on them when the user clicks on them.

Note: this tutorial assumes that you:

  • Have gone through the tutorial on how to work in Eclipse
  • Have the Furhat SDK version > 0.1.0 up and running on your dev-server
  • Have an external touch device (to be able to test this with a Furhat)
  • Have a Furhat robot with a Firmware later than 2017-01-17 (to run on Furhat)

Configuring the skill file to enable a display

Setting up a display can be done like this (see example code below):

  • First, in the constructor of our skill we need to set up the requirements so that the skill won't start unless a graphical interface is connected.
  • Secondly, in the init() we fetch a connected display from the skillhandler (LiveMode in this case).
  • Then we tell the display to subscribe to all events starting with action.game.
  • Then we register a webhandler (defined in the next section) to handle all requests on the url /cats.
  • Finally, we tell the display load the cat.html. We could also use a Java FreeMarker template if we want dynamic, server-rendered data but in this example a static html is fine.
// CatsGameSkill.java

// Set up a display variable
private Display display;

public CatsGameSkill() {

    // ...

    SkillRequirements requirements = getRequirements();
    requirements.setDisplay(Display.TOUCH_TABLE);

    // ...

}

@Override
public void init() throws Exception {

    SkillHandler handler = getSkillHandler();

    display = handler.getDisplay(Display.TOUCH_TABLE);

    display.subscribe("action.game**");

    handler.registerWebHandler(new CatsGameHandler(), "/cats");
    display.load("/cats/cat.ftl");

    // ...
}

Creating a web handler

A web handler handles a subset of all web requests that the Furhat's webserver receives. In this case, we want to serve an html file from a server in our package called web. We use the supplied FreemarkerHandler that is built for Freemarker templates but also works great for html files.

Since we don't have any data we need to pass on to the handler because we have a static html file, we pass an empty record in the superclass constructor.

// CatsGameSkill.java

private class CatsGameHandler extends FreemarkerHandler {

    public CatsGameHandler() throws IOException {
        super(new Record(), getPackageFile("web"));
    }
}

Creating a web template

Then, we define a simple webapp with a static HTML markup and a small JavaScript file.

The cats.js event listeners and senders

  • We first set up our system that we send and receive events from.
  • Secondly, we send a monitor.game.connect event to signal to the skill that we are connected.
  • Thirdly, we setup an event listener that shows our cat images when we receive the action.game.start event.
  • Finally, we define a function to send a action.game.catclick event that will be called when each cat picture is clicked and will pass on the picture's item-id attribute
// cats.js

// Accessees the system used to send events
var system = parent.system;

// Sends a monitor event to the flow. This triggers no action for now, but is a good practise.
system.sendEvent("monitor.game.connect", {});

// Event Listener
system.onEvent = function (name, params) {
    // Shows cats on game start
    if (name == "action.game.start") {
        cats = document.getElementsByClassName("cat");
        for (var i = 0; i < cats.length; i ++) {
            cats[i].style.display = 'block';
        }
    }
}

// Function to send the clicked cat name to Furhat
function catClick(item) {
    system.sendEvent("action.game.catclick", {
        cat: item.getAttribute("item-id")
    });
}

The cats.html markup

The markup is very simple.

  • First we load our cats.js (defined above).
  • Secondly, we insert two cat images and give them each an item-id attribute. This attribute will be directly usable from the skill Flow. We also add an onclick listener that calls our catClick function defined above.
<!-- cats.html -->

<html>
  <head>
    <title>Cat display test</title>
    <meta name="viewport" content="width=device-width, user-scalable=no, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0, minimal-ui">
    <style>
      body { margin: 0 !important }
    </style>
    <script type="text/JavaScript" src="cats.js"></script>
  </head>
  <body>
    <img class="cat" src="cat1.jpg" item-id="molly" style="display: none; width:50%; float:left" onclick="catClick(this)"/>
    <img class="cat" src="cat2.jpg" item-id="denise" style="display: none; width:50%; float:right" onclick="catClick(this)"/>
  </body>
</html>

Setting up the flow

The flow is very simple:

  • We send an action.game.start event when we enter the Game state.
  • We then react on each action.game.catclick event by attending the cat in question (passed on as the cat variable and defined in the markup by item-id) and commenting on it.
<!-- CatsGameFlow.xml -->

<!-- ... -->
<state id="Game">
    <onentry>
        <agent:say>I like cats. Here are two of my favourites. Click on them, why dont you?!</agent:say>
        <send event="action.game.start"/>
    </onentry>

    <onevent name="action.game.catclick">
        <exec>String cat = asString(event:cat)</exec>
        <agent:attend target="cat"/>
        <agent:say>This is <expr>cat</expr>. Isn't she cute</agent:say>
    </onevent>
</state>

Connecting a display and running the skill

Since we added a touch-display as a SkillRequirement, we won't be able to start the Skill unless a display is connected. To do this:

  • Browse to the web interface on a device (or in a separate browser window)
  • Click Displays in the left menu. Select size and position relative to the Furhat (so that the agent:attend commands work accurately)
  • Now you can start the skill normally from your other device or browser window and the GUI will automatically load in an Iframe on the display device.