On

I've been blogging about bots since April, 2016 which was about the time I discovered Microsoft's amazing botframework. I've written a series of how to detect user intent based on text messages using LUIS (Language Understanding Intelligent Service) you can find it here.

In this post I'll talk about understanding user emotion using the Giphy posts embedded within Microsoft teams. This can be generalized to any image communication between you and the bot.

First Let's get the Image Content

 Using Microsoft teams insert Giphy functionality we can see that it adds an attachment to the communication message between the user and the bot, this attachment is a mere link to the chosen Giphy.



Let's add the image understanding capability to our bot

 We will use emotion detection service which is a part of cognitive service to detect the giphy emotion which is basically a POST request to emotion detection service with the subscription key added as a "Ocp-Apim-Subscription-Key"  header

 request(
     {
         url: 'https://westus.api.cognitive.microsoft.com/emotion/v1.0/recognize',
                method: 'POST',
                headers: {
                    'Ocp-Apim-Subscription-Key': '*Add Your Subscription key here*',
                    'Content-Type': 'application/json'
                },
                body: {
                    'url': 'your Giphy image URL'
                },
                json: true
            }, function (err, response, body) {
                if (err) {
                    callback(err, null);
                }
                //successful call
                callback(null, getHighScoreEmotion(body));
            });

the successful response will contain an object with score of each possible emotion, I've created a function that returns the highest emotion and then send a message to the user to reflect the emotion detection.
function getHighScoreEmotion(body) {
    var val=0;;
    var emotion;
    if (body.length > 0) {
        for (score in body[0].scores) {
            if(body[0].scores[score]>val){
                val=body[0].scores[score];
                emotion=score;
            }
        }
        return emotion;
    }
    return null;
    
}

and now how it's look like when you send your bot a giphy :)


Now our bot can understand and response to the Giphys shared by the user