Trusted Application API (Skype for Business) – Authentication and Online Meetings

Programming for Trusted Application API (added table of contents…)

Skype for Business Development Platform provides the various developer experiences, but there’s been no way to develop server-side application with online SKUs for a long time. (With server SKUs, you can use existing UCMA for the server-side endpoint programming.)
Now you can use the Trusted Application API for this sort of server-side development with Office 365.  (Now in preview.)

This brand-new Trusted Application API covers the following server-side (backend) scenarios for developers. (See “MSDN : Trusted Application API” for details.)
Especially, you can assign the PSTN phone number to this trusted endpoint, and then you can provide the telephony solutions like IVR (Interactive Voice Response) for all users who is not having the Skype client.
Trusted Application API is not just for the bot, but including a lot of server-side powerful scenarios.

  • Bots and Notifications
  • Anonymous Customer Web Chat
  • PSTN audio conferencing
    (IVR to join the conference, in-meeting Personal Virtual Assistant, and in-meeting announcements)
  • Service-side meeting recording
  • Inbound/outbound IVRs
  • Helpdesk
  • Expert-finder
  • Customer engagement / Contact Center

Note : Several scenarios are not available in current preview (Feb 2017).

In this post, we focus on the authentication and online meetings for your first start.
Later I will show you the simple programming code with SDK, but first let’s look at the quick view of HTTP flows, because it helps you understand how it works on the bottom and trouble-shootings.

Endpoint Registration

Before building your applications, you must register your app (endpoint) in Azure AD (Azure Active Directory) and Skype for Business Online Trusted Application platform.

First you should go to Skype for Business Online Application Registration Portal, login with Office 365 admin account, fill the settings, and create your application. This application is registered in your Azure AD tenant. (You can see the registered application in Azure Portal without Azure subscription.)

Trusted Application API uses the application context token instead of the user context (see “Azure AD – Backend server-side application” in my early post). Then please select the appropriate application permissions (not delegated permissions) according to your application’s functionalities in the portal. (see the following screenshot)
For example, if your application handles the online meeting capabilities, select only [Create on-demand Skype meetings], [Join and Manage Skype Meetings], and [Guest user join services] in Application Registration Portal.

After you created, please copy the generated application id and client secret (key).

Note that you must create (or setup) your application using Skype for Business Online Application Registration Portal, not using Azure Portal. If you have already registered your application with Azure Portal, make sure to set up with Skype for Business Online Application Registration Portal again. (see the following screenshot)
Because the application must be stored in Skype for Business Online platform.

After you complete the application registration, you must access the following url with your Office 365 administrator account, and consent this application as administrator.
Please replace the following {application id} and {sign-on url} for your appropriate values. (Note that the all values must be url-encoded.)
As you can see, the resource id of Trusted Application API is https://noammeetings.resources.lync.com .

https://login.windows.net/common/oauth2/authorize?response_type=id_token
  &client_id={application id}
  &redirect_uri={sign-on url}
  &response_mode=form_post
  &nonce=123456
  &resource=https%3A%2F%2Fnoammeetings.resources.lync.com
  &prompt=admin_consent

When you login with admin account, the following consent UI is displayed. Please accept this consent.

Next you must register the trusted application endpoint in your Skype for Business Online Trusted Application platform.
Before doing that (registration), please download and install Skype for Business Online Windows PowerShell Module in your Windows client beforehand.

After the installation is done, launch PowerShell and run the following commands.

$cr = Get-Credential
# It prompts login. Enter your admin user id and password.
$session = New-CsOnlineSession -Credential $cr
Import-PSSession $session
New-CsOnlineApplicationEndpoint `
  -Uri "sip:{arbitrary unique name}@{your domain prefix}.onmicrosoft.com" `
  -ApplicationId "{application id}" `
  -Name "{your app name}" `
  -Tenant "{your tenant id}"

# If you get tenant id, please input the following
# $tenantId = (Get-MsolCompanyInformation).objectId
# echo $tenantId

Here’s my example.

$cr = Get-Credential
# It prompts login. Enter your admin user id and password.
$session = New-CsOnlineSession -Credential $cr
Import-PSSession $session
New-CsOnlineApplicationEndpoint `
  -Uri "sip:trustedapidemo01@mod776816.onmicrosoft.com" `
  -ApplicationId "d4daaf71-5f06-4f70-bf4b-418a97f34741" `
  -Name "TrustedApiTest01" `
  -Tenant "3bc5ea6c-9286-4ca9-8c1a-1b2c4f013f15"

As I described earlier, you can assign PSTN phone number to trusted endpoint, but here I skip this step.

Note : You can also remove (Remove-CsOnlineApplicationEndpoint) and modify (Set-CsOnlineApplicationEndpoint) the trusted endpoint by PowerShell.

Authentication and Initialization

Trusted Application API uses the application context token, not user context token. As I explained in my early post “Azure AD – Backend server-side application“, no interactive UI is needed for retrieving token.
We can get the application context token (access token) for the Trusted Application API (https://NOAMmeetings.resources.lync.com) with only application id and secret. (See the following HTTP request.)
Note that you need the tenant-aware uri (https://login.microsoftonline.com/{your tenant realm}/oauth2/token) as follows.

POST https://login.microsoftonline.com/mod776816.onmicrosoft.com/oauth2/token
Accept: application/json
Content-Type: application/x-www-form-urlencoded

resource=https%3A%2F%2FNOAMmeetings.resources.lync.com&
client_id=d4daaf71-5f06-4f70-bf4b-418a97f34741&
client_secret=kEQ2B1rCs...&
grant_type=client_credentials
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "token_type": "Bearer",
  "expires_in": "3600",
  "ext_expires_in": "262800",
  "expires_on": "1492076011",
  "not_before": "1492072111",
  "resource": "https://NOAMmeetings.resources.lync.com",
  "access_token": "eyJ0eXAiOi..."
}

After you’ve got access token, first we ask for the trusted endpoint url using the discovery service. (See the following HTTP request.)
You must set the retrieved access token in HTTP header as follows.

GET https://api.skypeforbusiness.com/platformservice/discover
Authorization: Bearer eyJ0eXAiOi...
Accept: application/json; charset=utf-8
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "_links": {
    "self": {
      "href": "https://api.skypeforbusiness.com:4443/platformservice/discover"
    },
    "service:applications": {
      "href": "https://ring2noammeetings.resources.lync.com/platformService/v1/applications"
    },
    "myApplications": {
      "href": "https://ring2noammeetings.resources.lync.com/platformService/v1/myApplications"
    }
  },
  "rel": "service:discover"
}

Next you get the access url for your sip application (sip:{unique name}@{your domain prefix}.onmicrosoft.comas follows.
Please replace endpointId with your app’s sip id.

GET https://ring2noammeetings.resources.lync.com/platformService/v1/applications?endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com
Authorization: Bearer eyJ0eXAiOi...
Accept: application/json; charset=utf-8
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "_links": {
    "self": {
      "href": "/platformservice/v1/applications"
    },
    "service:application": {
      "href": "/platformservice/v1/applications/3877116191?endpointId=sip%3atrustedapidemo01%40mod776816.onmicrosoft.com"
    }
  },
  "rel": "service:applications"
}

Next you retrieve all endpoints for each resources (resources for ad-hoc meeting, anonymous joining token, messaging, etc) as follows.

For example, if your app want to create new ad-hoc meeting, use the following /platformservice/v1/applications/3877116191/adhocMeetings .

GET https://ring2noammeetings.resources.lync.com/platformservice/v1/applications/3877116191?endpointId=sip:trustedapidemo01%40mod776816.onmicrosoft.com
Authorization: Bearer eyJ0eXAiOi...
Accept: application/json; charset=utf-8
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "_links": {
    "self": {
      "href": "/platformservice/v1/applications/3877116191?endpointId=sip%3atrustedapidemo01%40mod776816.onmicrosoft.com"
    },
    "service:anonApplicationTokens": {
      "href": "/platformservice/v1/applications/3877116191/anonApplicationTokens?endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com"
    }
  },
  "_embedded": {
    "service:communication": {
      "_links": {
        "self": {
          "href": "/platformservice/v1/applications/3877116191/communication?endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com"
        },
        "service:joinOnlineMeeting": {
          "href": "/platformservice/v1/applications/3877116191/communication/onlineMeetingInvitations?endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com"
        },
        "service:inviteUserToMeeting": {
          "href": "/platformservice/v1/applications/3877116191/communication/userMeetingInvitations?endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com"
        },
        "service:startMessaging": {
          "href": "/platformservice/v1/applications/3877116191/communication/messagingInvitations?endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com"
        },
        "service:startAudioVideo": {
          "href": "/platformservice/v1/applications/3877116191/communication/audioVideoInvitations?modalities=AudioVideou0026endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com"
        },
        "service:startAudio": {
          "href": "/platformservice/v1/applications/3877116191/communication/audioVideoInvitations?modalities=Audiou0026endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com"
        }
      },
      "rel": "service:communication",
      "etag": "4294967295"
    },
    "myOnlineMeetings": {
      "_links": {
        "self": {
          "href": "/platformservice/v1/applications/3877116191/myOnlineMeetings?endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com"
        }
      },
      "rel": "myOnlineMeetings"
    },
    "service:adhocMeetings": {
      "_links": {
        "self": {
          "href": "/platformservice/v1/applications/3877116191/adhocMeetings?endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com"
        }
      },
      "rel": "service:adhocMeetings"
    }
  },
  "rel": "service:application"
}

Sample – Interact with Adhoc Online Meeting

The schema of Trusted Application endpoint is based on existing UCWA schema. If you’re familiar with UCWA, you can easily understand the envelop of trusted application’s HTTP flow.
As I mentioned before, there are many scenarios (bot, IVR, call bridge, etc) for trusted api, and let’s see the simple ad-hoc meeting example here.

Skype for Business developer platform is providing the B2C scenario like “staff-customers”, “doctor-patients”, or “lawyer-clients” using the Skype guest online meeting join. (See the document of Skype for Business App SDK or Skype Web SDK in MSDN.)
In this scenario, the application must provide the following steps.

  • When it’s called, the application creates the ad-hoc meeting in the backend.
  • The staff joins in this generated meeting as Skype users with Skype client or Skype Web SDK etc.
  • On the other hand, the customer joins as the guest account (as anonymous user) without Skype license. The application must provide the token for anonymous join, and Skype Web SDK / Skype for Business App SDK will provide the user interface for customers. (They don’t have Skype client.)

As you can see, the backend api helps several tasks in this scenario. (For instance, creating ad-hoc meeting, providing the token for anonymous join, etc)
Now here I describe how you can leverage Trusted Application API along with this scenario.

For example, the following is creating new ad-hoc meeting with Trusted Application API. The uri fragment of /platformservice/v1/applications/3877116191/adhocMeetings is the previously retrieved uri.

The retrieved joinUrl is the meeting url. For instance, you can join the meeting by accessing this url with your Web browser.

POST https://ring2noammeetings.resources.lync.com/platformservice/v1/applications/3877116191/adhocMeetings?endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/vnd.microsoft.com.ucwa+json; charset=utf-8

{
  "subject": "meeting01",
  "accessLevel": "Everyone",
  "rel": "service:meeting"
}
HTTP/1.1 200 OK
Content-Type: application/vnd.microsoft.com.ucwa+json; charset=utf-8
ETag: "4221088333"

{
  "accessLevel": "Everyone",
  "entryExitAnnouncement": "Disabled",
  "automaticLeaderAssignment": "Disabled",
  "description": "",
  "expirationTime": "/Date(1492106326000)/",
  "onlineMeetingId": "PYQC0NM0",
  "onlineMeetingUri": "sip:DM20R04meet1467@noammeetings.lync.com;gruu;opaque=app:conf:focus:id:PYQC0NM0",
  "organizerUri": "sip:DM20R04meet1467@noammeetings.lync.com",
  "conferenceId": "PYQC0NM0",
  "phoneUserAdmission": "Disabled",
  "lobbyBypassForPhoneUsers": "Disabled",
  "subject": "meeting01",
  "joinUrl": "https://meet.resources.lync.com/NOAMmeetings/dm20r04meet1467/PYQC0NM0",
  "_links": {
    "self": {
      "href": "/platformservice/v1/applications/3877116191/adhocMeetings/PYQC0NM0?endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.comu0026onlineMeetingContext=sip:DM20R04meet1467@noammeetings.lync.com"
    },
    "service:discover": {
      "href": "https://noammeetings.resources.lync.com/platformService/discover?discoverContext=HSAUbe4hAr..."
    },
    "service:joinAdhocMeeting": {
      "href": "https://webpooldm20r04.infra.lync.com/platformservice/v1/applications/3877116191/communication/onlineMeetingInvitations?confUrl=sip:DM20R04meet1467@noammeetings.lync.com;gruu;opaque=app:conf:focus:id:PYQC0NM0u0026endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com"
    },
    "applications": {
      "href": "https://webpooldm20r04.infra.lync.com/ucwa/v1/applications"
    }
  },
  "rel": "service:adhocMeeting",
  "etag": "4221088333"
}

Next example is retrieving the anonymous token for this meeting.

POST https://ring2noammeetings.resources.lync.com/platformservice/v1/applications/3877116191/anonApplicationTokens?endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/vnd.microsoft.com.ucwa+json; charset=utf-8

{
  "meetingUrl": "https://meet.resources.lync.com/NOAMmeetings/dm20r04meet1467/PYQC0NM0",
  "allowedOrigins": "https://contoso.com/callback",
  "applicationSessionId": "test0001"
}
HTTP/1.1 200 OK
Content-Type: application/vnd.microsoft.com.ucwa+json; charset=utf-8

{
  "token": "psat=eyJ0eXAiOiJKV1QiLCJh...",
  "expiryTime": "/Date(1492108195460)/",
  "_links": {
    "self": {
      "href": "https://ring2noammeetings.resources.lync.com:4443/platformservice/v1/applications/3877116191/anonApplicationTokens?endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com"
    },
    "service:discover": {
      "href": "https://noammeetings.resources.lync.com/platformService/discover?anonymousMeetingJoinContext=psat%253deyJ0eXAiOiJKV1QiLCJh..."
    }
  },
  "rel": "service:anonApplicationToken"
}

Once the user gets the anonymous token, the user can join the meeting with Skype Web SDK (without Skype for Business client) as follows.
(Here I don’t explain about details, but please see “Introduction for Skype Web SDK” in my early post for Skype Web SDK programming.)

app.signInManager.signIn(
  {
    "name": "meeting01",
    "token": "Bearer psat=eyJ0eXAiOi...",
    "root": {
      "user": "https://noammeetings.resources.lync.com/platformService/discover?anonymousMeetingJoinContext=psat%253deyJ0eXAiOi..."
    },
    "cors": true
  }
).then(function () {
  ...

}, function (err) {
  ...
  
});

You can also invite the licensed user (Skype for Business user) for joining the adhoc meeting.
First your application joins adhoc meeting as follows. The uri (https://webpooldm20r04.infra.lync.com) and conversationId is retrieved (notified) by the callback endpoint, and I will explain about this callback process in the next post. (Here I don’t explain about the callback.)

POST https://webpooldm20r04.infra.lync.com/platformservice/v1/applications/3877116191/communication/onlineMeetingInvitations?confUrl=sip:DM20R04meet1467@noammeetings.lync.com;gruu;opaque=app:conf:focus:id:PYQC0NM0&endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/vnd.microsoft.com.ucwa+json; charset=utf-8

{
  "operationId": "4783b19f-0e0a-4889-aca6-8d2787859f0d",
  "callbackUrl": "https://contoso.com/callback",
  "rel": "service:JoinMeetingInvitationInput"
}
HTTP/1.1 201 Created
Location: /platformservice/tgt-46c902aa67d0567bab2d9d7a16414e98/v1/applications/3877116191/communication/onlineMeetingInvitations/d56657f4-1d04-4773-bf52-4d0ae218f960?endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com

Next your application invites the user to join. (Here we’re inviting BenW@MOD776816.onmicrosoft.com.)

POST https://webpooldm20r04.infra.lync.com/platformservice/tgt-46c902aa67d0567bab2d9d7a16414e98/v1/applications/3877116191/communication/participantInvitations?endpointId=sip:trustedapidemo01@mod776816.onmicrosoft.com&conversationId=c9076870-07fb-48bd-979f-49cc5498f982
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/vnd.microsoft.com.ucwa+json; charset=utf-8

{
  "operationId": "1ca3bf11-017f-418f-81d9-c80c69934b29",
  "to": "sip:BenW@MOD776816.onmicrosoft.com"
}
HTTP/1.1 201 Created

When HTTP request is sent, the licensed user (Skype for Business user) receives the following invitation in the Skype for Business client.

If the licensed user (Skype for Business user) has joined, the licensed user (staff) and the anonymous user (guest customer) can be connected through the online meeting.

Note : Currently the internal error (PlatformService.Web.PlatformServiceWebException, etc) will frequently occur, when your request is not valid. (For example, when you access the expired meeting, etc…) Sorry, but it’s difficult to know the reason of error.

SDK for Trusted Application API

If you’re .NET programmer, you can use the .NET SDK for calling Trusted Application API.
You just install the following NuGet package in your project. (Note that .NET 4.6.2 is needed, and select [include prerelease] like the following screenshot.)

Microsoft.SkypeforBusiness.TrustedApplicationAPI.SDK
Microsoft.SkypeforBusiness.TrustedApplicationAPI.ResourceContact

The following is the C# programming example of the previous HTTP flow (including authentication, initialization, ad-hoc meeting creation, and getting anonymous token).
Sorry, but it’s classic Win Form application … (In most cases, the real application will be hosted on Web.)

...
using Microsoft.SfB.PlatformService.SDK.Common;
using Microsoft.SfB.PlatformService.SDK.ClientModel;

...

public ClientPlatformSettings platsettings = null;
public ApplicationEndpoint appendpoint = null;
public string joinUrl = null;

private async void Initialize()
{
  platsettings = new ClientPlatformSettings(
    "kEQ2B1rCsf...", // secret (key)
    Guid.Parse("d4daaf71-5f06-4f70-bf4b-418a97f34741")); // application id
  var clplatform = new ClientPlatform(
    platsettings,
    new MyLogger());
  var endpointsettings = new ApplicationEndpointSettings(
    new SipUri("sip:trustedapidemo01@mod776816.onmicrosoft.com")); // sip uri
  appendpoint = new ApplicationEndpoint(
    clplatform,
    endpointsettings,
    null);
  await appendpoint.InitializeAsync().
    ConfigureAwait(false);
  await appendpoint.InitializeApplicationAsync().
    ConfigureAwait(false);
  MessageBox.Show("done");
}

private async void CreateAdhocMeeting()
{
  var createarg = new AdhocMeetingCreationInput("meeting01");
  var adhocmeetingResources =
    await appendpoint.Application.CreateAdhocMeetingAsync(createarg)
    .ConfigureAwait(false);
  joinUrl = adhocmeetingResources.JoinUrl;
  MessageBox.Show("done");
}

private async void GetAnonymousToken()
{
  var tokenResources =
    await appendpoint.Application.GetAnonApplicationTokenForMeetingAsync(
      joinUrl,
      @"https://contoso.com/callback",
      "test0001").ConfigureAwait(false);
  MessageBox.Show("done");
}

private async void InviteSkypeUser() // Join meeting -> Invite user
{
  // Here I don't describe the code, because it needs callback ...
  // I will explain about callback (webhook) in the next post.
}

...

public class MyLogger : IPlatformServiceLogger
{
  public bool HttpRequestResponseNeedsToBeLogged { get; set; }

  public void Information(string message)
  {
    MessageBox.Show($"[INFO]{message}");
  }

  public void Information(string fmt, params object[] vars)
  {
    MessageBox.Show($"[INFO]{string.Format(fmt, vars)}");
  }

  public void Information(Exception exception, string fmt, params object[] vars)
  {
    MessageBox.Show($"[INFO]{string.Format(fmt, vars)}");
  }

  public void Warning(string message)
  {
    MessageBox.Show($"[WARN]{message}");
  }

  public void Warning(string fmt, params object[] vars)
  {
    MessageBox.Show($"[WARN]{string.Format(fmt, vars)}");
  }

  public void Warning(Exception exception, string fmt, params object[] vars)
  {
    MessageBox.Show($"[WARN]{string.Format(fmt, vars)}");
  }

  public void Error(string message)
  {
    MessageBox.Show($"[ERR]{message}");
  }

  public void Error(string fmt, params object[] vars)
  {
    MessageBox.Show($"[ERR]{string.Format(fmt, vars)}");
  }

  public void Error(Exception exception, string fmt, params object[] vars)
  {
    MessageBox.Show($"[ERR]{string.Format(fmt, vars)}");
  }
}

 

Reference : early posts (all Japanese) for “Skype for Business” development platform

Advertisements

Build Bot with Voice (Skype)

In the previous post “BUILD BOT with Microsoft Bot Framework Rest Api” I described about the rest raw of text messaging bot. The SDK is built on the rest api. Therefore, you can understand the bottom technology by learning this rest api.
In this blog post, I show you the same for the calling (voice) bot, not text messaging bot. I hope this helps you to understand the essence of voice communications in Microsoft Bot Framework.

If it’s Skype, you (developers) can provide the calling (voice) bot using Microsoft Bot Framework. (In Microsoft Bot Framework, this calling bot is only supported by Skype channel.)

Note : If you use SDK (.NET, Node.js), you can refer the official document for the step-by-step tutorial of the calling bot.

Note : Currently only en-US culture is supported in the calling bot. Therefore, if you want to handle ja-JP (Japanese language), please use the following play and record communication now.

Settings

The prerequisite settings is the same as normal chatbot in Microsoft Bot Framework. (Please refer the document.)
But, in this case (when you create the calling bot), you need to do the additional settings for calling. First, open the bot settings page, go to Skype channel configuration page, and enable “Calls” feature in Skype channel. (See the following screenshot.)
You must also set the call endpoint (which I explain later) in this settings.

Authentication

If you proceed the bot communication in the webhook, all request is having Authorization header like following, and you must verify this token each time. For verifying this token, please refer the previous post “BUILD BOT with Microsoft Bot Framework Rest Api“.

POST https://example.com/yourbot/callback
Accept: application/json
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

...

Call endpoint and Callback endpoint

The calling (voice) communication needs two endpoints in your bot side: the one is “call” endpoint and the other is “callback” endpoint.

When the user starts to communicate with your bot, the first webhook arrives at your call endpoint like the following HTTP request.

POST https://example.com/yourbot/call
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "id": "13107307-7bd6-4c5e-9a1b-65b98464cee6",
  "participants": [
    {
      "identity": "29:12BgCTOvVtWCWb0LlRkes7g428GXh_A4Gl9qbfce7YteH4zcD5pqSlQB-OMF1MVRM",
      "languageId": "en",
      "originator": true
    },
    {
      "identity": "28:d82c7c25-ddb4-426a-8b59-76a8a034abb4",
      "originator": false
    }
  ],
  "isMultiparty": false,
  "presentedModalityTypes": [
    "audio"
  ],
  "callState": "incoming"
}

When your bot accepts this calling request, your bot reply the callback endpoint in the response body.

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "links": {
    "callback": "https://example.com/yourbot/callback"
  },
  ... (here, explain later)
  
}

After that, the Bot Framework (or Skype Bot Platform) calls this callback endpoint for subsequent communications. The all requests (webhook) in this communications is done by this callback url, after the acceptance.

In the Skype configuration settings page which I previously explained, you must set the “call” endpoint in “Calling Webhook” textbox.

Key concepts – Actions and Outcomes

All event is notified as the callback webhook, and you can reply as corresponding HTTP response. (the request-reply pattern)
When your bot want to act some operations, your bot must set these activities in HTTP respose as “action“. When the user has responded to this activity (or some system event has occured), this event is included as the “operation outcome” in the callback webhook (HTTP request body). That is, the outcome is corresponding to some specific action.

The type of actions (outcomes) are: answer (answerOutcome), playPrompt (playPromptOutcome), recognize (recognizeOutcome), record (recordOutcome), reject (rejectOutcome), and hangup (hangupOutcome).

Why this style of communication is used ?
When we were creating chatbot using only text messages (see “BUILD BOT with Microsoft Bot Framework Rest Api“), the communication pattern might be essentially one-way. Imagine if you are creating the alarm bot. This sort of bot might be idle while waiting, and is accidentally triggered by some occurrence.
But, when you create the calling (voice) bot, the state is always connected till hang-up. That is reason that the calling bot communication pattern is request-reply with “actions” and “outcomes”, and continuously communicates each other till hang-up.

Now, let’s see the communication flow !

First of all, when your bot accepts the initial call request (which is described above), your bot sends the following “answer” action. (If your bot refuses, please use “reject” action.)

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "links": {
    "callback": "https://example.com/yourbot/callback"
  },
  "actions": [
    {
      "operationId": "673048f9-4442-440b-93a3-faa7433c977a",
      "action": "answer",
      "acceptModalityTypes": [
        "audio"
      ]
    }
  ],
  "notificationSubscriptions": [
    "callStateChange"
  ]
}

You can include multiple outcomes in one HTTP response. For example, if you accepts the initial call request and reply some messages to the user, your bot can send like the following HTTP response.
When you use the “playPrompt” action with text value, Microsoft Bot Framework (Skype Bot Platform) transfers to the voice (speech) using the built-in text-to-speech engine.

HTTP/1.1 200 OK
Content-Length: 383
Content-Type: application/json; charset=utf-8

{
  "links": {
    "callback": "https://example.com/yourbot/callback"
  },
  "actions": [
    {
      "operationId": "673048f9-4442-440b-93a3-faa7433c977a",
      "action": "answer",
      "acceptModalityTypes": [
        "audio"
      ]
    },
    {
      "operationId": "030eeb97-8210-48fd-b497-d761154f0b5a",
      "action": "playPrompt",
      "prompts": [
        {
          "value": "Welcome to test bot",
          "voice": "male"
        }
      ]
    }
  ],
  "notificationSubscriptions": [
    "callStateChange"
  ]
}

When the playPrompt action is accepted, the Bot Framework calls the following callback (webhook) with the outcome. This outcome means that the prompt is successfully accepted by the user.

POST https://example.com/yourbot/callback
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "id": "13107307-7bd6-4c5e-9a1b-65b98464cee6",
  "operationOutcome": {
    "type": "playPromptOutcome",
    "id": "030eeb97-8210-48fd-b497-d761154f0b5a",
    "outcome": "success"
  },
  "callState": "established"
}

If your bot wants to ask something to the user, your bot use “recognize” action in the HTTP response.
For example, the following is requesting the choice of the dial pad digit to the user.

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "links": {
    "callback": "https://example.com/yourbot/callback"
  },
  "actions": [
    {
      "operationId": "32e1a166-f557-4b22-9fd7-64a742d5f040",
      "action": "recognize",
      "playPrompt": {
        "operationId": "bab59923-63d2-48a0-9d34-c0fbadb54435",
        "action": "playPrompt",
        "prompts": [
          {
            "value": "If you want to report technical issues, press 1. You want to ask about our products, press 2.",
            "voice": "male"
          }
        ]
      },
      "bargeInAllowed": true,
      "choices": [
        {
          "name": "1",
          "dtmfVariation": "1"
        },
        {
          "name": "2",
          "dtmfVariation": "2"
        }
      ]
    }
  ],
  "notificationSubscriptions": [
    "callStateChange"
  ]
}

When your bot want to provide the choice by the voice (not dial pad digit), your bot sends the following HTTP response. In this example, if the user speaks “yes” or “okay”, the “Yes” is returned as the result of choice.

Note that this isn’t the speech recognition feature itself (speech-to-text functionality), but choice by the voice. If the user speaks other words, your bot cannot recognize that.
If your bot needs speech-to-text functionality itself, please use the recording capability which I explain later.

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "links": {
    "callback": "https://example.com/yourbot/callback"
  },
  "actions": [
    {
      "operationId": "9a109320-7f30-46f4-a894-a47a4eb8b398",
      "action": "recognize",
      "playPrompt": {
        "operationId": "01105459-d549-4327-b85d-0b0c94b62e8e",
        "action": "playPrompt",
        "prompts": [
          {
            "value": "Please answer yes or no.",
            "voice": "male"
          }
        ]
      },
      "bargeInAllowed": true,
      "choices": [
        {
          "name": "Yes",
          "speechVariation": [
            "Yes",
            "Okay"
          ]
        },
        {
          "name": "No",
          "speechVariation": [
            "No",
            "None"
          ]
        }
      ]
    }
  ],
  "notificationSubscriptions": [
    "callStateChange"
  ]
}

When the user has responded for the “recognize” action, the following callback (webhook) is called. This is the example of the dial pad digit choice.

POST https://example.com/yourbot/callback
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "id": "13107307-7bd6-4c5e-9a1b-65b98464cee6",
  "operationOutcome": {
    "type": "recognizeOutcome",
    "id": "09c78c7c-33fc-488c-808b-0db83de1b433",
    "choiceOutcome": {
      "completionReason": "dtmfOptionMatched",
      "choiceName": "1"
    },
    "outcome": "success"
  },
  "callState": "established"
}

If the user is wasting the time to think, the following system event is returned as the recognize outcome.

POST https://example.com/yourbot/callback
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "id": "13107307-7bd6-4c5e-9a1b-65b98464cee6",
  "operationOutcome": {
    "type": "recognizeOutcome",
    "id": "32e1a166-f557-4b22-9fd7-64a742d5f040",
    "choiceOutcome": {
      "completionReason": "initialSilenceTimeout"
    },
    "outcome": "failure",
    "failureReason": "InitialSilenceTimeout"
  },
  "callState": "established"
}

If your bot wants to hang up (disconnect), your bot sends the following HTTP response with the “hangup” action.

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8

{
  "links": {
    "callback": "https://example.com/yourbot/callback"
  },
  "actions": [
    {
      "operationId": "d2cb708e-f8ab-4aa1-bcf6-b9396afe4b70",
      "action": "hangup"
    }
  ],
  "notificationSubscriptions": [
    "callStateChange"
  ]
}

Conversation with Play / Record

Using Microsoft Bot Framework, your bot can also play media, or record as binary. This capabilities are very much used in the real scenarios : playing music on hold, recording messages to someone, etc
Especially, you can do the interactive talks using this capabilities as follows. In the case of needing the high-quality voice guidance, you can also use these capabilities.

  • Record user’s request (speech) and get binary
  • Call external speech-to-text engine (like Bing Speech API) and get text value
  • Select speech binary for response and play

Let’s see these capabilities.
If your bot plays some existing audio file, your bot sends the HTTP response like follows. As you can see, your bot can use the audio file uri in the playPrompt action, instead of text value.

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "links": {
    "callback": "https://example.com/yourbot/callback"
  },
  "actions": [
    {
      "operationId": "d18e7f63-0400-48ff-964b-302cf4910dd3",
      "action": "playPrompt",
      "prompts": [
        {
          "fileUri": "http://example.com/test.wma"
        }
      ]
    }
  ],
  "notificationSubscriptions": [
    "callStateChange"
  ]
}

If you want to record the user’s response, send the “record” action. (The recording starts !)

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "links": {
    "callback": "https://example.com/yourbot/callback"
  },
  "actions": [
    {
      "operationId": "efe617d7-4de5-42e9-b4e1-90dfd5850e49",
      "action": "record",
      "playPrompt": {
        "operationId": "e2381379-20b0-4fae-b341-afcdd8187323",
        "action": "playPrompt",
        "prompts": [
          {
            "value": "What is your flight ?",
            "voice": "male"
          }
        ]
      },
      "maxDurationInSeconds": 10,
      "initialSilenceTimeoutInSeconds": 5,
      "maxSilenceTimeoutInSeconds": 2,
      "playBeep": true,
      "stopTones": [
        "#"
      ]
    }
  ],
  "notificationSubscriptions": [
    "callStateChange"
  ]
}

If the recording has completed, your bot receives the following “record” outcome as the MIME multipart format.
Your bot can retrieve the audio binary from this raw data, and proceed some subsequent operations.

For example, Microsoft Bot framework is not having speech recognition feature itself (speech-to-text functionality), but you can get the text string value with external speech recognition service (like Bing Speech API), and you might also proceed the language understanding using LUIS (language understanding intelligent service).

POST https://example.com/yourbot/callback
Authorization: Bearer eyJ0eXAiOi...
Content-Type: multipart/form-data; boundary="test-0123"

--test-0123
Content-Type: application/json; charset=utf-8
Content-Disposition: form-data; name=conversationResult

{
  "id": "13107307-7bd6-4c5e-9a1b-65b98464cee6",
  "operationOutcome": {
    "type": "recordOutcome",
    "id": "efe617d7-4de5-42e9-b4e1-90dfd5850e49",
    "completionReason": "completedSilenceDetected",
    "lengthOfRecordingInSecs": 5.0459999999999994,
    "format": "wma",
    "outcome": "success"
  },
  "callState": "established"
}
--test-0123
Content-Type:audio/x-ms-wma
Content-Disposition:form-data; name=recordedAudio

(This is the audio binary ...)
--test-0123--

 

When you need to handle your real-time (stream) voice or video messages, please refer “Bot Framework : Build a real-time media bot for Skype” for details. (Here I don’t describe about real-time media bot.)

 

Build BOT with Authentication (Microsoft Bot Framework)

In the case that your bot needs to communicate with some 3rd party api (for ex, Facebook api, Office 365 api, twitter api, and Google api etc), your bot must show the login UI for the user and get some security information (like “token”) as the authenticated result. (If you’re using OAuth, your bot can call the api using OAuth token.)
Please run and see the “FreeBusy” or “AzureBot” in the Bot Directory. The “FreeBusy” bot launches the web browser for authentication and you can login Office 365 or other calendar app. After logged-in, you can view or create the appointment in your calendar from the chat in FreeBusy bot.
The “AzureBot” interacts with Microsoft Azure using Azure ARM rest api. This also launches the web browser for logging-in to Azure.

Today, I will show you how to build and design this kind of authentication bot.

SignIn Exprience in Bot

It’s very easy to show the Login UI from your bot. As I mentioned in my previous post “Rich messages with Microsoft Bot Framework (Rich text, Image, Card, etc)“, you can show the openUrl button (the button which type is “openUrl”) and this button launches the web browser (some specific url).

You can also use the signin card and signin button in the Bot Framework. (For the usage of the “card” and “button” in bot Framework, see the previous post “Rich messages with Microsoft Bot Framework“.)
Let’s see the following example. The following rest shows the following UI in chat app. This sample shows how to use the signin card and the signin button, and how it’s displayed in the Skype.
When the user clicks the “connect” button in the following UI, the web browser (the url is “https://contoso.com/login”) is opened in the new window. Even if you’re using Skype for phone, the same experience happens.

POST https://skype.botframework.com/v3/conversations/29%3A1iFtpwQb.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "",
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.signin",
      "content": {
        "text": "Please login to Office 365",
        "buttons": [
          {
            "type": "signin",
            "title": "Authentication Required",
            "value": "https://contoso.com/login"
          }
        ]
      }
    }
  ]
}

How to work with your Bot

The openUrl button or the signin button only launches the web browser with the specific url, and this doesn’t do anything more. You must design and implement how to interact with your bot after logging-in.

There’re several patterns for designing, and I will show you some examples for your hint.

Pattern A. Using bot state

As I described in my previous post “BUILD BOT with Microsoft Bot Framework Rest Api“, the Microsoft Bot Framework is having the built-in state infrastructure called “bot state service”. By using the bot state service, you can save and retrieve the state, scoped by either the user or the conversation.
For instance, if your bot save the user bot state in the bot state service, only this user can retrieve this state from your bot. (The other user cannot retrieve this state.)

Now let’s see the detailed steps of authentication with the bot state service. (In this case, we assume that the OAuth flow is used as authentication.)

First, your bot shows the button to launch your web application (web site). As part of the url, your bot should pass the bot user id (for example, like “https://contoso.com/login?userid=29%3a1iFtpwQb…“).

Next, when your web application (web site) is opened in the browser, the page is redirected to the login url.
After the user log-in (login succeeded), your web application might get some authenticated security token. Using Bot Framework api, your web application stores the given token as user state into the bot state service. (At this time, the user id is used for identifier.)
The user can close your web application (web browser).

Finally, when the user inputs some chat in your bot, your bot (in server side) can retrieve the previous token from the bot state service. (From now, this bot can call some api using this retrieved token.)

The following illustrates this authentication steps.

I created the super super simple example accomplishing this flow. Please run, and see the source code for your reference. (This sample uses the Bot Builder SDK for .NET.)

Try the sample bot ! (Skype)
https://join.skype.com/bot/ed6d70b2-ddc8-4962-aa70-553884677652
(You need your Office 365 account.)

Source code (Github)
https://github.com/tsmatsuz/AuthDemoBot

Notice : This sample is implementing the minimal code for accomplishing these steps, and I’m not implementing the additional code for scaling or security. (Do not copy and use this sample code in your production.)

Notice : The default state service is only having the basic functionalities and allows limited access to the state information. For the needs of the real production (storage management, scalability, etc), please implement your own custom state client or use Bot Builder SDK Azure Extensions. (When using Node.js, see here for the additional module.)

Pattern B. Match using some magic code in your bot

Instead of using the bot state, you can use your own keyword (unique code) and match the logged-in info (authenticated token, etc) to the bot user.

In this scenario, after the login succeeds, your web application issues the unique code (magic code), and stores both this code and the authenticated information (token, etc) in your own repository.

Your web application shows this magic code to the user (see the following screenshot), and the user copies this code and pastes into the bot chat.

As a result, the bot (server side) can check if this code is valid, and can retrieve the related information (token, etc) from the repository.

The following illustrates these steps.

For example, the “AzureBot” displays this kind of magic code.

Pattern C. Providing some magic code in your bot

Vice versa, your bot can provide (issue) some magic code in your chat, and your web application can verify this code and save the authenticated information. (In the previous pattern, the web application provided the magic code. In this case, the bot provides the magic code first.)

Especially, if you’re using the Azure Active Directory (Azure AD, i.e, Office 365, Dynamics CRM Online, etc), you can use the device login (OAuth device profile flow) with this integration pattern. The device login is often used when you’re on the environment without the graphical interface (like console app, printer, robot, etc). For example, the Azure CLI (command line interface) is also using this flow.

First, your bot can retrieve the 2 types of magic code called “user code” and “device code” from Azure AD, and provide the user code to the bot user.
Next, the user goes to the https://aka.ms/devicelogin. (The user can also use the browser in their own handy device like the smart phone.) This url shows the following screen, and the user input the user code on this screen. If this code is valid, the user can login (using id and password) to the Azure AD.

Finally, after logging-in to Azure AD, your bot can retrieve the Azure AD token from https://login.microsoftonline.com/common/oauth2/token using device code. (Azure AD matches the client using this code, and passes the authenticated result.)

Notice : This retrieval expires in 15 minutes (900 seconds).

For details about this flow in Azure AD, please see my previous post in “Azure AD : OAuth flow when you cannot show the login UI“. (I’m sorry this post was written in Japanese, then please search for some English contents.)

 

Of course, so much other scenarios (patterns) exist for authentication, but I’ve just shown some typical examples in this post for your reference.

Thanks,

Rich messages with Bot Framework (Rich text, Image, Card, etc)

In the previous post, I explained the rest style (raw) messaging format in Microsoft Bot Framework.
In this blog post I will show you the various rich and functional chat bot messages using several examples. These capabilities are almost introduced by Microsoft Bot Framework version 3.

Notice : As I explained in the previous post, the required attributes of json format in the REST (HTTP) is different at the channel (Skype, Facebook, etc). In the following examples I’m using the Skype channel.

Rich Text

You can use not only the plain text but the markdown. (The default message setting is markdown.)
For example, when you send the following HTTP request from your bot, the message is displayed in the chat app like following image (screenshot).

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "# Welcome to **Botland**nnPlease visit [my blog](https://blogs.msdn.microsoft.com/tsmatsuz).nn---nnThis is a test."
}

Of course, you can use this markdown in other channels (Slack, Facebook, etc), if the channel supports this types of the rich text. For example, the following screenshot is the result of the e-mail channel.
If you’re using channels which doesn’t support the rich text (like Facebook Messenger Bot, etc), this markdown doesn’t work.

If the channel is Skype, you can also use the XML text (i.e, HTML markup) as follows.
Note that the supported tag is only <b/>, <i/>, <u/>, <s/>, and <a/>.

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "Welcome to <b>Botland</b>. Please visit <a href="https://blogs.msdn.microsoft.com/tsmatsuz">my blog</a>."
}

You can also use the special <ss/> tag for displaying emoji (emoticons) in Skype.

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "<ss type ="wink">;)</ss>"
}

For details, please see the official document “Bot Framework SDK – Activities“.

Attachment

You can attach the media (image/audio/video/file) to your activities using a link to the content.

First I show you the very simple example of the attachment. (As I show you later, you can use more rich expression using the attachment.) The following is the image attachment example.

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "This is test attachment.",
  "attachments": [
    {
      "contentType": "image/jpg",
      "contentUrl": "http://mydeploy.azurewebsites.net/matsu.jpg",
      "name": "matsu.jpg"
    }
  ]
}

If you attach the binary, the attachment is shown as follows. (This is the Skype screenshot.)

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "This is test attachment.",
  "attachments": [
    {
      "contentType": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
      "contentUrl": "http://myfiles01.blob.core.windows.net/sample/test01.xlsx",
      "name": "test01.xlsx"
    }
  ]
}

Card

If the channel (Skype, Slack, Facebook, etc) is having the button, template, or some sort of these capabilities, you can use the Card as the attachment in Microsoft Bot Framework.
For example, the Skype and Facebook is having this kind of rich expressions. (see “Facebook Messenger Platform – Button Template“) And you can send the following HTTP request using the Card attachment.

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "attachmentLayout": "list",
  "text": "",
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.hero",
      "content": {
        "text": "What kind of sandwich would you like on your sandwich? ",
        "buttons": [
          {
            "type": "imBack",
            "title": "BLT",
            "value": "1"
          },
          {
            "type": "imBack",
            "title": "Black Forest Ham",
            "value": "2"
          },
          {
            "type": "imBack",
            "title": "Buffalo Chicken",
            "value": "3"
          }
        ]
      }
    }
  ]
}

If you’re using Skype Bot, the message is displayed as follows.

If it’s Facebook Messenger Bot, the message is displayed as follows.

If you send to e-mail channel, the message is displayed as follows.

If the user pushes this button in the Card, the selected value is replied to the bot as the text message. It’s same like the user’s text input.
These capabilities (button, template, etc) is different at each channel (Skype, Facebook, etc). But the Microsoft Bot Framework converts the expression to fit to each channel, and the developer doesn’t need to care about the differences between channels.
For example, if the Facebook channel is used, the Microsoft Bot Framework converts to the button template in Facebook Messenger Bot. (see “Facebook Messenger Platform – Button Template” for details.) If the e-mail channel is used, the Microsoft Bot Framework converts to the “mailto:” alink with several parameters.

More advanced messages by Card

I show you more rich expression using the Card capability.

First, you can use the 3 types of Card: Hero Card, Thumbnail Card, and Receipt Card. (Actually, there’s 4 types of Cards including the SignIn Card. I will show you the SignIn Card in the next post.)
Let’s see how it looks like.

Hero Card

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "Sample with a hero card",
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.hero",
      "content": {
        "title": "I'm a hero card",
        "subtitle": "Please visit my site.",
        "images": [
          {
            "url": "https://mydeploy.azurewebsites.net/matsu.jpg"
          }
        ],
        "buttons": [
          {
            "type": "openUrl",
            "title": "Go to my site",
            "value": "https://blogs.msdn.microsoft.com/tsmatsuz"
          }
        ]
      }
    }
  ]
}

Thumbnail Card

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "Sample with a thumbnail card",
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.thumbnail",
      "content": {
        "title": "I'm a thumbnail card",
        "subtitle": "Please visit my site.",
        "images": [
          {
            "url": "https://mydeploy.azurewebsites.net/matsu.jpg"
          }
        ],
        "buttons": [
          {
            "type": "openUrl",
            "title": "Go to my site",
            "value": "https://blogs.msdn.microsoft.com/tsmatsuz"
          }
        ]
      }
    }
  ]
}

Receipt Card

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "Sample with a receipt card",
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.receipt",
      "content": {
        "title": "I'm a receipt card",
        "items": [
          {
            "title": "Sushi",
            "subtitle": "2 piece",
            "image": {
              "url": "https://mydeploy.azurewebsites.net/sushi.png"
            },
            "price": "16.25",
            "quantity": "1"
          },
          {
            "title": "Tenpura",
            "subtitle": "1 dish",
            "image": {
              "url": "https://mydeploy.azurewebsites.net/tenpura.jpg"
            },
            "price": "34.50",
            "quantity": "2"
          }
        ],
        "total": "275.25",
        "tax": "27.52",
        "buttons": [
          {
            "type": "openUrl",
            "title": "Go to my site",
            "value": "https://blogs.msdn.microsoft.com/tsmatsuz"
          }
        ]
      }
    }
  ]
}

For the “type” attribute in the Card, you can use the following values for your needs. (see Bot Framework SDK – Attachments, Cards and Actions)
Note that the supported values is different at each channel. For example, the Skype supports “imBack”, “openUrl”, “call”, “showImage”, and “signin” now.

imBack Text of message which client will sent back to bot as ordinary chat message. All other participants will see that was posted to the bot and who posted this.
postBack Text of message which client will post to bot. Client applications will not display this message.
openUrl URL to be opened in the built-in browser.
call Destination for a call in following format: “tel:123123123123”
playAudio playback audio container referenced by URL
playVideo playback video container referenced by URL
showImage show image referenced by URL
downloadFile download file referenced by URL
signin OAuth flow URL. This is almost used in the SignIn Card. (Next time I will explain this capability.)

For example, when you send the following HTTP request, the call button (which calls to some Skype user) is displayed. If the user push this button, the Skype call to tsuyoshi.matsuzaki starts.

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "",
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.hero",
      "content": {
        "text": "Welcome to our help desk !",
        "buttons": [
          {
            "type": "call",
            "title": "Call to tsuyoshi.matsuzaki",
            "value": "skype:tsuyoshi.matsuzaki"
          }
        ]
      }
    }
  ]
}

You can set the multiple cards in one message. In such a case, each Card is displayed vertically (as “list”) by default.
If you want to show by horizontal carousel style, you can specify “carousel” style in “attachmentLayout” attribute.
Let’s see how it looks like.

POST https://skype.botframework.com/v3/conversations/29%3A1iFtp.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "attachmentLayout": "carousel",
  "text": "Please select your food.",
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.hero",
      "content": {
        "title": "Sushi",
        "subtitle": "Very fresh shrimp or tuna.",
        "images": [
          {
            "url": "https://mydeploy.azurewebsites.net/sushi.png"
          }
        ],
        "buttons": [
          {
            "type": "imBack",
            "title": "1 piece",
            "value": "sushi,1"
          },
          {
            "type": "imBack",
            "title": "2 piece",
            "value": "sushi,2"
          },
          {
            "type": "imBack",
            "title": "more",
            "value": "sushi,3+"
          }
        ]
      }
    },
    {
      "contentType": "application/vnd.microsoft.card.hero",
      "content": {
        "title": "Tenpura",
        "subtitle": "Japanese first-class vegitables.",
        "images": [
          {
            "url": "https://mydeploy.azurewebsites.net/tenpura.jpg"
          }
        ],
        "buttons": [
          {
            "type": "imBack",
            "title": "1 piece",
            "value": "tenpura,1"
          },
          {
            "type": "imBack",
            "title": "2 piece",
            "value": "tenpura,2"
          },
          {
            "type": "imBack",
            "title": "more",
            "value": "tenpura,3+"
          }
        ]
      }
    },
    {
      "contentType": "application/vnd.microsoft.card.hero",
      "content": {
        "title": "Tofu",
        "subtitle": "Super healthy food condensed by soy milk.",
        "images": [
          {
            "url": "https://mydeploy.azurewebsites.net/tofu.jpg"
          }
        ],
        "buttons": [
          {
            "type": "imBack",
            "title": "1 piece",
            "value": "tofu,1"
          },
          {
            "type": "imBack",
            "title": "2 piece",
            "value": "tofu,2"
          },
          {
            "type": "imBack",
            "title": "more",
            "value": "tofu,3+"
          }
        ]
      }
    }
  ]
}

For details about Cards, please see the official document “Bot Framework SDK – Attachments, Cards and Actions“.

Custom Channel Message – Rich messages for some specific channel

If you want to use the channel specific expression, you can use the “channelData” attribute instead of “attachments” attribute.

For example, if you want to use the flight update template in the Facebook Messenger Bot Platform, you send the following HTTP request.
Of course, this message is supported only in the Facebook channel.

POST https://facebook.botframework.com/v3/conversations/1245513475467461-737400853067333/activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "from": {
    "id": "737400853067333",
    "name": "mytestbot01"
  },
  "recipient": {
    "id": "1245513475467461",
    "name": "Tsuyoshi Matsuzaki"
  },
  "text": "This is test",
  "channelData": {
    "notification_type": "NO_PUSH",
    "attachment": {
      "type": "template",
      "payload": {
        "template_type": "airline_update",
        "intro_message": "Your flight is delayed",
        "update_type": "delay",
        "locale": "en_US",
        "pnr_number": "CF23G2",
        "update_flight_info": {
          "flight_number": "KL123",
          "departure_airport": {
            "airport_code": "SFO",
            "city": "San Francisco",
            "terminal": "T4",
            "gate": "G8"
          },
          "arrival_airport": {
            "airport_code": "AMS",
            "city": "Amsterdam",
            "terminal": "T4",
            "gate": "G8"
          },
          "flight_schedule": {
            "boarding_time": "2015-12-26T10:30",
            "departure_time": "2015-12-26T11:30",
            "arrival_time": "2015-12-27T07:30"
          }
        }
      }
    }
  }
}

For details about the custom channel message, please see the official document “Bot Framework SDK – Configuring Channels“.

 

When you’re using SDK (.NET), you can use the encapsulated object (class) called “Dialog” or “FormFlow” and this object (class) creates the previous messages internally if possible. These are the very high-level object. (You can also use the primitive elements using SDK.)
But, please remember that these essential elements which I explained here are working in the bottom of APIs.

Please enjoy your Bot development.

BUILD BOT with Microsoft Bot Framework Rest Api

This post describes the basic steps to build your bot using Microsoft Bot Framework 3 REST API.
You know that it’s very easy to build the bot by using language SDKs (.NET, Node.js). But, if you are using other programming languages like PHP, Ruby, Python with scientific libraries, etc, this post could be helpful for your development. You can build Skype bot, Slack bot, Facebook Messenger bot, in-app hosted bot, and more using any languages with the consistent programming model.

If you have ever built your bot using previous Skype Bot Platform or Microsoft Bot Framework 1.0, there exists important notice for the new Microsoft Bot Framework 3.
Microsoft Bot Framework is basically the common developer platform for building and hosting your bot (the concept is similar with Hubot), and this connects to each bot infrastructures (Slack Bot, Skype Bot, etc). Therefore, in the v1 endpoint, Microsoft Bot Framework and Skype Bot Platform was separated each other. (If you want to use Skype in Microsoft Bot Framework 1.0, you must setup the connection to the Skype Bot.)
On the other hand, Microsoft Bot Framework version 3 is involving the developer experience for “Skype Bot” too, and all Skype bot’s set-up can be done by Microsoft Bot Framework only. i.e, If you want to build your Skype bot, you can simply use this Microsoft Bot Framework 3 only. In addition, as I show you later in this post, several concepts of Bot Framework (calling pattern, authentication, etc) is similar to the original Skype Bot Platform. (The platform design of Microsoft Bot Framework has changed from v1.)

Notice : As I show you later, Skype Bot Platform is not retired and you can use the v3 endpoint of Skype Bot Platform.

Notice : Microsoft Bot Framework v2 is not public (internal build). v1 and v3 only.

Overview of Call Flow

Before you start, you must register your bot in dev portal (https://dev.botframework.com/bots). In this blog post, we assume that this registration is all done.
When you register your bot, your bot is also registered in App Registration Portal (https://apps.dev.microsoft.com) and you can get the client id and client secret for your bot. This data (client id and client secret) is used for the authentication which I describe later.

The following picture illustrates the calling flow of the Microsoft Bot Framework.
Microsoft Bot Framework provides the basic platform for bot hosting and connects to the each communication channel (Slack, Skype, etc) fronting on end-users through Bot Connector. Your code (your bot) interacts with this Microsoft Bot Framework in the backend. That is, your code (your bot) must communicate with Microsoft Bot Framework only.

If you connect to the channels (Slack, Facebook Messenger, etc), you must set up in the portal (https://dev.botframework.com/bots). But Skype Bot infrastructure (Skype channel) is initially connected to the Microsoft Bot Framework. (No need to extra work except for publishing to the Skype bot directory.)

Authentication Flow (outgoing – your code to bot framework)

Before starting communications, you must learn about the authentication for secure communications.

The messages to Bot Framework (from your bot) must be protected by Azure AD v2 endpoint, otherwise the malicious code might call the Microsoft Bot Framework instead of your bot.
In this section, I explain about how to accomplish this flow.

The Bot Framework uses the app-only access token in Azure AD v2 endpoint. To get this kind of access token, you just send the HTTP request as follows.
As I described before, you must retrieve the client_id and client_secret from the app registration portal beforehand and set as follows.

POST https://login.microsoftonline.com/botframework.com/oauth2/v2.0/token
Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials&client_id=1f7dd6e9-cbd7-4c38-adf2-6e9bcab5310a&client_secret=6wyxeJ4...&scope=https%3A%2F%2Fapi.botframework.com%2F.default

As a result you can receive the following HTTP response, and this includes the following access token. (Note that this access token expires in 1 hour.)

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "token_type": "Bearer",
  "expires_in": 3599,
  "ext_expires_in": 0,
  "access_token": "eyJ0eXAiOi..."
}

Hold this access token in your bot application, because you must always set this access token as “Authorization” header for your outgoing messages as follows. And Microsoft Bot Framework verifies this token for checking whether this request is sent from the valid (registered) bot.

Notice : This access token also includes the claims (client id, expiration, etc) for communications, and the Microsoft Bot Framework can identify your bot using this access token. (Please see the previous post of “How to verify the OAuth token with the v2.0 endpoint“.)

POST https://skype.botframework.com/v3/conversations/29%3A1iFtpwQ.../activities/6Bt4f5iryCI
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "timestamp": "2016-08-18T09:22:54.1811797Z",
  "from": {
    "id": "28:1f7dd6e9-cbd7-4c38-adf2-6e9bcab5310a",
    "name": "Echo Bot"
  },
  "conversation": {
    "id": "29:1iFtpwQ..."
  },
  "recipient": {
    "id": "29:1iFtpwQ...",
    "name": "Tsuyoshi Matsuzaki"
  },
  "text": "Hello !",
  "replyToId": "6Bt4f5iryCI"
}

Note : See “How to use Application Permission with v2 endpoint and Microsoft Graph” for the details about this Azure AD v2.0 endpoint client credential flow.

Authentication Flow (incoming – bot framework to your code)

The message from Microsoft Bot Framework is also protected by Authorization header as follows. (see the following header in bold fonts.) In this case, your bot must verify the message for secure communication. (If you ignored this header, your code might be called by the malicious code.)

POST https://example.com/yourbot
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "contactRelationUpdate",
  "id": "6Bt4f5iryCI",
  "timestamp": "2016-08-18T09:22:50.927Z",
  "serviceUrl": "https://skype.botframework.com",
  "channelId": "skype",
  "from": {
    "id": "29:1iFtpwQ...",
    "name": "Tsuyoshi Matsuzaki"
  },
  "conversation": {
    "id": "29:1iFtpwQ..."
  },
  "recipient": {
    "id": "28:1f7dd6e9-cbd7-4c38-adf2-6e9bcab5310a",
    "name": "Echo Bot"
  },
  "action": "add"
}

How to verify this header value ?
In this case, the Azure AD is not used. The key in Bot Framework is used for this authentication (AuthN and AuthZ).
First, you must retrieve the OpenID / OAuth configuration information hosted at https://api.aps.skype.com/v1/.well-known/openidconfiguration. It returns as follows. (Note that this might change in the future, then don’t copy this json result in your production code.)

Notice : If you’re using Emulator (debugging), you must use https://login.microsoftonline.com/common/v2.0/.well-known/openid-configuration (Azure AD v2) instead of https://api.aps.skype.com/v1/.well-known/openidconfiguration.

GET https://login.botframework.com/v1/.well-known/openidconfiguration
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "issuer": "https://api.botframework.com",
  "authorization_endpoint": "https://invalid.botframework.com",
  "jwks_uri": "https://login.botframework.com/v1/.well-known/keys",
  "id_token_signing_alg_values_supported": [
    "RSA256"
  ],
  "token_endpoint_auth_methods_supported": [
    "private_key_jwt"
  ]
}

The public key (X.509 certificate) is stored in the previous “jwks_uri” location. Then you must retrieve the key list and verify the previous “Authorization” header (access token).
As I explained in the previous post of “How to verify the OAuth token with the v2.0 endpoint“, this verification is accomplished by the simple steps.
Here is the PHP example for this verification.

<?php
// Authorization header value
$token = "eyJ0eXAiOi...";
// 0:Invalid, 1:Valid
$token_valid = 0;
  
// 1 separate token by dot (.)
$token_arr = explode('.', $token);
$headers_enc = $token_arr[0];
$claims_enc = $token_arr[1];
$sig_enc = $token_arr[2];

// 2 base 64 url decoding
$headers_arr = json_decode(base64_url_decode($headers_enc), TRUE);
$claims_arr = json_decode(base64_url_decode($claims_enc), TRUE);
$sig = base64_url_decode($sig_enc);

// 3 get key list
$keylist = file_get_contents('https://login.botframework.com/v1/.well-known/keys');
$keylist_arr = json_decode($keylist, TRUE);
foreach($keylist_arr['keys'] as $key => $value) {
  
  // 4 select one key (which matches)
  if($value['kid'] == $headers_arr['kid']) {
  
    // 5 get public key from key info
    $cert_txt = '-----BEGIN CERTIFICATE-----' . "n" . chunk_split($value['x5c'][0], 64) . '-----END CERTIFICATE-----';
    $cert_obj = openssl_x509_read($cert_txt);
    $pkey_obj = openssl_pkey_get_public($cert_obj);
    $pkey_arr = openssl_pkey_get_details($pkey_obj);
    $pkey_txt = $pkey_arr['key'];
    
    // 6 verify signature
    $token_valid = openssl_verify($headers_enc . '.' . $claims_enc, $sig, $pkey_txt, OPENSSL_ALGO_SHA256);
  }
}

// 7 show result
if($token_valid == 1)
  echo 'Token is Valid';
else
  echo 'Token is Invalid';

// Helper functions
function base64_url_decode($arg) {
  $res = $arg;
  $res = str_replace('-', '+', $res);
  $res = str_replace('_', '/', $res);
  switch (strlen($res) % 4) {
    case 0:
    break;
    case 2:
    $res .= "==";
    break;
    case 3:
    $res .= "=";
    break;
    default:
    break;
  }
  $res = base64_decode($res);
  return $res;
}
?>

Messaging Flow – Make contact with your bot

The authentication flow is all done ! All you have to do is to communicate using HTTP (REST-styled messaging) with Microsoft Bot Framework. Let’s see this flow.

First, if your bot is added to the contact list (subscribed) by a user, the following HTTP webhook arrives to your bot endpoint.
As I explained in the above, you must check the following “Authorization” header value and proceed your arbitrary actions.

Notice : In this example, I’m using the Skype.

POST https://example.com/yourbot
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "contactRelationUpdate",
  "id": "6Bt4f5iryCI",
  "timestamp": "2016-08-18T09:22:50.927Z",
  "serviceUrl": "https://skype.botframework.com",
  "channelId": "skype",
  "from": {
    "id": "29:1iFtpwQ...",
    "name": "Tsuyoshi Matsuzaki"
  },
  "conversation": {
    "id": "29:1iFtpwQ..."
  },
  "recipient": {
    "id": "28:1f7dd6e9-cbd7-4c38-adf2-6e9bcab5310a",
    "name": "Echo Bot"
  },
  "action": "add"
}

The “type” and “action” attributes mean what kind of bot action is published. In this case, this means that your bot is added to the user’s contact list.

The “from” attribute is the user id. In this case, this means the Skype user which unique id (not Skype Id) is “1iFtpwQ…”. Your bot must store this “from” id in your database, because your bot can communicate with each bot’s user (bot’s subscriber) using this id.
The “recipient” attribute is the destination id. In this example this indicates your bot which client id is “1f7dd6e9-cbd7-4c38-adf2-6e9bcab5310a”.
The “id” attribute is called activity id. Sometimes this id is refered by other communication. (I show you later.)

Notice : The number of “29” means the Skype user, and “28” means the bot.

If your bot accepts this request, you just response HTTP status 202.

HTTP/1.1 202 Accepted

Of course, you can reply some messages (for example, bot usage info, etc) against this adding message, and I will show you how to post outgoing messages later.

When your bot is removed from the contact list of some user, the following HTTP request (webhook) is received. (As you can see, the action attribute is set as “remove”.)
In this case you also response the HTTP status 202 as successful response.

POST https://example.com/yourbot
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "contactRelationUpdate",
  "id": "X4KtWvi9XS",
  "timestamp": "2016-08-18T09:48:19.201Z",
  "serviceUrl": "https://skype.botframework.com",
  "channelId": "skype",
  "from": {
    "id": "29:1iFtpwQ...",
    "name": "Tsuyoshi Matsuzaki"
  },
  "conversation": {
    "id": "29:1iFtpwQ..."
  },
  "recipient": {
    "id": "28:1f7dd6e9-cbd7-4c38-adf2-6e9bcab5310a",
    "name": "Echo Bot"
  },
  "action": "remove"
}

Messaging Flow – Incoming message

If the user sends the message “Good morning !” to your bot, the following HTTP webhook arrives.

POST https://example.com/yourbot
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "id": "4GhGAlkzDAK2I5lw",
  "timestamp": "2016-08-18T09:31:31.756Z",
  "serviceUrl": "https://skype.botframework.com",
  "channelId": "skype",
  "from": {
    "id": "29:1iFtpwQ...",
    "name": "Tsuyoshi Matsuzaki"
  },
  "conversation": {
    "id": "29:1iFtpwQ..."
  },
  "recipient": {
    "id": "28:1f7dd6e9-cbd7-4c38-adf2-6e9bcab5310a",
    "name": "Echo Bot"
  },
  "text": "Good morning !",
  "entities": []
}

This incoming message is very similar to the previous one, and I think there’s no need to explain about details.
If your bot accepts this request, you just response HTTP status 202.

HTTP/1.1 202 Accepted

Messaging Flow – Outgoing message

On the other hand, when your code send the outgoing message (which is the message from your bot to the user), you send the following HTTP request to Microsoft Bot Framework.
In this example I’m using Skype and the endpoint domain is https://skype.botframework.com. If you’re communicating Facebook, this domain should be https://facebook.botframework.com. (The domain differs by channels.)

POST https://skype.botframework.com/v3/conversations/29%3A1iFtpwQ.../activities/4GhGAlkzDAK2I5lw
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "timestamp": "2016-08-18T09:31:36.2281894Z",
  "from": {
    "id": "28:1f7dd6e9-cbd7-4c38-adf2-6e9bcab5310a",
    "name": "Echo Bot"
  },
  "conversation": {
    "id": "29:1iFtpwQ..."
  },
  "recipient": {
    "id": "29:1iFtpwQ...",
    "name": "Tsuyoshi Matsuzaki"
  },
  "text": "Good morning !",
  "replyToId": "4GhGAlkzDAK2I5lw"
}

The “29%3A1iFtpwQ…” in the uri fragment (which is url-encoded) is the conversation id. When your bot is sending the message to some user, this conversation id is the user id itself.
You can respond (reply) against some incoming message (i.e, bidirectional messaging). The above “4GhGAlkzDAK2I5lw” is the incoming “id” attribute (i.e, activity id), and this sample is responding against this incoming message.

On the contrary, you can call the user using one-way style messaging like timer bot or some notification bot. If you do so, you must use the activity id with blank as follows.

POST https://skype.botframework.com/v3/conversations/29%3A1iFtpwQ.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "timestamp": "2016-08-18T09:31:36.2281894Z",
  "from": {
    "id": "28:1f7dd6e9-cbd7-4c38-adf2-6e9bcab5310a",
    "name": "Echo Bot"
  },
  "conversation": {
    "id": "29:1iFtpwQ..."
  },
  "recipient": {
    "id": "29:1iFtpwQ...",
    "name": "Tsuyoshi Matsuzaki"
  },
  "text": "Good morning !"
}

If Microsoft Bot Framework accepts this message, HTTP status 202 is returned. (As I explained, the “Authorization” header is checked by the framework.)

HTTP/1.1 202 Accepted

There exists some additional notice about outgoing messages.
First, almost all attributes is optional (see Microsoft Bot Connector API v3.0 document) and you can omit (skip) several attributes and can use the minimal attributes.
For example, the following is the omitted request using Skype channel. But, if you’re using Facebook channel, the “from” attribute is required for communication. That is, the required attributes differs by each channels.

POST https://skype.botframework.com/v3/conversations/29%3A1iFtpwQ.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message",
  "text": "Good morning !"
}

Second, if you’re communicating with only Skype, you can also use the Skype Bot Platform v3 endpoint alternatively.
Please see the following example. This uses https://apis.skype.com as endpoint domain instead of https://skype.botframework.com. As you can see, you must also keep in mind that several detailed specification (see the following value of “type” attribute and the HTTP response code) is different from Microsoft Bot Framework. (Similar, but different !)

POST https://apis.skype.com/v3/conversations/29%3A1iFtpwQ.../activities
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "type": "message/text",
  "text": "Good morning !"
}
HTTP/1.1 201 Created

State Handling

Microsoft Bot Framework itself is having the state infrastructure called “Bot State Service”. With this infrastructure you can build the stateful bot with scaling.
Now I show you how to use this state with rest api.

When you want to set user state in Bot State Service, you send the following HTTP request against Bot State Service endpoint (https://state.botframework.com).
The uri must be /v3/botstate/{channelId}/users/{userId}. The following example is using the skype as the bot channel.

POST https://state.botframework.com/v3/botstate/skype/users/29%3A1iFtpwQ...
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "eTag": "*",
  "data": {
    "DemoData1": "Test Data 1"
  }
}
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "data": {
    "DemoData1": "Test Data 1"
  },
  "eTag": "W/"datetime'2016-08-18T10%3A12%3A45.4398159Z'""
}

Saved data is stored in the state service, and you can pick up the state data by calling GET method.

GET https://state.botframework.com/v3/botstate/skype/users/29%3A1iFtpwQ...
Authorization: Bearer eyJ0eXAiOi...
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "data": {
    "DemoData1": "Test Data 1"
  },
  "eTag": "W/"datetime'2016-08-18T10%3A12%3A45.4398159Z'""
}

The bot state is having 2 types of scope. One is the user-scoped state (the same as previous example), and another is the conversation-scoped state. If you want to share the data across the all conversations (Skype conversation, Facebook conversation, etc), you must use the user state. If not, you must use the conversation state.

When you want to use the conversation-scoped state, you send the HTTP request to /v3/botstate/{channelId}/conversations/{conversationId} instead of /v3/botstate/{channelId}/users/{userId} as follows.

POST https://state.botframework.com/v3/botstate/skype/conversations/29%3A1iFtpwQ...
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

{
  "eTag": "*",
  "data": {
    "DemoData2": "Test Data 2"
  }
}

Note that these data objects will fail to be stored if another instance of your bot has changed the object already.

Notice : Even if it’s the user state, the state is not shared with another bot. (It’s secured.) The different state in the different bot.

Notice : If you’re using Bot Builder SDK, the serializable Dialog is persisted in the bot state service through IBotDataStore interface by default.

 

In this blog post I’ve just explained the basic concept and steps building your bot with rest api. As you see, don’t worry about the supported programming language in SDK, and please enjoy your bot development everyone !
Microsoft Bot Framework can handle more advanced scenarios like the attachment, rich text, card, and audio, etc. (The video in Skype will be available in the future release.) Next I will explain about these advanced scenarios.

For more details (api reference, arguments, etc), please see the following official document.  Thanks

Bot REST API Authentication
https://docs.botframework.com/en-us/restapi/authentication/

Bot Connector REST API 3.0
https://docs.botframework.com/en-us/restapi/connector/

Bot State REST API 3.0
https://docs.botframework.com/en-us/restapi/state/

Build Your Skype Bot with OAuth and REST

This page will be redirected to the “BUILD BOT with Microsoft Bot Framework Rest Api” after 10 seconds.
Because the Skype Bot development has moved to the new Microsoft Bot Framework 3. (Please refer the new post.)

As you know, you can use the .NET (C#) and node.js SDKs for building your Skype bot. But for other programming languages like PHP, Ruby, Python with scientific libraries etc, you must handle the HTTP and OAuth directly.
In this blog post I explain about this HTTP flow (incl. AuthN) of Skype Bot without SDK. Using this flow, you can develop Skype Bot with any programming language.

Beforehand, you must register your bot (here), and please see the tutorial document. (In this blog post we assume that this registration is done.)

Basic flow of calling

The following picture illustrates the calling flow in the Skype Bot Platform.
This platform (Skype Bot Platform) provides the basic bot infrastructures (Bot Directory, etc) fronting on end-users, and your code is called through this platform in the backend.
That is, your code (your bot) must communicate with this Skype Bot Platform.

If you are using Microsoft Bot Framework with Skype, eventually the Microsoft Bot Framework connects to this platform. See the previous post of “HTTP Flow of Microsoft Bot Framework” for details.

Authentication (outgoing – your code to skype)

Before starting messaging communications, you must learn about the authentication for secure communications.

The message from your bot must be protected by Azure AD v2 endpoint, otherwise the malicious code might call the Skype Bot Platform instead of your bot.
In this section, I explain about how to accomplish this flow.

Skype Bot uses the app-only access token in Azure AD v2 endpoint. To get this kind of access token, you just send the HTTP request as follows. (Beforehand, you must retrieve the following client_id and client_secret from the app registration portal. These are the bot’s client id and secret.)

POST https://login.microsoftonline.com/common/oauth2/v2.0/token
Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials&client_id=761b6e57-2dc1-4de9-b5d1-9599a9902117&client_secret=3ayhQ55FqU...&scope=https%3A%2F%2Fgraph.microsoft.com%2F.default

As a result, you can receive the following HTTP response, and this includes the following access token. (Notice : This expires in 1 hour.)

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "token_type": "Bearer",
  "expires_in": 3600,
  "ext_expires_in": 3600,
  "access_token": "eyJ0eXAiOi..."
}

Please hold this access token in your bot application, because you must always set this access token as Authorization header for your outgoing messages, and the Skype Bot Platform verifies this token for checking if the valid code is sending.

Notice : This access token also includes the claims (client id, etc) for communications, and the Skype Bot Platform can identify your bot using this access token. (Please see the previous post of “How to verify the OAuth token with the v2.0 endpoint” for retrieving this claim information.)

POST https://apis.skype.com/v2/conversations/8:live:tsuyoshi.matsuzaki/activities/
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json

{"message":{"content":"How are you ?"}}

Note : See “How to use Application Permission with v2 endpoint and Microsoft Graph” for the details about this Azure AD v2.0 endpoint client credential flow.

Authentication (incoming – skype to your code)

The message from Skype Bot Platform is also protected by Authorization header. (see the following header in bold fonts.) In this case, your bot must verify the message for secure communications by yourself. (If not, your code might be called by the malicious code.)

POST https://example.com/yourbot
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

[
  {
    "action": "add",
    "activity": "contactRelationUpdate",
    "from": "8:live:tsuyoshi.matsuzaki",
    "fromDisplayName": "Tsuyoshi Matsuzaki",
    "to": "28:761b6e57-2dc1-4de9-b5d1-9599a9902117",
    "time": "2016-07-11T06:30:09.512Z"
  }
]

In this case, the Azure AD is not used. The key in Skype Bot Platform is used for this authentication (AuthN and AuthZ).
First, you must retrieve the OpenID / OAuth configuration information of the Skype Bot Platform hosted at https://api.aps.skype.com/v1/.well-known/openidconfiguration. It returns as follows. (Notice : This might change in the future, so don’t copy this json value in your production code.)

{
  "issuer": "https://api.botframework.com",
  "authorization_endpoint": "https://anonymous.invalid.com",
  "jwks_uri": "https://api.aps.skype.com/v1/keys",
  "id_token_signing_alg_values_supported": [
    "RSA256"
  ],
  "token_endpoint_auth_methods_supported": [
    "private_key_jwt"
  ]
}

The public key (X.509 certificate) is stored in the previous “jwks_uri” location. Then you must retrieve these key information and you verify the previous “Authorization” header (access token) using this key.

As I explained in the previous post of “How to verify the OAuth token with the v2.0 endpoint“, this verification is accomplished by the simple steps.
Here is the PHP sample code for this verification.

<?php
// Authorization header value
$token = "eyJ0eXAiOi...";
// 0:Invalid, 1:Valid
$token_valid = 0;
  
// 1 separate token by dot (.)
$token_arr = explode('.', $token);
$headers_enc = $token_arr[0];
$claims_enc = $token_arr[1];
$sig_enc = $token_arr[2];

// 2 base 64 url decoding
$headers_arr = json_decode(base64_url_decode($headers_enc), TRUE);
$claims_arr = json_decode(base64_url_decode($claims_enc), TRUE);
$sig = base64_url_decode($sig_enc);

// 3 get key list
$keylist = file_get_contents('https://api.aps.skype.com/v1/keys');
$keylist_arr = json_decode($keylist, TRUE);
foreach($keylist_arr['keys'] as $key => $value) {
  
  // 4 select one key (which matches)
  if($value['kid'] == $headers_arr['kid']) {
  
    // 5 get public key from key info
    $cert_txt = '-----BEGIN CERTIFICATE-----' . "n" . chunk_split($value['x5c'][0], 64) . '-----END CERTIFICATE-----';
    $cert_obj = openssl_x509_read($cert_txt);
    $pkey_obj = openssl_pkey_get_public($cert_obj);
    $pkey_arr = openssl_pkey_get_details($pkey_obj);
    $pkey_txt = $pkey_arr['key'];
    
    // 6 verify signature
    $token_valid = openssl_verify($headers_enc . '.' . $claims_enc, $sig, $pkey_txt, OPENSSL_ALGO_SHA256);
  }
}

// 7 show result
if($token_valid == 1)
  echo 'Token is Valid';
else
  echo 'Token is Invalid';

// Helper functions
function base64_url_decode($arg) {
  $res = $arg;
  $res = str_replace('-', '+', $res);
  $res = str_replace('_', '/', $res);
  switch (strlen($res) % 4) {
    case 0:
    break;
    case 2:
    $res .= "==";
    break;
    case 3:
    $res .= "=";
    break;
    default:
    break;
  }
  $res = base64_decode($res);
  return $res;
}
?>

HTTP flow – Adding your bot

The authentication flow is all done ! All you have to do is to communicate using HTTP (REST-styled messaging) with the Skype Bot Platform.

First, if your bot is added (subscribed) by a Skype user (customer), the following HTTP webhook arrives to your bot.
As I explained in the above, you must check the Authorization header value and proceed your actions.

POST https://example.com/yourbot
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

[
  {
    "action": "add",
    "activity": "contactRelationUpdate",
    "from": "8:live:tsuyoshi.matsuzaki",
    "fromDisplayName": "Tsuyoshi Matsuzaki",
    "to": "28:761b6e57-2dc1-4de9-b5d1-9599a9902117",
    "time": "2016-07-11T06:30:09.512Z"
  }
]

The “action” attribute means what kind of bot action is published. In this case, this means that the user added your bot.
The “from” is the Skype user id, and the “to” is your bot id. Your bot must store this “from” id in your database, because your bot can communicate with your bot’s users (the bot’s subscribers) using this id.

Notice : The number of “8” means the skype user, and “28” means the skype bot.

If your bot accepts this request, you just response HTTP status 201.

HTTP/1.1 201 Created

When your bot is removed from the contact list, the following HTTP request (webhook) is received. (In this case you just also response the HTTP status 201.)

POST https://example.com/yourbot
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

[
  {
    "action": "remove",
    "activity": "contactRelationUpdate",
    "from": "8:live:tsuyoshi.matsuzaki",
    "fromDisplayName": "Tsuyoshi Matsuzaki",
    "to": "28:761b6e57-2dc1-4de9-b5d1-9599a9902117",
    "time": "2016-07-12T09:52:36.574Z"
  }
]

HTTP flow – Incoming message

If the Skype user sends the message “Good morning !” to your bot, the following HTTP webhook arrives.

POST https://example.com/yourbot
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json; charset=utf-8

[
  {
    "id": "0",
    "content": "Good morning !",
    "activity": "message",
    "from": "8:live:tsuyoshi.matsuzaki",
    "fromDisplayName": "Tsuyoshi Matsuzaki",
    "to": "28:761b6e57-2dc1-4de9-b5d1-9599a9902117",
    "time": "2016-07-12T06:26:08.231Z"
  }
]

This incoming message is very similar to the privious one, then I think there’s no need to explain about details. But there’s some notation for this incoming messages.

The “id” is called “activity id”. Sometimes this id is refered by other communication.
For example, when this incoming message is for attachment (images, etc), you can ask for the binary data (which is encoded by base64) using this id.
When not refered, this id is “0”.

As you notice, the message body uses the json array. When there are multiple messages, these can be included in one request using the json array.

If your bot accepts this request, you just response HTTP status 201.

HTTP/1.1 201 Created

HTTP flow – Outgoing message

On the other hand, when your code send the outgoing message (which is the message from your bot to the Skype user), you send the following HTTP request to the Skype Bot Platform.
Each HTTP request is one-way (not request-reply), and it can be bidirectional (incoming and outgoing). Your code can also call the Skype user using one-way messaging like timer bot or some notification bot.  (There’s no need to use request-reply style messaging.)

POST https://apis.skype.com/v2/conversations/8:live:tsuyoshi.matsuzaki/activities/
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json

{"message":{"content":"Good morning !"}}

The “8:live:tsuyoshi.matsuzaki” in the uri fragment is the conversation id. When your bot is sending the message to one Skype user, this id can be Skype Id itself. When group conversation, conversation thread id will be used.

If the Skype Bot Platform accepts this incoming message, HTTP status 201 is returned. (As I explained, the Authorization header is checked by the platform.)

HTTP/1.1 201 Created

You can also use emoji, attachment, and so on. Next is the emoji example.

POST https://apis.skype.com/v2/conversations/8:live:tsuyoshi.matsuzaki/activities/
Authorization: Bearer eyJ0eXAiOi...
Content-Type: application/json

{"message":{"content":"<ss type ="wink">;)</ss>"}}

 

In this blog post I’ve just explained the basic concepts and steps building Skype Bot, but Skype Bot can handle more advanced scenarios like the attachment, group chat, and audio/video calling, etc. (The audio is the limited preview, and the video will be in the future release.)

Don’t worry about the supported programming language in SDK, and please try !