How to use Azure REST API with Certificate programmatically (without interactive Login UI)

This post is just tips for Azure programming. (One of frequently asked questions by developers)

A long ago, I introduced how to use Azure REST API (in resource manager mode) and role model called RBAC (Role Based Access Control). (See my old post “Manage your Azure resource with REST API (ARM)“.)
In usual cases, API is called with the HTTP header of Azure AD user token, which is retrieved by the OAuth flow and user’s interactive login activity.

But, what if your application is some kind of the backend process like daemon ?

In this post I explain about this scenario as follows.

Note : If you want to run some kind of automation jobs (scheduled or triggered) for Azure resources, you can register your job (scripts) in Azure Automation. Of course, you don’t need to login using interactive UI for running the Automation job.
Here it’s assumed that we access Azure resources from outside of Azure.

What is concerns ?

In terms of AuthN/AuthZ flow in Azure Active Directory (Azure AD), you can use application permissions in order to access some API protected by Azure AD from the backend service like daemon. For more details about application permissions, see “Develop your backend server-side application (Deamon, etc) protected by Azure AD without login UI” (post in Japanese).

For example: if you want to create your application which sync all users’ information (in some specific tenant) in the background periodically, you set the application permission like following screenshot.

If you want to sync all users’ calendar data in some tenant in the background, select the following application permission. (See the following screenshot.)

But unfortunately there’s no application permission for Azure Management API ! Look at the following screenshot.
So, what should we do ?

The answer is “use service principal and RBAC” (not application permissions).

What is the role based access control (RBAC) ? As we saw in my old post “Manage your Azure resource with REST API (ARM)“, RBAC is used for the Azure resource’s access permissions in new resource manager (ARM) mode.
“Role” (ex: Reader role, Contributor role, Backup Operator role, etc, etc) is some set of Azure permissions, and you can assign some role to some users or groups. That is, RBAC provides the granular permissions for Azure resources.
This role assignment cannot only be set in each resource (storage, virtual machine, network, etc), but also can be set in subscription level or resource group level. If some role is assigned in resource group level, the user can access all the resources in this resource group. (i.e, the assignment is inherited through your subscription, resource groups, and each resources.)
What you should remember is that you can assign role to service principal, not only users or groups !

Let’s see the configuration and usage of these combination (together with service principal and RBAC).

Step1 – Register service principal

Here I describe how to configure the service principal.

First you add new service principal in Azure AD tenant as follows.
Login to Azure Portal (https://portal.azure.com), click “Azure Active Directory” on the left navigation, and click “App registrations” menu.

Press “Add” button, and register the new app (i.e, service principal) by pushing “Create” button.

The service principal is successfully generated.
Finally, please copy the application id and app id uri of this service principal (app) in Azure Portal. (We use these values later.)

Step2 – Set Azure permissions with service principal in RBAC

Next, please assign some role to this service principal in Azure Portal.
The following is this procedure. Here in this post, I set the read permission for all the resources in some specific resource group. (We set “Reader” role.)

  1. View the resource group in Azure Portal.
  2. Press “Access control (IAM)”
  3. Press “Add” button.
    Select “Reader” as role and select the previous service principal as members.
  4. Press “Save” button.

Step3A – Retrieve access token with app key

You can retrieve the access token (which is needed for calling Azure rest api) by either app key (app password, client secret) or certificate.
First we explain the case of app key.

To create the app key for your service principal, just you create the key in your service principal with Azure Portal. Select [Settings] and [Keys] in Azure AD application management, and you can generate the key (password string) for your service principal.

For retrieving access token, you just publish the following HTTP request. (The value of client secret is the previously generated app key.)
Note that you must use the tenant-aware url instead of using “https://login.microsoftonline.com/common/oauth2/token”.

HTTP Request (Get access token)

POST https://login.microsoftonline.com/yourtenant.onmicrosoft.com/oauth2/token
Accept: application/json
Content-Type: application/x-www-form-urlencoded

resource=https%3A%2F%2Fmanagement.core.windows.net%2F
&client_id=1c13ff57-672e-4e76-a71d-2a7a75890609
&client_secret=9zVm9Li1...
&grant_type=client_credentials

HTTP Response (Get access token)

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "token_type": "Bearer",
  "expires_in": "3600",
  "ext_expires_in": "10800",
  "expires_on": "1488429872",
  "not_before": "1488425972",
  "resource": "https://management.core.windows.net/",
  "access_token": "eyJ0eXAiOi..."
}

The returned value of access_token attribute is the access token for your Azure REST API calls. Later we use this token value.

Step3B – Retrieve access token with certificate

If you want to use certificate (instead of app key), first you must create and register the certificate into this service principal.
Here we create the self-signed certificate for demo purpose with the following command. (We use makecert utility in Windows SDK.)
As a result, 3 files named “server.pvk”, “server.cer”, and “server.pfx” are generated with this command.

rem -- create self signed CA cert (CA.pvk, CA.cer)
makecert -r -pe -n "CN=My Root Authority" -ss CA -sr CurrentUser -a sha1 -sky signature -cy authority -sv CA.pvk CA.cer

rem -- create self signed server cert (server.pvk, server.cer, server.pfx)
makecert -pe -n "CN=DemoApp" -a sha1 -sky Exchange -eku 1.3.6.1.5.5.7.3.1 -ic CA.cer -iv CA.pvk -sp "Microsoft RSA SChannel Cryptographic Provider" -sy 12 -sv server.pvk server.cer
pvk2pfx -pvk server.pvk -spc server.cer -pfx server.pfx -pi {password}

The generated “server.cer” file includes the public key. You can get this encoded raw data and base64 encoded thumbprint by the following PowerShell command, and please copy these strings. The result is in raw.txt and thumbprint.txt.

Note that the base64 encoded thumbprint is not the familiar hexadecimal thumbprint string. If you have the hexadecimal thumbprint, you can get this string by converting to the binary and encoding with base64. (You can use online site like here for this encoding …)

$cer =
  New-Object System.Security.Cryptography.X509Certificates.X509Certificate2
$cer.Import("C:Demotestserver.cer")

# Get encoded raw
$raw = $cer.GetRawCertData()
$rawtxt = [System.Convert]::ToBase64String($raw)
echo $rawtxt > raw.txt

# Get thumbprint
$hash = $cer.GetCertHash()
$thumbprint = [System.Convert]::ToBase64String($hash)
echo $thumbprint > thumbprint.txt

Note : You can also get key pair and .pfx with openssl as follows.
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -config "C:Program Files (x86)Gitsslopenssl.cnf"
openssl pkcs12 -export -out server.pfx -inkey key.pem -in cert.pem
You can also get the hexadecimal thumbprint as follows, and convert to base64 encoded thumbprint.
openssl x509 -in cert.pem -fingerprint -noout

In Azure Portal, please view the previously registered app in Azure AD, and press “Manifest” button. (The application manifest is showed in the editor.)

In the manifest editor, please add the following key value (bold fonts) in the manifest. Here customKeyIdentifier is the base64 encoded thumbprint of certificate (which was retrieved by the previous command), and value is the raw data. Moreover you must create unique GUID for the following keyId.

After that, please press the “Save” button.

{
  "appId": "1c13ff57-672e-4e76-a71d-2a7a75890609",
  "appRoles": [],
  "availableToOtherTenants": false,
  "displayName": "test01",
  "errorUrl": null,
  "groupMembershipClaims": null,
  "homepage": "https://localhost/test01",
  "identifierUris": [
    "https://microsoft.onmicrosoft.com/b71..."
  ],
  "keyCredentials": [
    {
      "customKeyIdentifier": "CTTz0wG...",
      "keyId": "9de40fd2-9559-4b52-b075-04ab17227411",
      "type": "AsymmetricX509Cert",
      "usage": "Verify",
      "value": "MIIDFjC..."
    }
  ],
  "knownClientApplications": [],
  "logoutUrl": null,
  ...

}

Now you start to build your backend (daemon) application.
Before you connect to the Azure resources using REST API, your program must take the access token for the REST API calls.

First you must create the RS256 digital signature using the previous private key (server.pfx). The input string (payload) for this signature must be the base64 uri encoded string of the following 2 tokens delimited by dot ( . ) character. That is, if it’s assumed that the base64 uri encoded string of {"alg":"RS256","x5t":"CTTz0..."} (1st token) is eyJhbGciOi*** (it’ll be large string and here we’re omitting…) and the base64 uri encoded string of {"aud":"https:...","exp":1488426871,"iss":"1c13f...",...} (2nd token) is eyJhdWQiOi***, then the input string must be eyJhbGciOi***.eyJhdWQiOi***. You create digital signature using this input string and previously generated key.

  • x5t is the certificate thumbprint which is previously retrieved by PowerShell command.
  • nbf is the start time (epoch time) of this token expiration, and exp is the end time.
  • iss and sub is your application id (client id).
  • jti is the arbitary GUID which is used for protecting reply attacks.

1st token

{
  "alg": "RS256",
  "x5t": "CTTz0wGaBvl1qhHEmVdw04vExqw"
}

2nd token

{
  "aud": "https://login.microsoftonline.com/microsoft.onmicrosoft.com/oauth2/token",
  "exp": 1488426871,
  "iss": "1c13ff57-672e-4e76-a71d-2a7a75890609",
  "jti": "e86b1f2b-b001-4630-86f5-5f953aeec694",
  "nbf": 1488426271,
  "sub": "1c13ff57-672e-4e76-a71d-2a7a75890609"
}

In order to create the RS256 digital signature for that input string (payload), you can use some libraries. For example: you can use openssl_sign (which needs pem format private key) for PHP programmers, or you might be able to use jsjws for JavaScript. For C# (.NET), I’ll show the complete code later in this chapter.
For more details about this format, please see “Build your API protected by Azure AD (How to verify access token)” (sorry, it’s written in Japanese).

After you’ve created the signature, you can now get the client assertion as follows. This format is the standardized JWT format (see RFC 7519).

{base64 uri encoded string of 1st token}.{base64 uri encoded string of 2nd token}.{base64 uri encoded of generated signature}

Finally you can get access token for Azure REST API with the following HTTP request.
The each attributes are :

  • The url fragment yourtenant.onmicrosoft.com is your tenant domain. In this application flow, you cannot use https://login.microsoftonline.com/common/oauth2/token instead.
  • https://management.core.windows.net/ is the resource id of the Azure REST API (fixed value). The value of resource must be url-encoded string.
  • 1c13ff57-672e-4e76-a71d-2a7a75890609 is the application id of your service principal.
  • eyJhbGciOi... is the client assertion which is previously created. (See above)

HTTP Request (Get access token)

POST https://login.microsoftonline.com/yourtenant.onmicrosoft.com/oauth2/token
Accept: application/json
Content-Type: application/x-www-form-urlencoded

resource=https%3A%2F%2Fmanagement.core.windows.net%2F
&client_id=1c13ff57-672e-4e76-a71d-2a7a75890609
&client_assertion_type=urn%3Aietf%3Aparams%3Aoauth%3Aclient-assertion-type%3Ajwt-bearer
&client_assertion=eyJhbGciOi...
&grant_type=client_credentials

The client assertion is checked by the stored raw key (public key), and if it’s verified, the result is successfully returned as follows.

HTTP Response (Get access token)

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "token_type": "Bearer",
  "expires_in": "3600",
  "ext_expires_in": "10800",
  "expires_on": "1488429872",
  "not_before": "1488425972",
  "resource": "https://management.core.windows.net/",
  "access_token": "eyJ0eXAiOi..."
}

The returned value of access_token attribute is the access token for your Azure REST API calls.

As we described above, we saw how to get access token with raw HTTP request.
But if you’re using C# (.NET), you can get access token by a few lines of code as follows with ADAL (Microsoft.IdentityModel.Clients.ActiveDirectory libarary).

...
using Microsoft.IdentityModel.Clients.ActiveDirectory;
using System.Security.Cryptography.X509Certificates;
...

AuthenticationContext ctx = new AuthenticationContext(
  "https://login.microsoftonline.com/yourtenant.onmicrosoft.com/oauth2/authorize",
  false);
X509Certificate2 cert = new X509Certificate2(
  @"C:tmpserver.pfx",
  "P@ssw0rd", // password of private key
  X509KeyStorageFlags.MachineKeySet);
ClientAssertionCertificate ast = new ClientAssertionCertificate(
  "1c13ff57-672e-4e76-a71d-2a7a75890609", // application id
  cert);
var res = await ctx.AcquireTokenAsync(
  "https://management.core.windows.net/",  // resource id
  ast);
MessageBox.Show(res.AccessToken);

Step4 – Connect Azure resources !

Now let’s connect using Azure REST API.

Here we’re getting a Virtual Machine resource using Azure REST API. The point is to set the value of access token as HTTP Authorization header as follows.

HTTP Request (Azure REST API)

GET https://management.azure.com/subscriptions/b3ae1c15-...
  /resourceGroups/TestGroup01
  /providers/Microsoft.Compute/virtualMachines
  /testmachine01?api-version=2017-03-30
Accept: application/json
Authorization: Bearer eyJ0eXAiOi...

HTTP Response (Azure REST API)

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "properties": {
    "vmId": "525f07f9-d4e9-40dc-921f-6adf4ebb8f21",
    "hardwareProfile": {
      "vmSize": "Standard_A2"
    },
    "storageProfile": {
      "imageReference": {
        "publisher": "MicrosoftWindowsServer",
        "offer": "WindowsServer",
        "sku": "2012-R2-Datacenter",
        "version": "latest"
      },
      "osDisk": {
        "osType": "Windows",
        "name": "testmachine01",
        "createOption": "FromImage",
        "vhd": {
          "uri": "https://test01.blob.core.windows.net/vhds/testvm.vhd"
        },
        "caching": "ReadWrite"
      },
      "dataDisks": []
    },
    "osProfile": {
      "computerName": "testmachine01",
      "adminUsername": "tsmatsuz",
      "windowsConfiguration": {
        "provisionVMAgent": true,
        "enableAutomaticUpdates": true
      },
      "secrets": []
    },
    "networkProfile": {
      "networkInterfaces":[
        {
          "id":"..."
        }
      ]
    },
    "diagnosticsProfile": {
      "bootDiagnostics": {
        "enabled": true,
        "storageUri": "https://test01.blob.core.windows.net/"
      }
    },
    "provisioningState": "Succeeded"
  },
  "resources": [
    {
      "properties": {
        "publisher": "Microsoft.Azure.Diagnostics",
        "type": "IaaSDiagnostics",
        "typeHandlerVersion": "1.5",
        "autoUpgradeMinorVersion": true,
        "settings": {
          "xmlCfg":"PFdhZENmZz...",
          "StorageAccount":"test01"
        },
        "provisioningState": "Succeeded"
      },
      "type": "Microsoft.Compute/virtualMachines/extensions",
      "location": "japaneast",
      "id": "...",
      "name": "Microsoft.Insights.VMDiagnosticsSettings"
    }
  ],
  "type": "Microsoft.Compute/virtualMachines",
  "location": "japaneast",
  "tags": {},
  "id": "...",
  "name": "testmachine01"
}

Of course, you can also execute create/update/delete or some other actions (manage, etc) along with the role assignment in your Azure subscription.

 

Advertisements

2 thoughts on “How to use Azure REST API with Certificate programmatically (without interactive Login UI)

  1. I receive this error when I’m following all your steps : AADSTS50027: Invalid JWT token. AADSTS50027: Invalid JWT token. No certificate thumbprint specified in token header

    Can you please help me with this.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s