How to use Azure Storage without SDK

By Tsuyoshi Matsuzaki on 2016-07-06 • ( 4 Comments )

Note (Added on June 2018) :
Now you can also access Azure blob by Azure AD token without the following shared key. (You can simply run backend integration using Managed Service Identity (MSI) or Azure AD service principal.)
See this announcement in team blog.

==========

If you want to get, add, and update objects in Azure storage (Blob, Table, Queue, Files), of course, you can manipulate these objects using Azure SDK (Node.js, .NET, PHP, Python, Java, etc).
Actually the easiest way to access Azure Storage using programming language is to use the Azure SDK (libraries), but what if we suddenly encounter the case that we cannot depend on these libraries ? For example, the case of using the programming language which is having no Azure SDK library, the case of distribution issue due to the size or other reasons, etc…

Calling REST APIs

In such a case, you can directly call REST API. This REST API provides all fundamental operations against Azure Storage, and Azure SDK is also calling these REST APIs in the bottom.

MSDN : Azure Storage Services REST API Reference
https://msdn.microsoft.com/library/azure/dd179355

For example, if you want to get (download) the blob (which is https://tsmatsuzsttest0001.blob.core.windows.net/container01/tmp.txt), you just send the following HTTP request.

GET https://tsmatsuzsttest0001.blob.core.windows.net/container01/tmp.txt
User-Agent: Test Client
x-ms-version: 2015-07-08
x-ms-date: Tue, 05 Jul 2016 06:48:26 GMT
Authorization: SharedKey tsmatsuzsttest0001:{shared key}
Host: tsmatsuzsttest0001.blob.core.windows.net

It’s very simple !

But one pain point is “How to get the shared key ?”, then I explain this how-to as follows.

How to create shared key (signature) using access key

The shared key is the signature derived (computed) from symmetric key called “Storage Access Key”, and you can get this key from Azure Portal.

When you’re using Microsoft technologies, this kind of computed signature is almost the base64 encoded string of HMAC with SHA256 algorithm. (see Azure AD, Power BI Embedded, Azure DocumentDB, Azure Batch services, etc)
The Azure Storage shared key (signature) is also the same manner !

The following programming example shows how to get this signature (shared key).
We assume that the access key is “93K17Co74T…” as follows.

PHP

<?php
$accesskey = "93K17Co74T2lDHk2rA+wmb/avIAS6u6lPnZrk2hyT+9+aov82qNhrcXSNGZCzm9mjd4d75/oxxOr6r1JVpgTLA==";
$inputvalue = . . . (show you later);

// create base64 encoded signature
$hash = hash_hmac('sha256',
  $inputvalue,
  base64_decode($accesskey),
  true);
$sig = base64_encode($hash);

// show result
echo $sig;
?>

Node.js (JavaScript)

var http = require('http');
var crypto = require("crypto");

http.createServer(function (req, res) {
  var accesskey = "93K17Co74T2lDHk2rA+wmb/avIAS6u6lPnZrk2hyT+9+aov82qNhrcXSNGZCzm9mjd4d75/oxxOr6r1JVpgTLA==";
  var inputvalue = . . . (show you later);

  // create base64 encoded signature
  var key = new Buffer(accesskey, "base64");
  var hmac = crypto.createHmac("sha256", key);
  hmac.update(inputvalue);
  var sig = hmac.digest("base64");

  // show result
  res.writeHead(200,
    { 'Content-Type': 'text/plain; charset=utf-8' });
  res.write(sig);
  res.end();
}).listen(8000);

C# (.NET)

using System;
using System.Text;
using System.Security.Cryptography;

static void Main(string[] args)
{
  var accesskey = "93K17Co74T2lDHk2rA+wmb/avIAS6u6lPnZrk2hyT+9+aov82qNhrcXSNGZCzm9mjd4d75/oxxOr6r1JVpgTLA==";
  var inputvalue = . . . (show you later);

  // create base64 encoded signature
  var hmac = new HMACSHA256();
  hmac.Key = Convert.FromBase64String(accesskey);
  byte[] sigbyte = hmac.ComputeHash(Encoding.UTF8.GetBytes(inputvalue));
  var sig = Convert.ToBase64String(sigbyte);

  // show result
  Console.WriteLine(sig);
  Console.ReadLine(); // Wait
}

As you can see, the input value (signed value) is needed for signing. This input value differs from each services (Azure Storage Services, Azure DocumentDB, Azure Batch Services, etc), and I explain about the case of Azure Storage Services.

How to create input value (challenge value) for Azure Storage

Next I show you how to get the input value for Azure Storage Services.

The input value (challenge value) is the bytes of UTF-8 string constructed from HTTP Request envelope of REST call.
This format is the following. (Please be sure to use “\n” as the line feed character, not “\r\n”.)

Notice : I will explain the details of {Canonicalized Header String} and {Canonicalized Resource String} later.

{HTTP VERB}\n
{Header value of Content-Encoding}\n
{Header value of Content-Language}\n
{Header value of Content-Length}\n
{Header value of Content-MD5}\n
{Header value of Content-Type}\n
{Header value of Date}\n
{Header value of If-Modified-Since}\n
{Header value of If-Match}\n
{Header value of If-None-Match}\n
{Header value of If-Unmodified-Since}\n
{Header value of Range}\n
{Canonicalized Header String (repeated)}\n
{Canonicalized Resource String of URI path}\n
{Canonicalized Resource String of query parameters (repeated)}

The canonicalized header is the HTTP header which starts with “x-ms-” (x-ms-date, x-ms-version, etc).

The canonicalized resource of uri path is like /{storage account name}/{container}/{blob} separated by slash (/), if you access to the uri “https://{storage account}.blob.core.windows.net/{container}/{blob}”.

The canonicalized resource of query parameters is like {parameter name}:{parameter value} separated by colon (:).

For instance, we assume the following HTTP request of REST call.

Notice : The following request will fail, because it’s including both If-Modified-Since and If-Match. (These header is not supported in this REST API request.) Sorry, but this is just the sample for your understanding.

PUT https://test01storage.blob.core.windows.net/container01/tmp.txt?timeout=20&paramtest=value1
User-Agent: Test Client
x-ms-version: 2015-07-08
Content-Type: text/plain; charset=UTF-8
Content-Language: ja
Content-Encoding: gzip
Content-MD5: aQI49bNvDYLLD0DrOMtETw==
x-ms-blob-type: BlockBlob
x-ms-client-request-id: 80f5bd4a-56ed-4ffa-9d04-afd73fda5c9c
x-ms-date: Tue, 05 Jul 2016 01:46:24 GMT
If-Match: etg23vfj
If-Modified-Since: Mon, 27 Jul 2016 01:46:24 GMT
Host: tsmatsuzsttest0001.blob.core.windows.net
Content-Length: 3000

. . . Body (byte) . . .

Then the input value (challenge value) is the following string bytes. (Please be sure to use “n” as a line feed character.)

PUT
gzip
ja
3000
aQI49bNvDYLLD0DrOMtETw==
text/plain; charset=UTF-8

Mon, 27 Jul 2016 01:46:24 GMT
etg23vfj



x-ms-blob-type:BlockBlob
x-ms-client-request-id:80f5bd4a-56ed-4ffa-9d04-afd73fda5c9c
x-ms-date:Tue, 05 Jul 2016 01:46:24 GMT
x-ms-version:2015-07-08
/test01storage/container01/tmp.txt
paramtest:value1
timeout:20

As you know, this mechanism (logic) prevents the malicious users to change the HTTP request without access key. Because the signature should be re-written when the HTTP request envelop (byte) is changed.

Notice : You must also care about “Date” header. The web proxy often changes this “Date” header in HTTP request. As a result, the call for REST API would fail. (Because the signature is invalid.)
It’s better to use “x-ms-date” header instead of “Date” header, when you use REST API.

There are other several notice. (Please refer “MSDN : Authentication for the Azure Storage Services” for details.) :

Each query parameter name and value must be url-decoded.
Sort the canonicalized headers lexicographically by header name
The characters of canonicalized header name (x-ms-…) should all be lowercase. (On the contrary, the canonicalized header value can include the uppercase.)
Replace any breaking white space of the canonicalized header value to a single space
Trim (left and right) the canonicalized header value.
If the api version is prior to 2015-02-21 and Content-Length is blank, the Content-Length shoud be zero (0).
If there are the same (repeated) query parameters, sort all values lexicographically and include them in a comma-separated list.

Programming Examples (PHP, JavaScript, C#)

I show you the programming example to get the shared key.
Assuming that we want to publish the next HTTP request (as REST call).

GET https://tsmatsuzsttest0001.blob.core.windows.net/container01/tmp.txt
User-Agent: Test Client
x-ms-version: 2015-07-08
x-ms-client-request-id: 9251fa41-0ca4-4558-84ac-44ab027b8f1e
x-ms-date: Tue, 05 Jul 2016 06:48:26 GMT
Host: tsmatsuzsttest0001.blob.core.windows.net

In this case, you can get shared key (base64 encoded signature) using the next programming example. (Please change the access key with your own.)

PHP

<?php
$accesskey = "93K17Co74T2lDHk2rA+wmb/avIAS6u6lPnZrk2hyT+9+aov82qNhrcXSNGZCzm9mjd4d75/oxxOr6r1JVpgTLA==";

// construct input value
$inputvalue = "GET\n" . /*VERB*/
  "\n" . /*Content-Encoding*/
  "\n" . /*Content-Language*/
  "\n" . /*Content-Length*/
  "\n" . /*Content-MD5*/
  "\n" . /*Content-Type*/
  "\n" . /*Date*/
  "\n" . /*If-Modified-Since*/
  "\n" . /*If-Match*/
  "\n" . /*If-None-Match*/
  "\n" . /*If-Unmodified-Since*/
  "\n" . /*Range*/
  "x-ms-client-request-id:9251fa41-0ca4-4558-84ac-44ab027b8f1e\n" .
  "x-ms-date:Tue, 05 Jul 2016 06:48:26 GMT\n" .
  "x-ms-version:2015-07-08\n" .
  "/tsmatsuzsttest0001/container01/tmp.txt";

// create base64 encoded signature
$hash = hash_hmac('sha256',
  $inputvalue,
  base64_decode($accesskey),
  true);
$sig = base64_encode($hash);

// show result
echo $sig;
?>

Node.js (JavaScript)

var http = require('http');
var crypto = require("crypto");

http.createServer(function (req, res) {
  var accesskey = "93K17Co74T2lDHk2rA+wmb/avIAS6u6lPnZrk2hyT+9+aov82qNhrcXSNGZCzm9mjd4d75/oxxOr6r1JVpgTLA==";

  // construct input value
  var inputvalue = "GET\n" + /*VERB*/
    "\n" + /*Content-Encoding*/
    "\n" + /*Content-Language*/
    "\n" + /*Content-Length*/
    "\n" + /*Content-MD5*/
    "\n" + /*Content-Type*/
    "\n" + /*Date*/
    "\n" + /*If-Modified-Since*/
    "\n" + /*If-Match*/
    "\n" + /*If-None-Match*/
    "\n" + /*If-Unmodified-Since*/
    "\n" + /*Range*/
    "x-ms-client-request-id:9251fa41-0ca4-4558-84ac-44ab027b8f1e\n" +
    "x-ms-date:Tue, 05 Jul 2016 06:48:26 GMT\n" +
    "x-ms-version:2015-07-08\n" +
    "/tsmatsuzsttest0001/container01/tmp.txt";

  // create base64 encoded signature
  var key = new Buffer(accesskey, "base64");
  var hmac = crypto.createHmac("sha256", key);
  hmac.update(inputvalue);
  var sig = hmac.digest("base64");

  // show result
  res.writeHead(200,
    { 'Content-Type': 'text/plain; charset=utf-8' });
  res.write(sig);
  res.end();
}).listen(8000);

C# (.NET)

using System;
using System.Text;
using System.Security.Cryptography;

static void Main(string[] args)
{
  var accesskey = "93K17Co74T2lDHk2rA+wmb/avIAS6u6lPnZrk2hyT+9+aov82qNhrcXSNGZCzm9mjd4d75/oxxOr6r1JVpgTLA==";

  // construct input value
  var inputvalue = "GET\n" + /*VERB*/
    "\n" + /*Content-Encoding*/
    "\n" + /*Content-Language*/
    "\n" + /*Content-Length*/
    "\n" + /*Content-MD5*/
    "\n" + /*Content-Type*/
    "\n" + /*Date*/
    "\n" + /*If-Modified-Since*/
    "\n" + /*If-Match*/
    "\n" + /*If-None-Match*/
    "\n" + /*If-Unmodified-Since*/
    "\n" + /*Range*/
    "x-ms-client-request-id:9251fa41-0ca4-4558-84ac-44ab027b8f1e\n" +
    "x-ms-date:Tue, 05 Jul 2016 06:48:26 GMT\n" +
    "x-ms-version:2015-07-08\n" +
    "/tsmatsuzsttest0001/container01/tmp.txt";

  // create base64 encoded signature
  var hmac = new HMACSHA256();
  hmac.Key = Convert.FromBase64String(accesskey);
  byte[] sigbyte = hmac.ComputeHash(Encoding.UTF8.GetBytes(inputvalue));
  var sig = Convert.ToBase64String(sigbyte);

  // show result
  Console.WriteLine(sig);
  Console.ReadLine(); // Wait
}

This example returns “sGX7uEBy8i9ldZtx8nLDeD3vX3AI/LB/3msK0oL7oMI=”.
As a result, you must set the authorization header as follows.

GET https://tsmatsuzsttest0001.blob.core.windows.net/container01/tmp.txt
User-Agent: Test Client
x-ms-version: 2015-07-08
x-ms-client-request-id: 9251fa41-0ca4-4558-84ac-44ab027b8f1e
x-ms-date: Tue, 05 Jul 2016 06:48:26 GMT
Authorization: SharedKey tsmatsuzsttest0001:sGX7uEBy8i9ldZtx8nLDeD3vX3AI/LB/3msK0oL7oMI=
Host: tsmatsuzsttest0001.blob.core.windows.net

Using Shared Key Lite

You can also use the light-weight input value, when you use the signature called “Shared Key Lite”. This type of shared key uses the following input value.

Blob, Queue

{HTTP VERB}\n
{Header value of Content-MD5}\n
{Header value of Content-Type}\n
{Header value of Date}\n
{Canonicalized Header String (repeated)}\n
{Canonicalized Resource String of URI path}\n
{Canonicalized Resource String of query parameters}

Table

{HTTP VERB}\n
{Header value of Content-MD5}\n
{Header value of Content-Type}\n
{Header value of Date}\n
{Canonicalized Resource String of URI path}\n
{Canonicalized Resource String of query parameters}

The programming example is the same, and you must set Authorization header as follows. (use “SharedKeyLite” instead of “SharedKey”.)

Authorization: SharedKeyLite {account name}:{signature}

Calling REST API using Shared Access Signature (SAS) URI

You can also use the URI-based signature called Shared Access Signature (SAS) as following.

This only uses uri fragment and doesn’t need the specific HTTP header values (Authorization header, etc) for the secure communication. You can get this uri using Azure Portal (pushing “Generate SAS” button as following picture shows), and just put this uri in your code.
This is very portable !

This uri is used for “sharing” the resource for other users, but it’s having the expiration (not permanent).

For example, we assume that we get the following URI. (The operations using Shared Access Signature URIs should only be performed over an HTTPS connection.)

https://tsmatsuzsttest0001.blob.core.windows.net/?sv=2015-04-05&ss=bfqt&srt=sco&sp=rwdlacup&se=2016-07-08T04:41:20Z&st=2016-06-29T04:41:20Z&spr=https&sig={signature}

Each query string (sv, ss, srt, etc) means as follows. (Please see “MSDN : Constructing an Account SAS” for details.)

sv	(signed) api version
ss	(signed) service (b=blob, f=files, q=queue, t=table)
srt	(signed) resource types (s=service, c=container, o=blob object)
sp	(signed) permissions (r=read, w=write, d=delete, l=list, a=add, etc)
se	(signed) expire time
st	(signed) start time
spr	(signed) protocol
sip	(signed) allowed ip addresses

This signature also expires when the expire time arrives, and you can also get this Shared Access Signature (SAS) with your programming code using the same manner like shared key (previously explained).
The input value (challenge) is the following.

{account name}\n
{signed permissions}\n"
{signed service}\n"
{signed resource type}\n"
{signed start time}\n"
{signed expire time}\n"
{signed allowed ip addresses}\n"
{signed protocol}\n"
{signed version}\n"

Notice : There exist service-level SAS and account-level SAS, and the signed input value differs from each other. In this example, we are talking about the account-level SAS. (For service-level SAS, please see “MSDN : Constructing a Service SAS“.)

For instance, you can get the signature using the following programming example (PHP).

<?php
$accesskey = "93K17Co74T2lDHk2rA+wmb/avIAS6u6lPnZrk2hyT+9+aov82qNhrcXSNGZCzm9mjd4d75/oxxOr6r1JVpgTLA==";

// construct input value
$inputvalue = "tsmatsuzsttest0001\n" . /* account name */
  "rwdlacup\n" . /* signed permissions */
  "bfqt\n" . /* signed service */
  "sco\n" . /* signed resource type */
  "2016-06-29T04:41:20Z\n" . /* signed start time */
  "2016-07-08T04:41:20Z\n" . /* signed expire time */
  "\n" . /* signed ip */
  "https\n" . /* signed protocol */
  "2015-04-05\n"; /* signed version */

// create base64 encoded signature
$hash = hash_hmac('sha256',
  $inputvalue,
  base64_decode($accesskey),
  true);
$sig = base64_encode($hash);

// show result
echo $sig;
?>

It returns “+XuDjuLE1Sv/FrJTLz8YjsaDukWNTKX7e8G8Ew+5aps=”. As a result, the complete SAS URI would be the following. (The signature must be url-encoded.)

https://tsmatsuzsttest0001.blob.core.windows.net/?sv=2015-04-05&ss=bfqt&srt=sco&sp=rwdlacup&se=2016-07-08T04:41:20Z&st=2016-06-29T04:41:20Z&spr=https&sig=%2BXuDjuLE1Sv%2FFrJTLz8YjsaDukWNTKX7e8G8Ew%2B5aps%3D

As you can see, this type of signature (account-level SAS) is not including resource data. Then you can also get the blob using the same signature as follows. (This example is “tmp.txt” in the container named “container01”.)

https://tsmatsuzsttest0001.blob.core.windows.net/container01/tmp.txt?sv=2015-04-05&ss=bfqt&srt=sco&sp=rwdlacup&se=2016-07-08T04:41:20Z&st=2016-06-29T04:41:20Z&spr=https&sig=%2BXuDjuLE1Sv%2FFrJTLz8YjsaDukWNTKX7e8G8Ew%2B5aps%3D

Please refer the following document for details.

MSDN : Authentication for the Azure Storage Services
https://msdn.microsoft.com/en-us/library/azure/dd179428.aspx

MSDN : Constructing a Service SAS (service-level)
https://msdn.microsoft.com/en-us/library/azure/dn140255.aspx

MSDN : Constructing a Account SAS (account-level)
https://msdn.microsoft.com/en-us/library/azure/mt584140.aspx

Categories: Uncategorized

Tagged as: Azure

4 replies »

John says:

2017-05-13 at 00:36

Ridiculous … this is too complex to use. Big fail Microsoft.

LikeLike

Daniel Walker says:

2017-12-12 at 02:11

It would be nice if there was an example for Azure Batch signature because I can’t find any examples of that!!

LikeLike

rishiuttam says:

2018-03-19 at 20:25

needs to be simplified… both the sdk and api for node is overly complex.

LikeLike

Alexis says:

2018-06-21 at 23:28

does not work anymore with last version 2017-11-09

LikeLike

tsmatz

Professional Development, Data Science

How to use Azure Storage without SDK

Calling REST APIs

How to create shared key (signature) using access key

How to create input value (challenge value) for Azure Storage

Programming Examples (PHP, JavaScript, C#)

Using Shared Key Lite

Calling REST API using Shared Access Signature (SAS) URI

4 replies »

Leave a Reply Cancel reply

Recent Posts

Reinforcement Learning

Language (NLP)

Databricks Hands-On

Azure ML Hands-On

Others (GitHub)

Tags

Follow

How to use Azure Storage without SDK

Calling REST APIs

How to create shared key (signature) using access key

How to create input value (challenge value) for Azure Storage

Programming Examples (PHP, JavaScript, C#)

Using Shared Key Lite

Calling REST API using Shared Access Signature (SAS) URI

Share this:

Related

4 replies »

Leave a Reply Cancel reply

Recent Posts

Reinforcement Learning

Language (NLP)

Databricks Hands-On

Azure ML Hands-On

Others (GitHub)

Tags

Follow