Despite the hype, especially in mainstream media, let’s be honest Machine Learning technologies like speech recognition are still in their infancy, but it is an exciting time in AI/ML and the pace of improvement is beyond rapid. This article is about my own experiences with Google’s Speech-to-Text AI and provide insights into how to improve results predominately from a C# programmer’s standpoint. I’ll revisit this post as and when I find more ways to enhance the process.
I have recently been involved in a lot of software engineering and C# coding for a project requiring integration of various AI tech but especially Speech-To-Text. As part of the project we carried out exhaustive trials of various Speech-To-Text services including IBM Watson (Our second place candidate) and finally settled on Google. Other services we tested delivered poor recognition results.
In our application we are storing our audio recordings in WAV format in Azure. Like most Speech-To-Text services, Google recommends the use of Lossless encoding formats for best results. For speech recognition greater than a minute in duration you will need to copy your WAV audio file to Google Cloud before invoking the Speech-To-Text API. Shorter WAV files can be uploaded directly to the API but we will not be covering that feature here today.
The following code shows a simple C# console app to take a WAV file from an Azure storage container, place it in a Google storage bucket and then call the Speech-To-Text API to get a Transcription.
- Create a new C# .Net Core console app and call it GoogleS2TTool.
- Add the following NuGet packages:
- Google.Cloud.Language.V1
- Google.Cloud.Speech.V1
- Google.Cloud.Storage.V1
- WindowsAzure.Storage
You’ll need to set the using statements in program.cs to:
using Google.Cloud.Language.V1;
using Google.Cloud.Speech.V1;
using Google.Cloud.Storage.V1;
using Google.LongRunning;
using Microsoft.WindowsAzure.Storage;
using System;
using System.Collections.Generic;
using System.Data;
using System.Data.SqlClient;
using System.IO;
using System.Linq;
using System.Text;Then replace the Main in your program.cs with the following code:
static void Main(string[] args)
{
Console.WriteLine("Google Speech to Text Tool");
if (args.Length == 0 || args.Length > 1)
{
Console.WriteLine("Google Speech to Text Tool");
Console.WriteLine("");
Console.WriteLine("Usage:");
Console.WriteLine(" GoogleS2TTool fileID");
Console.WriteLine("");
Console.WriteLine(" fileid = Filename in Azure Storage");
Console.WriteLine("");
return;
}
var fileID = args[0];
// Set some Boost phrases to improve speech recognition. You can have up to 5000 at time of writing
var boostPhrases = new string[] {"BMW", "Audi", "Mercedes", "horsepower" };
var result = Transcribe(fileID, boostPhrases);
if (result == null)
{
// Error returned from Transcribe
return;
}
Console.WriteLine($"Speech results for {fileID}:");
foreach (TranscriptionResult speechLine in result)
{
// If the Speaker value of the TranscriptionResult object is = 1 then it is the caller speaking, else it's the callee
Console.WriteLine($"{(speechLine.Speaker == 2 ? "Callee" : "Caller")}: {speechLine.SpeechText}");
}
Console.WriteLine("Done!");
}As you can see, the Main sets up a string array of what I have termed Boost Phrases, then calls a procedure to do the main body of the operation, copy the WAV and execute the calls to get a transcription.
var boostPhrases = new string[] {"BMW", "Audi", "Mercedes", "horsepower" };This line is the first hint on improving transcription quality. As you will see later, we pass this array to the Speech-To-Text API as part of our request. At the time of writing you can have up to 5000 boost phrases. More than enough for anybody. Use the array to load up on product names, brands and other terminology that is unique to the WAV telephone call sample you are going to get a Transcription for.
Before we take a look at the Transcribe method code, a quick word about Google Credentials. I am assuming you are familiar with setting up a Speech-To-Text app in Google Cloud. Don’t forget to download your json file for credentials and save it in the root of your Solution. go to your project properties in Visual Studio and set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the name of the json file you placed in the root of your application.
Now lets take a look at our Transcribe code that the Main calls:
static List<TranscriptionResult> Transcribe(string fileID, string[] boostPhrases)
{
// First transfer our stereo (2 channel) Wav file from Azure to Google Cloud Bucket
Console.WriteLine($"Uploading Azure Wav file to Google Storage bucket {fileID}");
var googleSClient = StorageClient.Create();
var googleBucket = "<Your Google Bucket name>";
try
{
// Get some Azure storage values from the app configuration
var azureStorageAccount = "<your Azure Storage Account connection string>";
var azureCallContainer = "<container name case sensitive!!!>";
var storageAccount = CloudStorageAccount.Parse(azureStorageAccount);
var blobClient = storageAccount.CreateCloudBlobClient();
var blobContainer = blobClient.GetContainerReference(azureCallContainer);
var blobFile = blobContainer.GetBlockBlobReference(fileID);
using (var streamAzure = new MemoryStream())
{
blobFile.DownloadToStreamAsync(streamAzure).Wait();
streamAzure.Position = 0;
// Got stream of Azure Source, Now pass this to Google Bucket
googleSClient.UploadObject(googleBucket, fileID, null, streamAzure);
}
}
catch (Exception err)
{
Console.WriteLine($"Error uploading Azure Wav file to Google Storage bucket {fileID} {err.Message}");
return null;
}
Console.WriteLine($"Azure Wav file uploaded to Google Storage bucket {fileID}");
// Wav now in Google Storage bucket. Lets get a Speech to Text Transcription
// Create a Google client. Note invoking causes Google Cloud SDK to locate credentials in the
// GoogleCreds.json file in the root of the project
var gsURI = $"gs://{googleBucket}/{fileID}";
var speech = SpeechClient.Create();
// Invoke Google LongOperation to get a transcription
// While we are going to wait for the LongOperation in this code, we could issue the instruction
// and return later to poll for completion
Operation<LongRunningRecognizeResponse, LongRunningRecognizeMetadata> longOperation = null;
try
{
RecognitionConfig recConfig = new RecognitionConfig
{
Encoding = RecognitionConfig.Types.AudioEncoding.Linear16,
// UseEnhanced = true,
// Model = "phone_call",
LanguageCode = "en-GB",
AudioChannelCount = 2, // Important!! Our Wav is Stereo
EnableSeparateRecognitionPerChannel = true, // This will ensure we get our transcription in separate channels
EnableAutomaticPunctuation = true // Still in beta
};
// See if we need to add some boost phrases?
// Boost phrases improve the quality of transcription for Proper Nouns and Technical phraseology
if (boostPhrases.Length != 0)
{
var tmpWatchPhrase = new SpeechContext { Phrases = { boostPhrases } };
recConfig.SpeechContexts.Add(tmpWatchPhrase);
}
Console.WriteLine($"About to commence Google LongOperation to transcribe WAV in storage bucket {fileID}");
longOperation = speech.LongRunningRecognize(recConfig, RecognitionAudio.FromStorageUri(gsURI));
longOperation = longOperation.PollUntilCompleted();
Console.WriteLine($"WAV Transcription completed {fileID}");
}
catch (Exception err)
{
Console.WriteLine($"Error calling Google LongOperation to transcribe WAV {fileID} {err.Message} {err.InnerException.Message}");
return null;
}
// We now have our result of transcription but first delete the WAV from Google Bucket to save money
googleSClient.DeleteObject(googleBucket, fileID);
Console.WriteLine($"WAV deleted from Google Storage Bucket {fileID}");
var response = longOperation.Result;
// Build an array of TransciptionResult for each line of results
// Speaker is set by looking at channelTag in the result
///** 1=Caller and 2=Callee */
var listResults = new List<TranscriptionResult>() { };
foreach (var result in longOperation.Result.Results)
{
listResults.Add(new TranscriptionResult { Speaker = result.ChannelTag, SpeechText = result.Alternatives[0].Transcript });
}
return listResults;
}Lets break that code down and go through it:
var googleSClient = StorageClient.Create();
var googleBucket = "<BucketName>";The first line creates a Google Storage client for us to use later to upload the WAV file and will invoke the Authentication of your Google Credentials. If you get authentication errors here then you have most likely not setup your Google Credentials correctly or not set the environment variable in the project settings for GOOGLE_APPLICATION_CREDENTIALS.
Replace the <BucketName> with the name of a Bucket in Google Storage to use.
var azureStorageAccount = "<your Azure Storage Account connection string>";
var azureContainer = "<container name case sensitive!!!>";
var storageAccount = CloudStorageAccount.Parse(azureStorageAccount);
var blobClient = storageAccount.CreateCloudBlobClient();
var blobContainer = blobClient.GetContainerReference(azureContainer);
var blobFile = blobContainer.GetBlockBlobReference(fileID);
using (var streamAzure = new MemoryStream())
{
blobFile.DownloadToStreamAsync(streamAzure).Wait();
streamAzure.Position = 0;
// Got stream of Azure Source, Now pass this to Google Bucket
googleSClient.UploadObject(googleBucket, fileID, null, streamAzure);
}- Set azureStorageAccount to the connection string for your Azure Storage. See here for help on how to get the string value.
- Set azureContainer to the name of the container where your WAV file of your Telephone Call recording is placed
At this point then we have copied the WAV file to a Google Storage bucket.
var gsURI = $"gs://{googleBucket}/{fileID}";
var speech = SpeechClient.Create();Now we initialise the Speech Client in preparation for calling Google Speech-To-Text and we setup a URI for our WAV file we just transferred into Google Storage.
In our Try..Catch we now have the following:
RecognitionConfig recConfig = new RecognitionConfig
{
Encoding = RecognitionConfig.Types.AudioEncoding.Linear16,
// UseEnhanced = true,
// Model = "phone_call",
LanguageCode = "en-GB",
AudioChannelCount = 2, // Important!! Our Wav is Stereo
EnableSeparateRecognitionPerChannel = true, // This will ensure we get our transcription in separate channels
EnableAutomaticPunctuation = true // Still in beta
};
// See if we need to add some boost phrases?
// Boost phrases improve the quality of transcription for Proper Nouns and Technical phraseology
if (boostPhrases.Length != 0)
{
var tmpWatchPhrase = new SpeechContext { Phrases = { boostPhrases } };
recConfig.SpeechContexts.Add(tmpWatchPhrase);
}
Console.WriteLine($"About to commence Google LongOperation to transcribe WAV in storage bucket {fileID}");
longOperation = speech.LongRunningRecognize(recConfig, RecognitionAudio.FromStorageUri(gsURI));
longOperation = longOperation.PollUntilCompleted();
Console.WriteLine($"WAV Transcription completed {fileID}");
}The recognition configuration object recConfig has a few mandatory and optional parameters and they are as follows:
- Encoding – Linear16 states that the Speech-To-Text API should expect a WAV encoding type for the file.
- AudioChannelCount – Because our WAV telephone recording is two channel (stereo) we state the number of channels here. It is possible to use a Mono recording (And leave this option out) but getting identification of “who said what” would then require we use a feature called Speaker Diarisation. The problem is that at the time of writing this, Speaker Diarisation is only available presently for ‘en-US’ language which wasn’t much use to us. So we elected to use separate channels instead to try to get “who said what”.
- EnableSeparateRecognitionPerChannel – Again, as with AudioChannelCount, because we are using a dual channel WAV and we want the results to reflect who was speaking the text we set this option to true. If your file is not Dual channel then you can leave this option out.
- EnableAutomaticPunctuation – This option is only available in Beta at present. It will add punctuation like comma, question marks and full stops to your transcription results.
A few options in the code above are commented out but I wanted to cover them here and what their impact is. Model and UseEnhanced need to be used in conjunction with each other if you are going to make use of this. Google has a special Model called ‘phone_call’ to enhance the results of a Phone Recording transcription. Again, at time of writing, this was only available for ‘en-US’ so we couldn’t use it. If you set the Model, you will also have to set the UseEnhanced as well and BEWARE, this incurs higher transcription costs unless you offset it by opting in to logging.
if (boostPhrases.Length != 0)
{
var tmpWatchPhrase = new SpeechContext { Phrases = { boostPhrases } };
recConfig.SpeechContexts.Add(tmpWatchPhrase);
}If we have passed in any Boost Phrases to the Transcribe method (See above for more information) then they are added as an array content value to a SpeechContext called Phrases. These phrases dramatically improve Transcription Quality. Even using complicated Medical Terms. In our tests we noticed 90% of the phrases were picked up in transcription.
longOperation = speech.LongRunningRecognize(recConfig, RecognitionAudio.FromStorageUri(gsURI));
longOperation = longOperation.PollUntilCompleted();And finally we get to call Google’s Speech to text API with a request to analyse a large sample. This is the best way to pass a sample larger than a minute in length. Obviously, you wouldn’t sit with the thread normally frozen waiting for the results. You could get your code to go and do other things and return to poll the longOperation object for completion. Our simple Console implementation however will sit and wait. In our tests, Google was significantly faster than others in providing results. IBM Watson for instance would take 20 minutes to return a result for a 20 minute recording. I have no definite stats handy but a sample I ran this morning for a 25 minute call was done in under 3 minutes.
var response = longOperation.Result;
// Build an array of TransciptionResult for each line of results
// Speaker is set by looking at channelTag in the result
///** 1=Caller and 2=Callee */
var listResults = new List<TranscriptionResult>() { };
foreach (var result in longOperation.Result.Results)
{
listResults.Add(new TranscriptionResult { Speaker = result.ChannelTag, SpeechText = result.Alternatives[0].Transcript });
}
return listResults;And finally we take our response from Speech-To-Text and add it to a List<TranscriptionResult>. Note how the identification of the speaker of the text is caught in the ChannelTag property. 1 is the Caller and 2 is the recipient of the call (Callee). By the way, TranscriptionResult looks like this:
public class TranscriptionResult
{
public int Speaker { get; set; }
public string SpeechText { get; set; }
}When you run the code from Visual Studio, don’t forget to setup a project command line argument in Project properties to the name of the WAV file you are going to send from Azure to Google for Transcription.
As I said at the start of this article. I’ll be revisiting this and updating it as Google brings out new options and enhancements to the Speech-To-Text service.