Build AI-Enabled Apps with .NET MAUI and Azure Cognitive Services by Aditya Oberai

I am Aditya, I am currently a developer advocate at AppRite, I also work as a hackathon coach

with major league hacking and I have been a part of Twilio's Champions program for a

while now. And you know, just aside from that my free time I love working with .NET, I love learning

more, exploring, working with APIs, working with cross-platform apps. And yeah, that's a little bit about me

Now, before I jump in, I do want to mention, if anyone wants to click pictures and post

about them, clicking and sharing is highly encouraged, so if anyone wants to do that

please feel free to do so. That's going to be totally all right. Now, if I talk about the session that we have here and the content we're going to discuss

I think the session title was rather self-explanatory, but just to cover the agenda once

we're going to be talking a little bit about cognitive services, the suite of services that this offers

the sort of APIs it offers. Aside from that, we will bring a special focus

towards the computer vision API, and that is primarily because the demo we have

will comprise the computer vision API. So I'll give a little brief introduction to that as well

Aside from that, we will discuss what .NET MAUI is, what it enables you to do

I will cover that in brief rather than in detail because I know there is a dedicated session on .NET MAUI in the C-sharp and .NET track as well tomorrow if I'm not wrong

So I will give a brief introduction to that. And after that, we will go ahead and explore the example app that I have that you can look at as well

So, yeah. First up, let's talk about Azure Cognitive Services. Just before I jump into this, is there anyone here who is aware of what Cognitive Services is

We have a few folks here. Have you tried it out in the past as well

Have you tried it in the past as well? Okay, nice. What about you, sir

Have you had a chance to work with Cognitive Services in the past as well? Yeah, I've been able to talk with Cognitive Services

Oh, wow. That's amazing. That's actually wonderful

Thank you so much for sharing that. Now, it's great to hear about your experiences

but I do believe for a lot of folks, this might be their first introduction to what Cognitive Services is

Let me hop ahead and tell you a little bit about that. So Azure Cognitive Services is, well

a cloud-based set of services with REST APIs and SDKs that lets you build in cognitive intelligence

into your applications. In simple words, what that means is you can add cognitive features

cognitive features meaning abilities around speech, vision, and so on. And you can do this and add these features

into your applications without actually having any AI or data science skills

which is actually really interesting, right? Because for me, I like working with APIs

but I can't figure out the statistics behind a lot of this work

That's beyond me. But, you know, that's exactly what Cognitive Services enables

As I've already talked about one of the benefits, let me jump into a few more that Cognitive Services has

First thing, it is customizable. You can create customizable pre-trained models with the AI research

You know, it's built on the foundations of the AI research that Microsoft has been doing for a while

It's really, really convenient because you can deploy cognitive services anywhere across the world

whether it's, you know, from the cloud, whether it's on the edge. The good thing is, like, for example

if you ever look at cognitive services or with other Azure services, you would notice they've got

different regions available, right? Yeah, you can essentially deploy cognitive services resources

across any of those. Next up, as I've been touting from the beginning

you need absolutely no ML expertise. I, for example, have absolutely none

and yet I have been able to build ML-enabled applications, AI-enabled applications using cognitive services

because that's how easy it makes life for you if you ever want to do that. And last but by no means the least

cognitive services is developed with the strict ethical standards and it does prioritize on empowering

and enabling responsible usage of AI across the industry. And that has been a priority that Microsoft has had for a long while

So it does follow very, very strict ethical standards that way. Now, I did mention that, you know

cognitive services is a suite of cognitive features, right? Let's learn a little more about what these categories

what these types of services are. Like, for example, first up, we've got vision

And, you know, you can imagine that we'll be talking about vision because I already mentioned we'll be touching the Computer Vision API, right

But, you know, even aside from the Computer Vision API, which they use to yze content in images and video files

you've also got Custom Vision, which lets you actually create customizable image recognition models

You've got the Face API, which is used to detect emotions in faces of people

And, you know, aside from that, it can be used to check whether, you know, you've got the same face across

so fun fact, Uber actually used the Face API for onboarding of their drivers

for their authentication and recognition and that's a case study you can actually check out

on the Microsoft website right now as well then of course we've got speech

which includes speech to text transcribing audible speech you've got text to speech similarly

speech translation is available speaker recognition is also now available where you can actually detect people based on voice

that's a service that has been made available now as well. They've got the language APIs

which features language understanding, or LUIS, which lets you build in natural language

into your applications. What that means is you could essentially build specific pieces of data

There is a content moderator, which detects offensive or unwanted content. There is a personalizer

which is used for creating personalized experiences across user bases. If there's anyone working in the customer data space as well

that might be something that you've heard of. The search APIs are there, which actually feature

a variety of Bing-based APIs, ranging from News Search to Image Search

Web, AutoSuggest, and so on. So that with cognitive services is actually a really, really vast suite

and it's very, very handy to use. Coming to the API that I mentioned, the API in focus

that is, the computer vision API, This is one that we're going to be looking at later on as well

And Azure's computer vision service here, it does give you access to algorithms that process images

and return information based on them. One of the biggest features of, you know

like, of the computer vision API is their optical character recognition API or OCR which lets you extract text from images And you know with the new Read API they have you know within their OCR offering

you can not just detect printed text, but also handwritten text, which is incredibly difficult

because, just for context, is there anyone who knows why you can never get a perfect

handwritten text match anywhere? If you ever try OCR on handwritten text, it's impossible to get

a perfect match 100% of the times. Exactly. You can't have all of this text

available across the internet, right? It's going to be different. So, you know

the read API is surprisingly accurate with the handwritten text as well. And it doesn't just support English

but multiple languages beyond as well. Then, you know, after that, we've got image ysis

which lets you extract features from images such as objects, faces, text descriptions and so on

which is, again, very, very convenient if, you know, you're working with an application

Let's say, for example, for folks who might be differently enabled, right? Something like image ysis

could really, really help you build tools to work with data in real time

and extract and share that information. The Face API does technically fall under

their computer vision offering as well, and it allows you to detect, recognize

and yze faces in images. And, you know, as we mentioned, that Uber has used this for the drivers as well

And, you know, even aside from that, facial recognition software is pretty important across scenarios

such as identity verification, touchless access, face blurring for privacy, right? That's something that you could enable with the face API as well

if you're working on any privacy-centric applications. And then last, but not the least

We've got spatial ysis, which lets you yze and presence movement of people

distancing and so on across video feeds. And, you know, based on ysis of space, essentially

it will give you responses and produce events that you can have other systems respond to

A few things about the computer vision API are that if you are working with images

the image does need to be one of the conventional formats like JPEG, PNGs, GIFs

or bitmap BMP files. At this moment, you do need the file to be less than 4 MB

And the minimum size of the image should be 50 by 50 pixels

For the read API that is capped at 10,000 by 10,000 as well

which probably most of us aren't actually hitting in most cases, if I'm honest

So I think it's a fairly good range for you to work with if you have a plan to

Having talked about cognitive services, Let's talk about MAUI. Quick question. Is there anyone who's aware of what .NET MAUI is

Is there anyone aware of what Xamarin was, by any chance? Yeah, yeah

So Xamarin basically was a cross-platform framework that let you work and build applications across

And the thing with Xamarin was, it was initially its own organization, its own entity

and it used mono, but Xamarin's internal runtime and solution was basically built on .NET Framework 4

And, till last year, that was what Xamarin used. This year though, Xamarin has entirely been merged into the .NET SDK

and is now called MAUI, or, you know, Multi-Application UI. So, and then you know with multi-platform application UI, that's the full form

And in its essence, it is a cross-platform framework that lets you build mobile and desktop applications with C Sharp and XAML

So, if anyone's aware of Flutter or React Native here, chances are that there might be a few folks aware of that

MAUI is another solution you could work with, which lets you work with a C Sharp code base

which of course has a lot of its benefits, primarily being that it has a C Sharp

and a .NET-based code base, right? Because the C Sharp community, the C Sharp ecosystem is so extensive

it lets you work across systems and solutions, and it will definitely enable your team a lot more

if you're working with .NET already. And with MAUI right now, you do have support for Android, iOS

Tizen, Samsung Tizen, you've got Mac OS, you've got Windows, so you could essentially build applications across all of these platforms

and essentially allow right ones run anywhere experience, right? Currently, MAUI does support .NET 6, and I know .NET 7 is in preview

so by the time .NET Conf happens later this year, .NET 7 support should also be rolled out

and, you know, they've got individual frameworks across, you know, the MAUI offering

So just for example, we had Xamarin.iOS to build iOS apps. You've got .NET for iOS now

Xamarin.Android is now .NET for Android. You've got .NET for Mac OS as well, which uses Mac Catalyst

And for Windows applications, it is using WinUI 3. So all of these frameworks all use the .NET 6 base class library

And this will allow you to abstract away the underlying platform entirely from your code

And it's going to use the .NET runtime to provide an execution environment. I'm not going to go into too much more detail

because I know there's going to be a dedicated session on this as well. But that being said, .NET MAUI is definitely very handy to work with

If you've worked with Xamarin, you will like this a lot more. There has been a massive focus on developer experience with MAUI's upgrades

Resource management has become far, far easier as well. You can now use declarative UIs as well

That's entirely an option. So MAUI is definitely a really good solution for you to work with

if you want to work on cross-platform applications. Now, I did mention that, you know

after covering all of these topics, we're going to have a little example app

that we'll work with as well, right? So for this application, we're going to basically be building an application

that clicks a picture of any piece of text and extracts and shows it in the app

Now, for this, like, I do have a GitHub link that I will be sharing with you just after this

so if you want to follow along, you can do that, and you can work on it later on in your own free time as well

In the meantime, though, in order to build this sort of an app, the prerequisites that you would need would be an Azure account, of course

and you would have to create a computer vision resource on your Azure portal

It could be a free resource. You could go with paid. Either way is fine. Whatever works for you

We, at the end of the day, essentially just need the API key and the endpoint that it offers

Next up, the MAUI application that we have is .NET 6 based, so it would be good to have

.NET 6 installed on your systems as well. And we would need the latest versions of Visual Studio IDE for that

Just to clarify once again, it is the Visual Studio IDE and not VS Code

I know that is a question that comes across a lot but at the moment in order to build in MAUI applications you will need the IDE VS Code will not do So yeah the IDE is going to be necessary

And within that, so if anyone's worked with Visual Studio IDE, you would probably know that it offers quite a variety of workloads

whether to build internet-based applications, whether to work with Azure Cloud Services and build those SDKs

whether to build desktop applications. There is similarly a workload for MAUI applications as well

And that's another thing you will need because it's going to not just add all the necessary dependencies for you to build a MAUI application

It will also give you a vast variety of templates that you can use across all of these platforms

So those are the prerequisites if you want to work with a MAUI application and, you know, at least build this particular one

So next up, firstly, of course, we need to create a computer vision resource, right

you can do that by just going ahead to your Azure portal and searching and

Once you do that you probably come across one of these pages where you've got a mention a resource name

It'll ask you for your resource group your region and so on. It's a very simple setup in general

pretty straightforward The only thing I will you know Recommend you all to keep an eye out on is is that whenever you're creating a computer vision resource

You could just directly go ahead and review and create it after this first page

after you've checked the responsible AI notice. But make sure to give a look at the networking settings once

because some of your applications you will want to expose across public networks

Some cases you might not, right? And it's good to be wary about that. If you're working on a private network

and you want to make sure that your computer vision resource is not exposed publicly, just be mindful about that

That being said, I've got the GitHub link here before we talk about the applications

So if anyone wants to get that open, I've got the QR as well and the GitHub link

So whatever works for you should be good. I'm going to take 15 seconds or so

just to make sure everyone's got this captured and ready. Yeah, so once that is done

once you've got this repository open, you should see an application there

which is built and ready. Just an easy way to understand the application

is if you enter the C Sharp Con 2022 demo directory within that repository

you will come across quite a few different files, one of them being mainpage.xaml

and mainpage.xaml.cs. That is where we've written the code for running these functionalities

That is the landing page, the first page that we come across

So at the moment, I would recommend you folks to take a look at that first

before anything else, because that's going to help you follow along with the examples and the code that I show you after this

So the XAML essentially is called Extensible Application Markup Language. It is an extension of the XML format you might have come across

and it is a markup language that lets you design UIs for applications using MAUI or WinUI or in the past UWP as well

You also used XAML across WPF and its later editions and so on

So XAML is a markup language that lets you design a UI. For this application, the UI I designed was pretty simple

It's just a picture along with a capture image and an extract text button

The picture is the picture you will be clicking through the application, and the extract text

button will extract the text and give you the response. Right. Along with that, we also have a helper file in this application under the helpers folder

which is basically where you would add your API key and endpoint

And this is just for reference because we're going to be looking at code later on and referencing to this

but I wanted to not hard-code my API key into the functions directly

In the future, what I do recommend, though, is in any case, in anywhere, wherever you have API keys or private information

at least with .NET-based applications or any mobile applications or so on

please do use Web API backends. That is something that I highly recommend

because you can keep this information a lot more secure and environment variables and so on

That's not necessarily as easy to do when you're on a mobile application on a local device

where you've got all the information stored, right? So, yeah, that's a recommendation from a security perspective

Coming to the main page.xaml.cs, the first major function you will see

is the capture image function, which uses the media picker in MAUI to capture a file

If anyone was a Xamarin developer, you may have been aware of the media plugin by James Montemagno

which basically let you capture pictures or pick out files. Now, with Maui, they actually merged that functionality

into Maui's Essentials. So you can actually do that directly with Maui's inbuilt libraries

So it is pretty simple that way. The media picker lets you capture a file

and after that, what we have just done is we have created a local path

and saved the bitstream of this image locally to ensure that this image can be accessed later on as well

The one thing I will note is the file path variable at the bottom, that is a global variable across this class

and that is just there so that we can store this file path

and use it later on across another function because we don't have a direct call from this function

into this extract text function. I kept a dedicated button and an event for that

So we are keeping the local file path stored separately in another variable in the meantime

But otherwise, this code here is majorly boilerplate. You can actually go to the .NET MAUI documentation

and find this as well. Next up, we've got the extract text functionality

which is using the computer vision SDK here, which is available for .NET

So what here we're going to do is we're going to use the credentials of our cognitive service resource

and we will authenticate our client first with that. The SDK has a provision for that, as you can see at the top

We've got the Computer Vision client, which is being authenticated with these credentials

Now, after that, the Computer Vision API has a two-step flow. The first step of this flow is to send the image data over to the API

it is then processed and sends back a header which contains the location detail

which you can then again hit later on to get back your response

So there is a two-step approach here with the computer and vision API and that is basically because it needs more processing time

So it chooses to keep this function a little more async and they've stored the response at a separate location

We got our thread going off to sleep for a small little period of time and then we send a call back again So that is something to keep in mind whenever you use the computer vision API It is going to be a two approach

It's not just going to be you hit the API and you get back the response. So in the first step, we essentially send the

file, the byte stream, you know, the byte added the stream of this

of this image that we've clicked and we send that over to the computer vision API

We get back a location which is stored in the operation location string there

We then put the thread to sleep for just a couple of seconds

while we wait for this to end. And after that, we again hit the API with this location in mind

to get back our response. After that, we've just gone through this entire response

that we've received in the text URL file results. And then we just iterate through this

to get back all the different lines of, you know, extracted from this text

So if we look at this application, the way it's going to run is you capture an image

We've got the text there. It's going to click a picture. You know, you can click a picture, and the moment you extract text

just wait for a couple of seconds, and you will see a display alert pop up

which has our output there. So yeah, it's pretty simple that way, you know

It's a very comfortable experience, and just with a few lines of code, we've basically built an application

that can extract text from handwritten stuff. Another fun solution that I just wanted to showcase

aside from the Computer Vision API, was one that I built with the Face API a little while back with Xamarin and Face API

and what this did is it clicked a picture of a person, you know, took their facial information

extracted the predominant emotion, and then placed an emoji on top. So that way you can actually build really, really fun and handy solutions with the face

API and with cognitive services in general. And by the way, the GitHub link for that is available there as well

So if you want to look at that, you can do so later on

So yeah, those were a few fun use cases of cognitive services and how you could use those

with MAUI and build AI-enabled applications. And that's about it for me

Thank you so much. By the way, if anyone wants to connect with me, I've got my Twitter handle there as well

You can look at that if you want. I've also got my website which has all my other socials

So feel free to connect with me over there. Does anyone have any questions in the meantime

Yeah, I'll just take this and then you. I am Arun from Bhartavidyapeet

So basically I have a question like how is the cognitive services is different from OpenCV and MediaVibes

Like we have few inbuilt libraries also in Python or MediaVibes. So like for face recognition, it's a Cascade classifier

So like we can easily detect the face using Cascade. So how is the cognitive services is more easy and beneficial for us

So just a question because I'm not a Python developer myself. if you're working with OpenCV, would you have to define the logic for extraction of this information

detection and so on on your own? Manvili, just a question there

No, no. Like, we can simply call out the function from the library

Inbuilt libraries we are having. So, as I told, like, for detecting out the face, we have cascade

So, we just have to call the library and we have to put the image path

and easily we can detect out the face. Okay. Are these libraries available across languages other than Python

Yes. It's available in C sharp, Python. And I might guess it's also available in C

And that's particularly for, and this is dedicated to computer vision based, right

Yeah, yeah, absolutely. So with OpenCV, I personally haven't done a benchmark across, and you know, comparison across handwritten text, I think that would be something to compare with

Beyond that, because Cognitive Services is a vast suite that goes beyond vision

and I think a lot of the services involved here, whether it's LUIS

which lets you enable and implement natural language processing, or if we're talking about the decision APIs

you probably wouldn't get as simple an experience as consuming REST APIs across all of these

OpenCV is one example, which is pretty cool, but you may not get that same experience across this entire suite

so that way I think with cognitive services it will make the friction point

between implementing AI within your applications a lot simpler across a variety of use cases for a lot of people that way

Answer the last question like could we change out the algorithm working

behind the cognitive services? So I do believe cognitive services offers you

customizability to an extent yes there are a few different models that they have

worked on in the past I would say that you know that is something that we can

explore once it should be possible and there are some services for sure that

let you are trained a lot more manually right like when when I talk about Louise

and with the pot maker I know that lets you take a lot more custom input custom

vision purely takes your own information for you to build a custom model there so

in quite a few services yes there is more customizability in some relatively

a little lesser. Okay, sir. Thank you so much. I think I

saw a question from you as well. Hello, I'm Akshay. So, my

question is, how are MAUI apps better than progressive web apps? Or

is this like a C++ versus Python or Java or something sort of question? So, with

a progressive web app, you're essentially taking the web experience and allowing that on your local systems

right? With MAUI, you're building fully native applications. So, it's a little more closer to

building something like a dedicated native Android app and so on. So these aren't hybrid apps

Although you can do that with MAUI now. With laser support, you can actually build hybrid apps as well

similar to what you would be doing with Electron. But using MAUI this way will let you

and provide you a dedicated native experience. So if you want to work with native APIs on your system and so on

MAUI will enable you to do that. Yeah. Thank you. We had a question there as well

I think we can take this one and then probably move forward. So if anyone asks further questions, we can take those afterwards as well

But I'll take this one last question now. Thank you. So how is the support for this OCR cognitive service in Indian languages

So does this work on Indian languages? I do believe that they have support beyond English as well

So there should be support across quite a few vernaculars. But I will have to check which ones

I'm not 100% sure on that. But yes, I do know that they do claim

that the Read API works beyond English as well. So we can definitely explore that

I'll actually look into that and get back to you once a little later today as well. If that's all right

Thanks, that works. Yeah. Awesome. Well, thank you so much. And hope you all have a fun conference