Run the world from the palm of your hand by Jared Rhodes || Lightup Conference

OK, so this is the run the world from the palm of your hand talk. If you were looking for one of the other talks, just stay here

Alright, so if you want to go ahead, you can scan the QR code and donate in support of UNICEF

I'll give you just a second if you didn't have your mobile phones ready. Normally with an audience, I can actually look out into the crowd and see how many phones are pointed at the screen

So I'll just talk right now to kill time in case you didn't get it. All right, and then we'd also like to thank our sponsors

They're helping make this possible. Okay, so about me, my name is Jared Rhodes

I'm an MVP for Microsoft Azure, a Pluralsight author. And if you need to contact me, you can use my email, my GitHub, my Twitter handle

And pretty much follows this theme if you need to find me anywhere on social media

I'm the owner and operator of commodity technologies. We do coaching. So if you need training

onsite training, learning, we do consulting as well. So if you need sort of high level architecture

planning or anything of that sort, we can do consulting. And then finally creating

we can actually do implementations and build software. And if you'd like to check out my latest course on Pluralsight, it's creating

responsive layouts in Xamarin forms. I've got content on Pluralsight for Xamarin, Azure, and

machine learning. OK, so in this presentation, what we're going to do is first we're going to do a

brief overview of the um i don't know if we should call it the mobile landscape at this point of the

application development landscape landscape um for anything that's not a um i don't know just

anything that's not server development so we'll quickly very quickly talk about wi-fi and lte

then we'll go over bluetooth and then we'll look at nfc and then lidar sms uh the camera and the

microphone all for these different devices. So let's get started with our overview

So we'll break it down into three categories for current development and again I'm trying not to

say mobile development and it's the word that keeps popping into my head but we really mobile is now

the phone the laptop and more. So we'll break down into three categories that'll be native

cross platform and hybrid. So for native development you have, and in this case we'll talk about it

in the mobile context, you really have two choices. You're going to want to develop for iOS

so if you want to do that natively, you're going to be using objective Swift

or excuse me, objective C and or Swift. You'll be using Apple's Xcode

and you'll be using the iOS SDK. If you want to do native Android

You'll be using Java and or Kotlin. You'll be using Android Studio

and the Android developer tools, and you'll be using the Android SDK

If you want to do cross platform development, you have a few more options

So you can use React Native. So React Native allows you to use TypeScript

and JavaScript and you can use the tools of your choice to develop native applications

So React Native will compile down into a native application. It's not just a web view rendering an HTML page

This actually creates a native binary for your application. So you can use the tools of your choice for that

anything that supports TypeScript and JavaScript. You can do Xamarin or Xamarin Forms

which allows you to use C-sharp and F-sharp. And you can create applications in Visual Studio

Visual Studio Code, and Writer. If you are more familiar with C and C++, you can do a C++ application using C++ or any other

language that will allow you to create a linkable library, and you can use the tools of your choice

iOS supports C++ through Clang. Android has the IndyK, so you can use C++, and you can actually

to build a complete application without ever having to write a line of Java or a line of

Objective-C or Swift. It's just, you're going to have, you probably, I mean, most likely you'll have to write a

little bit. But you can, using those NDK or Clang, create an entire application with just C++

And then finally, when I first gave this talk, I actually did it and I did those three

And then someone stopped me after and they said, well, you forgot, you know, you can create cross platform applications in all the different game engines and they were right I completely

missed it but you can create applications for Android iOS Windows Linux using unity unreal and

other game engines that give you integration to create a UI in that way

and then we have hybrid applications which I've seen less and less and less and less When I say hybrid applications

that means some form of an application that is not running true native

It doesn't compile down to the native components. It actually runs partially or fully in a web view

control running an HTML page. So there's Cordova. So Cordova is what was that called? PhoneGap

AirGap? PhoneGap? So Cordova is the PhoneGap open source? I forget exactly what Cordova's relationship is to PhoneGap

I know it is PhoneGap. I'm just forgetting exactly if it's the open source version or not

But you can use that and you can use JavaScript and HTML to write a complete application

Just understand that it's hosted in a web view on the device

And there are plugins that you can use to reach outside of that web view and interact with the native components

And then there's Ionic. and you can use ionic and sort of the same way giving you that html javascript build experience

to run on the app run on the device okay so now we've had a high level overview of mobile development

let's move into talking about wi-fi and lte very quickly because one of the things i want to do

during this talk is to give you an idea of what technology to use when and then what's available

to you through these different devices and obviously wi-fi and lte are available almost

all these devices. So when do we use Wi-Fi and LTE? Well we use it almost all the time without

even thinking about it. So we use it because we want the high data rate. If we want to do

video streaming or we want to do large content, large amount, large, what am I trying to say

You get high data rate. You have to use a lot of bits and a lot of bytes. You're going to have to use Wi-Fi or LTE. These other protocols aren't going to support it as well

But the main reason you're going to use Wi-Fi and LTE is because it's common

I'm sure everyone listening on this call is most likely either on the Wi-Fi where they

are or they using LTE on their mobile device Most places probably aren even hardwired anymore at least for whoever listening listening And then finally it obviously you going to want to use it to connect to the Internet

If I go ahead and I'm using these other things that were listed that I showed earlier, whether I'm using SMS or Bluetooth

the Internet isn't a easy access point at that from that. From that wireless protocol

So if I want to access servers or just any sort of backend I'm developing over the internet, I'm going to default to Wi-Fi and LTE

And that's really what I want to say about Wi-Fi and LTE because it's so ubiquitous in our development

I don't think I can teach you anything about Wi-Fi or LTE. What I'd rather do is I'd rather move on to talking about the other wireless protocols that you have available to you on these devices that you may not regularly use

So one of those would be Bluetooth. Bluetooth is pretty universal on devices nowadays

In fact, a lot of the chip manufacturers that make the Wi-Fi chips just have Bluetooth capabilities built in

So some of the uses for Bluetooth, you already know. The original Bluetooth component was the earpiece

to the point that we all know the person walking by at the airport or walking around outside talking to themselves

because they had the Bluetooth earpiece in. It has other uses as well

You can obviously use it with the same sort of technology to listen to music

but it can also be used to transfer files. I believe with the iPhone and a Mac or an iPhone and any other Apple product

it behind the scenes default connects over Bluetooth when you want to do things like local file share

So you can easily transfer those files over. And then we have tethering or just because I'm from Georgia, I'll call it piggybacking

So tethering or piggybacking so that I can connect my phone to another device over Bluetooth and then let that device, which is Internet capable

Be my Internet connection or vice versa. A device can connect to my device and now my device, which has LTE or Wi-Fi, can act as the Internet for that device

And you just do a protocol translation of Bluetooth to Wi-Fi or LTE

and that way each device doesn't have to have its own connection they can

share a connection. We'll go into this a little bit more but you can use it in

advertising there's actually a component of the Bluetooth spec that is built

specifically for advertising and you've probably had it on your phone and you

just haven't noticed but advertising actually built into the spec is a major

component of Bluetooth and finally you can use it for location services

So along with the advertising spec, Bluetooth has built-in capabilities so that you can basically broadcast

So let's say you're in a store. Normally I do this with the room that I'm in

but let's say that you are in a big room like a theater where someone's presenting

So you're in a theater where someone's presenting and there is in the room that you're in

there's a device that's broadcasting bluetooth signal and then in the room next door there's a

device also broadcasting bluetooth signal what you can do is you can actually take the signal

strength of the different access points for bluetooth sending these advertising packets

and you can determine how far you are away from each one and that'll actually give you an indoor

location you use this uh commonly when you need indoor location on top of gps so i use gps for

your outdoor location but once you get inside gps doesn't work as well so i can actually use

bluetooth and bluetooth beacons to track your indoor movements and i say track it sounds really

bad but you'd be surprised if you ever turn on your your phone and look at your bluetooth utilization

for like the walgreens app you'll probably see that the bluetooth is on in the background and

it actually is using beacons to help uh collect data while you're in the store

Alright, so Bluetooth can be broken down into two major specs. There is the Bluetooth basic rate and enhanced data rate

This is what I'll call original Bluetooth. And if you want to program for Bluetooth or develop for Bluetooth basic rate or enhanced data rate

you can think of it as developing for a client to server application

where you're just working on open sockets, meaning that there is a component where I can send bytes

and I can receive bytes, and I can do this as a stream. We can have an open conversation

where you get bytes, I get bytes back, and everything built on top of that is going to be

your application determining exactly what each byte means and how to have a conversation with the other side

And you use basic rate or enhanced data rate when you want to do things like the music streaming like the the headphones because you want that open streaming communication

As a Bluetooth 4.0 they came out with Bluetooth low energy and Bluetooth low energy had two major

components. One was Bluetooth low energy, well it's called low energy for a reason. The low energy

portion means that the device can go to sleep. So when you create the Bluetooth connection to a

device they actually talk to one another and they determine how often are we actually going to send

information and that determines how often the device will go to sleep and by going to sleep

if you think about if you think about how your chip the chips in your computer actually work

once you power them on they're always going they're always moving but they also have this ability

think about it when you put your computer into standby mode it goes into a low power state and

bluetooth low energy is a spec built for that so that we start a conversation we determine how often

we're going to talk and then you determine or once we determine that you can now go to sleep when

we're not going to talk and you can enter that low power mode to give you an idea if you go into

low power mode let's say half the time so we decide we're going to talk and you're going to go to sleep

one second out of every two seconds then we can extend the life of the battery two times well if

we do that one second every 10 seconds is when we're going to talk well then we can extend the

battery life nine to ten times In Bluetooth Low Energy, we have three different topologies

So one is point to point. So point to point is, oh, you know what

I skipped it. Let me step back actually. I forgot one thing. I told you about Bluetooth Low Energy being Low Energy

I forgot the second really important part, which is now there is contract-based communication

So instead of just having raw bits and bytes that are being handed back and forth and you're determining everything

Bluetooth Low Energy actually has the idea of a contract. And so you can create a contract

and we'll go into that a little bit more later so that now instead of building like you're creating on top of a raw socket

you can think of it more as building an application like you're building an HTTP API

so that you can send this command response style of communication. Okay, so now moving into the topologies

We have three topologies for Bluetooth. So one is point to point

For point to point, you can guess, right? We have a client, we have a server, we connect

And that is point to point. Now we can communicate directly. Broadcast

So broadcast is what I was talking about earlier with the Bluetooth beat or the Bluetooth advertising So now you can create a Bluetooth device put it in somewhere

and all it does is it broadcast a set. A set of bytes, I always call it bytes, a set of packets that

now anyone roaming and listening for Bluetooth broadcast can hear. So it has a very specific format of this is me telling you

who I am, this is me telling you what company I am, and then here's the actual broadcast that I'm

trying to send you. And then as the receiver, you can retrieve that broadcast and get the information

out of it. Because once we've established that first part of me telling you who I am, what company

I am, the rest of that packet can change. And this is used all the time for various different things

We already talked about the Bluetooth beacons for indoor GPS. But when this was first used, it was

I shouldn't say when it was first used, but one of the original uses I think was the

World Cup in Brazil. And Coca-Cola was using it so that when you get close to the Coca-Cola

booth, it would just be sending out a broadcast and a little notification would pop up on your phone

saying, come get a free Coke over here at the Coca-Cola booth. And companies can use this for

all kinds of different things to broadcast. We can send information to you that we can use within

our application that you don't need directly at all times but if there's something important i

can show you a notification and i can do this advertising through the broadcast spec or the

broadcast topology and then the lesser used mesh topology mesh topology is if we go back to the

scenario where you're in a room and there's a bluetooth device that you can connect to and then

in the next room there's a bluetooth device let's keep that going let's say there's another device

device and another device and at the end of all those devices that are way away from you

The last device can connect to the Internet. What the mesh spec will allow for is if I can connect to the one that's local to me and

the mesh spec has all the different ones routed out to the Internet interconnected in the mesh

topology, it will automatically route to the one that I'm asking for over by to get the Internet

So I connect to the local one. I'm making a request to the one that's over too far

for me to connect directly to. The mesh back will actually just figure out, okay, who's all in my network

And then it'll route you through all the different ones that needs to to finally get you to where your request was sent

and send you a response. So I'm going to start showing you some code

This is Xamarin code, so it's C sharp. So if you're a C sharp developer, it'll look familiar

For each of these that I'm showing you, this is actually just Xamarin's way of facilitating

the native Java or Swift in C Sharp. So that it's C Sharp Lite

These will look different if you're actually programming them in the different languages I talked about

But these are the C Sharp Xamarin code snippets. So let's look at connecting to a Bluetooth device

It's actually as easy as this. say we have a device list and every time we connect to a device we just want to add that device

well once we have our bluetooth adapter which is a global variable we just do adapter

scan for devices and when it discovers one we can add it to a list of devices

okay so we talked about the bluetooth low energy spec and i want to go into that a little bit more

because that actually allows you to connect easily and easily create devices for your

applications or clients for the app clients for the devices for your applications so there's the

gat server so the generic attribute profile the gap is built on top of the attribute protocol act

and establishes common operations and a framework for the data transported and stored by the attribute

protocol okay that doesn't make any sense let me break it down for you so that it makes some sense

So the GAT server can be broken up into four major categories and they're hierarchical

meaning at the top level, there's a profile and a profile is pretty straightforward

If you were to, let's take open API as an example to relate to. So if you want to develop for an API

and it has an open API spec, you can go, you can grab that open API spec and now you know

of the contract that's set up for you to use it tells you all the different methods and whatever

you can use for this api the profile is the same way except the profile is at the device level

so i'm connecting to this device and this device profile tells me all the different things it can

do except it's broken down a little bit differently it actually tells me all of its services so i go

I ask the device. I say, hey device, you support Bluetooth low energy because we connected. It says

yes, I do. Here's my profile. I go, OK, that's a cool looking profile. What services are in this

profile? Well, within the services, it tells me a set of characteristics. In the open API world

we can think of it as the service is actually what relates to the open API spec. The profile

would be more of looking at a set of APIs that you need to integrate with. The service is the

individual API and the characteristic would be the rest component of it. So let's say that there was

a person and I wanted to be able to get and put and post or create, delete that person

Well, characteristic would be equivalent to that. The characteristic is that entity that I want to

operate on and i'm sort of using if there's any people who do bluetooth i i'm i'm gearing this

more towards developers um i know that the characteristics aren't entities but you get

the idea as a developer you would think of it more as here's the entity that i want to operate on

and that's the characteristic now within that characteristic i can look at different descriptors

properties and the value of that characteristic okay probably so doesn't make much sense so let's

let's break down a little bit further. So we have the profile. That's what the device just gave me

and it said this is all the things that I can do and all things are a list of services

Within that list of services, each of those services could have different characteristics

In each of those characteristics, I can then now ask for the different properties

the values, and the descriptors of that characteristic. So when I actually programming

I probably already know the services that are going to be available to

me when I connect to a profile. Because think about it when you

program an open API against an API. You don't query the API and dynamically

figure out what am I going to be. No, probably before that you

generated a whole bunch of code. You decided which endpoints you were

going to interact with for your application. And you didn't really care after that

when you connected. What you really did was you said, OK, well, there's a characteristic of a

person so let me connect and I want to talk to that characteristic So I want to create a new person so I send a message to that characteristic And it creates a response and sends it back

So let's take a look at a predefined. So the Bluetooth spec

the Bluetooth low energy spec has pre-built services that you can look at the definition of

And that way, if you actually want to create a application that can connect

two different Bluetooth devices that already have predefined services, they're common. You know what they would look like

So the heart rate service is known. So it actually has a unique identifier

When you pull a service in, you can check against that unique identifier. And if it is the heart rate service

you can already have a predefined way to interact with it. Well, let's take a look at the heart rate service

It's gonna have two characteristics and a descriptor in the service. And the characteristics, if I'm remembering correctly

are the beats per minute. And then there's another characteristic that you use to

what was the other characteristic? I'm forgetting off the top of my head, but let's just worry about that beats per minute characteristic

So that beat per minute character, oh, I think the other one is, I think the other one is so that I can set alerts

So I've got beats per minute and that is a read field

I can read beats per minute. I can't write to beats per minute

I can only read, right? Because it's a sensor, reading values would make no sense to write to it but then there's another characteristic i believe

that i should be able to write to so that i can see i can set alerts i can say if beats per minute

goes above x or y now i want to receive alerts normally i would ask are there any questions

but uh if there are i don't see any so we're moving on this is some code for once we've got

that device you remember we made that device list earlier now let's interact with the devices

that we've talked about so if i want to get the list of services from the profile after i've

connected to the device i just say get the services async and if i want to get the characteristics

from that service i just call a get characteristics async and if i want to read the characteristics

it'll send me the byte array and i just say read that characteristic if i want to write to the

the characteristic I just write and I send to the bytes. If I want to receive alerts for something

like the heart rate monitor, I can actually subscribe to the update events by just giving the

events a callback and the calling start updates async. If I want to get the descriptors, you can

guess. I just call get descriptors async. If I want to read that descriptor, I can just read it

If I want to write to it, I can just write to it. Seems pretty simple, right

We can talk a little bit about the physical layer because Bluetooth 5 added stuff to the physical layer that's kind of important

not truly important to most people in this talk, but it will cover it anyways just in case it comes up

So if we look at the Bluetooth physical layer, you'll see how apps

you know, we've got apps up here at the top and then it's highlighted and it says apps and then below that's host

And then the first thing in host is the generic access profile

As an application developer, you're done. You're probably not going to go anywhere below the generic access profile

The physical layer, the link layer, the direct test mode, host controller

logical link, attribute protocol, security manager. If you're building hardware, you're still most likely going to have all that provided to you by

the hardware manufacturer of the Bluetooth chip itself. However, if you are creating Bluetooth chips

then the rest of those layers are important to you. Changes in Bluetooth 5 to the physical layer

So they added three new physical layers. So what we've done is we've renamed Bluetooth Low Energy

The original spec is now called, the original physical layer is called LE1M

So all Bluetooth 5 devices have to have one or LE1M, it's required for all devices

Meaning that all Bluetooth 5 devices are Bluetooth 4 compatible by default

On top of LE and LE1M, we now have LE2M. So now when the devices start a conversation

they can actually swap out of their original mode of 1M and they can swap to a new physical layer, 2M

So now they can actually have a higher data rate between the two of them

There's also LE coded, so we can do that for longer range. If we start a conversation, we say

hey, I'm gonna walk away. We can change physical modes and now we can have a long range conversation

So you have an idea of the changes. There's two LE coded for distance

so you can get two times as far, four times as far in your data rate, half about each time

Now you're still looking at 125 kilobits per second, which is better than Comcast gives me sometimes

So it's still not a bad data rate and you get a decently large range out of your device

And then for LE2M, you get double the speed, but a little bit less range

and they're all optional, and there's different error correction. Again, not that important to application developers

but in case you run into a Bluetooth project and all of a sudden you need to learn about Bluetooth 5

these are the primary differences. Okay, so we talked at length about Bluetooth

because as a spec, it is a lot of devices supported now

Your car supports it, a lot of devices in your house. Let's talk about some of the lesser used

wireless capabilities available to you on your device that you can still utilize

So let's talk this time about near field communication. So near field communication is a form

of contactless communication between devices like smartphones or tablets. So that contactless communication allows you

to wave your smartphone over the NSE compatible device, and you can send information without needing to go

through multiple steps or setting up a connection. Okay, so for near field communication

talk about the uses and then i want to talk about the the nfc standard and this will go a lot faster

than bluetooth so if you're in for a long break bluetooth was the one we covered the most okay

so nfc is useful for let's say you want to connect two devices right nfc allows you to get two

devices close together and now we can swap over to a different protocol that gives us longer range

and you see devices doing this all the time with sort of a tap to connect and then they use a

different communication protocol after that and you can use it for different things too if you

want to let's say you're installing water meters or an appliance you can just go ahead install it

and then have the person who's using your device just open the application tap to whatever that

that installation was and if that device is nfc and you've programmed it correctly

you can use the phone's um the phone's internet connections now register that in your service so

we do this in the field for let's take example of a water meter what i want to be able to do is once

the installer is done installing the water meter they now open the application they tap it i grab

their user id that they're logged in with and i grab the geolocation and the timestamp and we log

all of that and then we start provisioning the device and our back-end services to connect

over the nfc so that way when the installer's done all they do is tap the phone and that allows the

initial communication to start all the different provisioning in the background And you can do it if you want to

like you would Bluetooth, right? If you want to, there's a good one here

disabling your residential alarm. I used to have something set up on my computer

my tower, the same right next to me. I could just bring my phone over the NFC chip

and it would turn on the computer without me having to press the button

It was a neat project, but I realized I could just press the button after a while

OK, so this NFC standard we can break down into two major components, tags and modes

So for tags, there are four different tag types and we're going to see if I can remember these off the top of my head

because I didn't put them in my notes. I'm used to doing this in person. So the first two are type 1 and type 2. Small and extra

small. I believe the difference in small and extra small is small is 128 bytes and extra small is 64 bytes

That's the total amount of information you can pack onto I can put a GUID on an extra small and it's fine

So if I'm doing device provisioning, I can get a GUID, bring it to the card

put that right to the card, put that card into the device

and now whenever I scan, the unique identifier for that device is readily available

For small and extra small in some cases, I can actually fit a whole URL in there

so that if I need even more information, I can keep that information dynamic

but I can use the URL and I can that's unique for that tag to go get

And the reason there's a type one small and type two extra small, and there's such a little difference in it's so small in bytes

just has to do with price. If you want to manufacture extra small type twos

you can create, I mean, if you save a dollar going to extra small from small, usually when you're provisioning NFC

tags, and you type it into millions. So if we can save a dollar just going extra small

we saved a million dollars. There is type three, which is Felicia and if memory serves Felicia just has to do with the Japanese card reader technology. So in Japan they have a specific spec for card reader technology and Felicia is the tag type that is compliant with Japanese card readers. So it just exists for compliance reasons

And then finally type four is special. Type four is locked and that is actually meant for rewrite for an actual communication and we'll get into that in just a second

That takes us into the two different types of modes. Now the mode we've talked about up to this point has been reader, writer and card mode. So reader, writer and card mode is exactly what you would think. You can see here we've got the tap to pay graphic, which is reader and writer. One is the reader, one is the writer, and they communicate in one burst of information

But if you want to get really fancy, and you can, you can actually go into initiator and target mode

And what initiator and target mode does is, let's say we start in this reader, writer, and card mode

where one reads and the other one writes. Well, what if we just flipped it after we had our first read and write

And now we're communicating the other way. So you can actually create two NFC devices to have a full conversation

and it's actually a surprisingly high-throughput conversation if you need to transfer data or have a full conversation

full conversation so that two devices can connect to one another and then rotate NFC mode and actually

have a full conversation. Last one when I made this on my last update I didn't update it for

this talk so there may be better Xamarin support for NFC but right now you need to do it

to where you expose the NFC on each platform. So let's say we're doing it on android if I want to

turn my device into a NFC beacon on Android, I could just go through, I create an in-depth record

I figure out what kind of mine type and what kind of payload I want to put on there

and then I just create that and it's a byte array embedded into an in-depth record

And then whenever I want to do it, so if I'm creating that in-depth message

you can see it right here, I'm just going to grab whatever text I want, I create that indie

message with an indie ref in-depth record array and then I create my mind record the previous

slide for each message within for each record within the message so there's a message there's

an array of records and within that I'm looking at each message so for the longest time NFC wasn't

available on iOS. It was actually on the device. There was an NFC chip and Apple used it internally

for Apple Pay. And for the longest time, they just never exposed it through the SDK

Now it's available since, I don't know, 13, 11, something. It's been available for, I guess

in mobile terms a while, but still pretty recent. And if you want to do it on iOS, it's always

it's always a simple you just in a nfc in-depth reader session and you call begin session you'll

see this screen ready to scan and then whatever you put in the info p list for the object name

and then you have callbacks so did it detect and then if it did you'll get a list of messages you

can grab those messages and if it did not you'll get an um an invalid message error and you can

handle that as well. Again, this is the point where I would ask are there any questions and I don't see any in chat, so moving on. This is the one I actually added for today's talk. This is the update. I should have updated everything else, but this is the update I made for today. So we're going to talk about LiDAR. So LiDAR is a method for measuring distance by illuminating the target with a laser light and measuring the reflection with a sensor. So the difference in the return times and the wavelength can be used to make a digital 3D representation of the target

What does that mean? Okay, so let's say that we have a blue square, which is our LIDAR sensor

And so we go ahead and we want to scan the area around us using LIDAR

So we send it out and then it hits an object. So the laser returned quicker and then it hit another object, it returned quicker

So what happens is, is actually on our end, as LIDAR detects, we're going to start seeing these different

obscurements in our local area. So we're using anything about it. It's like our its radar

It's actually figuring out where objects are in the 3D space. Why you probably like Jerry

Why are you telling me about LIDAR? That's neat, but we talking about devices here Also recently Apple announced that the new iPad Pro is going to have a LiDAR detector built into or LiDAR sensor built into it

Currently, there's no plans to expose LiDAR directly through the SDK. But the purpose of the LiDAR detector, it can have it can

have plenty of use cases. So let's go over the different use cases you can have for LiDAR. This is general use cases, so

can have agricultural robots they can use lidar for a variety of purposes for ranging so if it's

trying to disperse seeds or fertilizer it uses sensing techniques as well as just crop scouting

and looking for weeds all kinds of stuff are done in agriculture with lidar autonomous vehicles so

for most autonomous vehicle research going on right now and even in the i don't know i think

and the Tesla and a few others, they have LIDAR sensors built in so that the system can get a 3D

measure of what's going on around it to determine or to help it make decisions, right? You want to know

where the distance is between you and other things real time, not just by trying

to look at it on a camera, but getting this LIDAR information as well. There's military applications

applications. So there's few that are public. There's a lidar based speed measurement of the AGM

129 ACM stealth nuclear cruise missile. But there's a considerable amount of research underway in use

for them, so they use it for higher resolution systems to collect enough data to identify targets like tanks

So the military applications include the airborne laser mine detection system for counter mine warfare and a few of that there's a few other different

things that LIDAR is used for that is publicly known and a lot of different stuff that is

still classified. So airborne LIDAR sensors are used by companies for remote field sensing

So survey. So one of the projects I had heard about was we were working in a facility for emergency

first responders. And in that facility, they actually could blow up a building and then train on that blown up building

So one of the projects that was going on there was actually using LIDAR attached to drones

And what you were trying to do was you were trying to give the first responders information about the building layout as it had collapsed

So using LIDAR to sort of get a feeling of the terrain and giving that information to first responders when they couldn't get it elsewhere

It's used heavily in atmospheric research and atmosphere sensing and it's very different in that case

So we won't go into it too much, but just understand that once you send the laser into the atmosphere

it disperses in certain patterns and you can actually get information. The big one for the iPad Pro is augmented reality

Because once you're doing augmented reality and you're building those 3D meshes around you and in the space that you're going to be

I don't know, I don't want to call it playing because it's mainly for games, mainly for games, but in that space that you're going to be working in, augmented reality can be

enhanced by having that LiDAR sensor help build out the 3D world that you're going to be operating

in so that when you take the classic example is in Unity or Unreal, you just take a sphere and

you give it physical properties and then you start the game and it'll just drop and it'll roll around

in the world that you're in, that you're looking at, and it understands where the slope is and

understands where that is and that can all be enhanced on the iPad Pro with LiDAR and I think

you'll see that in more and more applications or more and more devices going forward because of

that application. All right I'm probably going to start talking faster because I'm looking at the

clock and I want to get everything in but if you have any questions just type them in chat

Let's talk about SMS we all know what SMS is on our phone but what we primarily use it for

is texting we can send text to each other or we can send multimedia messages right we can send

gifs and videos and whatever but uh what's often not used but it's available within the spec

is the binary mode and you can send binary messages um if you're working at the device

level as opposed to more of the application level so sms is the short message service

So as it's used on modern devices, it originated from radio, from radios and radio memo pagers

that use standardized phone protocols. And these were defined in 1985 as part of the Global Systems for Mobile Communication

Series of Standards. And the first message was sent in 92. So it predates probably some of you on this call

So for text, there's a couple of different use cases you can get out of SMS

can create a chatbot so you can have it to where someone can just text the number it hits your

back-end services and then your back-end services can interpret what that message means and you have

a full conversation that way without even having to make an app a mobile app you can just have the

SMS endpoint be your entire interface for your user or just a component of your interface

you can do it for notifications so I don't know I get the notifications from the Walmart pharmacy

hey your prescription is ready and then it just shows up on my phone and and that's what a lot

of companies use um and i'm sure we all use it too for the would you like to sign in for your

two-factor here's your code and then you can just send data raw data packaged up in a way and since

the device can read the sms message you can actually look at the text maybe it's a url parse the url

and react to it. So if we're using Xamarin and we're on a device, this is how simple it is

send an SMS message. We just create an SMS message, we do compose a sync with that message

after setting the recipients and we're done. This will actually just open the native SMS

application on your phone with the message populated. If you want to receive it on the

the back end, I'm actually unintentionally wearing the Twilio shirt, but you can use Twilio and it's

actually really easy to set up a Twilio SMS ASP net controller that will listen to the incoming

messages for the incoming messages and you can easily respond to it. As you can see from this

code, I just create the post endpoint, I create a Twilio response with a message of hello from and

and then you said this and I just return it and Twilio handles all that behind the scenes

So you can also on top of SMS you can send multimedia

which we all used to So the multimedia messaging service is actually a different spec on top of SMS So it extends the core and it allows the exchange of text messages greater than the 160 characters that you limited to on SMS

So what can you do with MMS? Well you can send about 40 seconds of video. You can send an image

You can send a slideshow, probably one smaller than the one you're watching right now, or you can send audio

So the SMS specification has two modes in which a GSM or GPRS modem or mobile phone can operate

And they are called SMS text mode and SMS PDU mode for protocol data unit. And protocol data unit

is how we're going to be able to send just a packet of bytes one way so that we can just do a sort of

binary communication. I don't know where I was going with that. Yeah, so that way we can do a

binary communication. So if we have it, what we do in the field sometimes is we'll have devices that

go farther out than the LTE is covered, and we can actually swap over to PDU mode for SMS

and we can send out binary messages over SMS, capture them on the back end through Twilio

and that way we can continue our conversation with the device even once it goes far out

way far out into the field. And that is sometimes for compliance reasons too

Let's say that you're logistics hardware and you're going across the US border

Federal law, we have to track that. So we have to keep in constant communication

I just to give you an idea for, I should have updated this too. This is when I first made this slide

this is what it cost to send messages through Twilio. And you can see the idea, you get the idea, right

I mean, 0.0075 cents for a message and then .02 to send

It's amazingly cheap, and for you to actually run up a noticeable bill

you really gotta send a lot of messages. Okay, so that's SMS in a nutshell

We've still got the camera and the microphone, so we'll go through those really quickly so that I can get to the questions at the end

Okay, so the camera, let's cover five things. We're gonna cover object recognition, computer vision

barcodes, optical character recognition, and facial recognition. Some of these overlap, but you get the general idea

These are all use cases that we have for the camera on these devices

So for image processing, you can think of it, or at least I think of it in four different ways

So one would be formatted images, which we'll go over in the next slide

There are algorithmic ways to take a look at images. There are data-driven ways to look at images

or process images. And then finally, you can combine those algorithmic and data driven and have a multi factor approach

So formatted images, you've already seen them in this presentation, right? Barcodes and QR codes

I was doing work for National Cash Register and if you go up

and use the NCR checkout self checkout, you can just use the

mobile app for most of these stores and it'll present you with a QR code. You just scan that QR code and you check out

Right, so I can actually embed a large amount of information. I've actually got one of my side projects. These are defunct eyes on view cameras and the way they connect when you kick them off is they use the camera built into them and your phone actually creates the QR code and it reads the QR code off of your phone so that it knows how to connect to your Wi-Fi and provision itself

So you can use formatted images both to send messages to your device, or you can create a QR code to send messages to another device as a way of sending that dynamic information

Sort of the same way the NFC can do reader writer, right? We can have a full conversation with barcodes if we wanted to get really fancy

Barcode scanning in Xamarin is a pretty simple and robust at this point

If you use the ZXing, ZXing, I don't know how exactly pronounce it, but ZXING NUGA package

you just, there's a barcode scan, click event, so you grab the barcode scanner object, you got a

click event, and then you just, hey, give me a scan and tell me what the results of the text

coming out of that barcode scan is. Simple as that. So I didn't update my QR codes. So last time I

this talk that was actually a qr code with the the url of the talk of the conference i was at

and then here's one for my plural site course i can just you just scan those barcode scanner

it takes you straight to that website for algorithmic image processing we've got a couple

of different things i'm gonna go through them very quick these are full topics you can you can have

experts that have lifetime experience with them i don't have that i'll go through them really quick

so that we can stay on time so you can use histograms histograms give you a general idea

of color gradients so if I take a look at an image and it is green let's say take a picture of a

forest I can use a histogram of that to compare to other images this works for quick matching right

if I have pictures that have nowhere near the same color gradient it's an automatic rejection

I know these pictures aren't similar however it can sometimes see a forest a green forest as a pile

of money and it just wouldn't be able to tell the difference so it's for that quick gradient check

and you can filter out images quickly. You can do template matching which is sort of a way to do

let's say we take a large picture of the Mona Lisa and a small picture of the Mona Lisa. If I want

to find the large picture of the Mona Lisa and I have the picture of the small, I can use template

matching to basically look for that exact picture but maybe not the same size. So template matching

will be able to extrapolate say hey that's the same picture if we expand it out and it's the exact

replica of this picture and then we have feature matching feature matching is a much more in-depth

topic but you can do something like a key point ysis which is using different algorithms that

can look at an image and determine what are the and we'll put it in quotes important parts of this

image and you can take two images take those important parts of those images and compare the

distance between the important parts and determine how closely related these images are and that's

a good way to take it if you're looking for images of things that just are not in the same rotation

orientation or maybe not even fully similar but look similar for data driven you interrupt you

jared i want to interrupt you and let you know what uh we're 10 till the hour and we do have one

question oh i do where are we at What your opinion on that net Maui anything will dominate cross platform You OK so the question is what your opinion about that

Maui and do you think it will dominate the cross platform UI market? As of right now, I don't want to shoot myself in the

foot, but it seems like Maui is a rebranding of forms as they recreate

As they recreate it to sort of take away some of the stuff they

don't like anymore. I mean, Xamarin. Forms is now how old is it

8? 7? 8 years? It's old enough six years. It's old enough to where it's generated

enough or the market and the libraries have changed enough and C Sharp has

changed that I think they're just sort of doing it to get a fresh store in the

same way ASP net core was. So I think Maui will dominate sort of

If you think basically, I was just going to be forms again

However you think forms is doing is how how MAUI will do. For me, Forms has been great. I've made tons of apps. They're in the store

They do well. They're performant. I think MAUI will just be a more performant way to do that

once they hammer everything out. Okay. So I'll be wrapping up shortly. I've got this, the microphone

and then put your questions in and I'll keep a lookout for them. Okay. So for the data-driven

approaches, you're thinking about the machine learning and the AI-driven approaches. So just

go through this real quick you can actually still use opencv so opencv is still available and it

actually has a bunch of the different stuff that we a lot of people think of tensorflow or or any of

the other machine learning pytorch opencv has a lot of stuff built in that is in tensorflow and in

pytorch available to it you can actually do some pretty uh intense machine learning for images

in it and there's also as i said tensorflow core ml so i put tensorflow in core ml specifically

because tensorflow for android when you want to run your model and then core ml for ios

i'm going to skip through the next slide because i'm not irrelevant it's um cognitive services and

they're real easy to do computer vision with and facial recognition and stuff like that you can

easily just send it a file and it gives you a list of features back that it thinks that file has

so the microphone all right microphone you can really think of it in and for two separate things

One would be audio detection and language processing. So for audio detection, what we're doing is we're, we're

it's a process that we can infer if a sound or sounds is found within a set of sounds

So what does that mean? So there are these audio detection devices on the

stoplights in places like Chicago, and those devices can actually be used

use they detect gunshots and if you have multiple of them you can detect the center point for those

gunshots going off and it helps for response times. You can also use audio detection let's say we're on

a factory floor so if you're on a factory floor you can use audio detection to sort of determine

is this person in a place they're not supposed to be is this a dangerous piece of equipment and we've

got the audio profile for it and have they wandered too closely to that when they're supposed to be

outside of it and I can't track you any other way because of all of the other and we'll say noise

but I mean more wireless noise on the factory floor. The other way the microphone can be used is

language. So we all remember back when Siri, OK Google, Cortana were supposed to be the

I mean the life-changing things that they were supposed to be and we still are supposed to have

that with the Alexa so you can use it for assistance right just let me let me set up set a meeting in my calendar

intent recognition um and sort of the same way you can do the chatbot as we talked about earlier

now we're just instead of text we're using direct audio if you're using cognitive services with

lewis you can use the audio stream directly you don't even have to worry about parsing out the text

text. In your own code or back end service, so you just tell it

you know, speaking to the phone and on the back end it will do

the intent and it's the recognition for you. And then finally search I want pizza

Where are they serving pizza nearby? This is an example of a Lewis request, so if I want to send

in that request to Lewis, I'm showing you the text version just because it's easier than showing you the

it just makes more sense than showing you the audio version because the audio version is just an array of bytes

So for this, you want a prediction request. So turn on the bedroom light

I send that up to Lewis. This is how I call it

I say, get the slot prediction. I give it my app ID

I tell it the slot name, production, and then I give it my request. The response it gives me will look something like this

I just say get the prediction async. It'll give me what my query was

and it'll tell me all my different intents, ranking them by the probability

that that was the intent I was looking for. And then as a helper, it'll just tell me which was the top intent

so that I can just quickly say what do I think they were looking for

and create an action based on that. Okay, so what did we talk about

Well, we went over, what do we talk about? We did a brief overview of mobile development

or device development. I keep going mobile device development. And then we looked at quickly Wi-Fi and LTE, not quickly, Bluetooth, and then NFC, LIDAR, SMS, the camera, and the microphone

So I'm going to go back and see if there are any more questions. Nope, just .NET Mac. All right

This is the QR code for your speaker feedback and your event feedback

So if you would like to now, go ahead and scan those. Remember if you stand the speaker feedback

it's only if it's out of five stars, it's only five stars, right

In fact, if you didn't like the topic, don't give any feedback

And then you can also do the event feedback as well. So I was just killing times and giving you time to scan it

Hopefully you scan it by now. And that's it. I'm Jared Rhodes

and this was a talk about controlling or running the world from the palm of your hand

Thank you for coming