Page 1

SUNY Technology Colleges ​augmented cognition that's the answer I want to say that right up front so my whole talk the thing to remember is augmented cognition because I think that's really the next X that's the next brain is what I've kind of talked about i'm going to share some of those thoughts with you today that will key into what i mean by augmented cognition first of all they mentioned I'm doing this I I've had a great time and what I really want to see is how do we make people more prepared more informed more capable I don't know about you but I sometimes feel like things haven't progressed as fast as I had wanted them to in terms of the future when I was five years old I was told by the time I was 25 there'd be people on Mars that we would have a colony on the moon seriously just not popular not science fiction really believe this when I was 25 didn't really happen that way now when I was 25 they said when you're 45 will be all this technology around now we've got a little better but we've kind of fallen into a little trap that we sometimes look around us there is so much technology around us you have a flat panel I mean how many computers do you carry on your body now you have an iphone and android tablet your iPad a galaxy samsung device your laptop you just walk around with more computational power than pry this the space program had in the 60s so I also hear that our refrigerators are dryclean you know everything in your house is going to have computers in it I don't have time to interact with every device that I that's around me I mean I don't need to be how do I interconnect things how do I relate to it right now I'm afraid I'm very afraid when I walk around the streets of Philadelphia then I'm going to step into traffic I'm looking at my device sometimes I you know I'm feeling different ways you know not only emotions maybe I'm tired Dylan tomorrow this morning is not the same Dylan tomorrow last night isn't the same Dylan tomorrow after lunch what's going on in our bodies is constantly changing so how come all the technology around me doesn't know what's going on inside of me we spent a lot of time with our sensors and technology understanding what's going on in the world what's going on in the physical space hey there's a parking spot available over there you can have your device tell you that you can have a device tell you this is what's going on in the virtual space right now there's so many of these things but I'm overwhelmed with information so comes down to this how can technology help us reap the rewards of the digital age without overwhelming us with complex systems and impossible interfaces or if you like bumper stickers how do we remove the burdens of technology because if you think about it we're supposed to be empowered by all this technology but in so many ways it's a burden it's a burden i'm looking at it i'm not ready for it it chimes you got to make sure you turn your phone off when you're up here so that you know i don't want to get distracted and this isn't a novel concept that wouldn't be neat if a system knew what was going on inside of me had some good behavior measures of what was going on around me knew what was going on in the virtual world the real will do all this and then could intelligently adapt to me could support me could make it seamless so instead of humancomputer interaction which I love we could have human-computer symbiosis where it really is this completely transparent interaction and it's in it's this interaction not with the device it's me doing what I need to do without the burdens of Technology so why can we do this now what I'm sharing with you isn't radical you know people have thought of this well before me Moore's law so we throw that up there Moore's laws up there Moore's law means Wow anything's possible from a technology standpoint they're making things faster and faster we hear about how they're going to you know it's just all there but that's also part of the problem but it's also part of the solution so a couple other things that I think really make what I'm going to share with you augmented cognition is there's two other big things that have happened why is it possible not only from the technology standpoint but there's been the cognitive revolution the decade of the brain things like this suggests that now is the time that we can actually have humancomputer symbiosis that I can have that system understand what I'm doing so I want to i'm going to share with you two videos today very short clips just as a teaser to show that things are possible and what I'm talking about because really what I'm hoping to convey with this is a truly interactive symbiosis the human computer it's a partnership again I don't want to say well I'm walking around and interacting with technology all the time I'm not suggesting that it's you know i'm i'm having to deal with robots I don't want it i want to just live my life but I want it to self synchronize around me here's an example let's say you're a professor I'm teaching a class it's a big one lots of people i'm in i'm in the

groove i'm really doing it i'm communicating to the students one-on-one and my best friend that's known me for like 30 years who happens to work there too he comes in the back and he wants to have lunch after my class now if in the middle of my speech he walks and goes hey Dylan you want to have some pizza in about 10 minutes you know I might go hope what okay yeah yeah yeah yeah man I want to have some pizza oh shoot what was I talking about cuz wasn't my brain just kinda I was doing a verbal task right there my brain was completely focused on that now if my friends know me now this why say he's known me for a long time he knows how to interact with me and communicate with me he can sense no you can't necessarily read my mind but if he had a good friend like that he knows how to communicate what if he walks in and just kind of goes well I'm in the middle of talking waves to me kind of goes ten minute you know kind of his hold his hands up shows himself eating pizza and I can just keep talking to everybody like am now and just go you know like that and he knows okay yep I can do it if I just go I can't do he hasn't interfered what is it it's almost as if he could read something going on in my brain and adapt me in real time so how do we do that for real a little 10 second clip here computational partners activated you neural interface engaged preferences important okay that's a little teaser a little bit longer movie but I just showed you some people what is that well it'll today right now how does my systems whatever devices i have the you know the InfoSphere ami how does it know what's going on well again as I said earlier their sensors devices everywhere that kind of keeps track where there's an open parking spot what's if it's going to rain all this other information but I would kind of like something and it doesn't have to be I'm walking around with a huge brain cap that looks like I'm in the hospital but where it can sense that now one important distinction here is right now it's really cool because in our society we're very excited about brain computer interaction and brain computer interfaces however a lot of that or most of it if not all of it is focused on my brain is going to control this little car or my brain is going to control this cursor on the screen that goes up and down so we can play brain pong so I'm the play brain pong against whom we can just sit there in our brains could oh my goodness why on earth that's kind of cool but why on earth would we do that I can play bruh I can play pong really easy with my hand I can drive a little toy car really easy I have good monkey skills you know I'm good I can do stuff I don't have to be using my brain that does a lot of cognitive thought that thinks to like do these very mundane things now there might be examples we want that certainly something like a prosthetic I might need that but hopefully for many people were all overwhelmed with the world going on around us wouldn't it be neat though is instead if I had a device that could actually measure my cognitive state now when I use that word I'm a good skin Aryan I want to be careful cognitive state it's not a definitive thing I'm but and I get in a lot of arguments with some friends some good scientists Dylan but you don't understand the fundamental mechanisms of the brain we you know another 100 years of basic science how long ago man we're always going to be studying the brain we need to it's phenomenal but you know what sometimes have enough information have enough information would it be neat did you know like you could put like a little sensor on my forehead just kind of look here and you could tell whether I'm having in verbal or spatial working memory so think about those cartoons excuse me my brain is full we can build a gauge now that can tell you that your verbal working memory it's full or it has capacity your spatial working memory is full or has capacity we have that ability really what are we talking about there is closed loop systems using modern tools now again you don't necessarily have to understand the fundamental mechanisms of the brain I'm trying to learn as much as I can my colleagues are I support lots of basic science in the brain because it makes it easier but we have things right now that can adapt not to play brain pong not necessarily play games although they're fun and it's not necessarily to even consciously train my brain so as I get older those are all good things but imagine everybody magic madness if if you didn't have a little bit of problem with the tension did you know that I can actually build a gauge and people have that can detect whether you're paying attention not just general so already people in education are starting to use this with people who have maybe some attentional problems tensional challenges but right now would be cool so if my wife called me right now and said hey Dylan make sure you pick up some milk on the way home tonight and I go okay I got it honey my system go whoa whoa no you do not got it you did not encode that into your long-term memory it did not signal the p300 response bring home some milk I go yeah okay I got it okay you really do have it but I want to tell you that because the technology exists today to do that I mean it even exists today that i can with a nice camera look into your eye look in the way not just your I direction where you're looking that's cool but that's not what I'm talking about the way your pupil is twitching we can actually tell with fairly good actors if you're doing a math task or a verbal task in your brain wow I want these devices to sense all of that and really here's just some examples I mean it's got to be down right now the way you do it is we have some high density EEG so we have people can make those and really understand it but remember i said i don't need to understand way too much i want to have 21st century humancomputer interaction 21st century human systems integration I want 21st century human-computer symbiosis and I believe with augmented cognition we can do that we can do that by sensing what's going on in the brain the body and

you know what I'm not discounting everything else I want to include all that other stuff too so really what are we talking about a closed-loop system a system so that those burdens of technology have been moved away that can really anticipate predict in augment I honestly used to think when i first started down this about a decade ago the hard part was going to be detecting cognitive state what these states mean turns out that's it's a challenge but there were some miracles that's not the hard part mitigation that's where we're at today if there's one thing I can get people today to do after listening to this talk would be to think about ways if you can detect cognitive state what would you do what are the mitigations you know so let's say I'm looking at a map I'm overwhelmed with information there's critical things I have to do well the system might be have the gray things out you know there's maybe the way it temporarily sequences stuff those are mitigations and we need those and we need a lot of research but I want you to understand the technology exists today to do a lot of this for my construct and there's many constructs you could create I wanted to build those simple gauges a real simple gauge high medium low for let's let's say these bottlenecks there's a lot of folks who believe kind of from a functional standpoint we have bottlenecks and attention executive function working memory wouldn't you love all your computational systems your refrigerator to your iPhone to your samsung tablet they all know kind of you they have these they understand what it's getting in there you also have a challenge of how much information how much you can got to get into you at one time would it be nice okay Dylan you're doing a verbal task right now so I'm not going to interrupt you with that Pizza example I'm going to do a spatial thing and I can I can handle spatial and verbal at the same time now it's really hard to handle verbal verbal at the same time really hard to have spatial spatial at the same time so simple things like that so add a little bit longer of a clip here then I want to show of how it would actually work and it's not just about sensing i just want to remind you that it's not just about sensing we really have this time where it's also about what's going on outside and around the world and what to do this I think some of that sensing before I show you the video i want to say artificial intelligence and I'm not just talking I love all sorts of AI I come from an AI world I love swarms I love chaos cereal of ants I love poly agents but to really do a lot of this stuff I need higher level cognitive architectures artificial intelligence that I can actually go I can ask the system hey what are you doing and it can actually describe to me in a goal-oriented fashion what it's doing just talk to me and those exist today great language I'm very familiar with is sore and I think that might be one of the keys so in the video this is there's a video that you can get called augmented cognition it's the futures of augmented cognition and Alex singer put this together for me as the director and I really wanted to convey what this would look like into the future and this is his little glimpse one minute of a 30-minute DVD movie you can download here it is audio receives concurrent dual streams of information from two of our CMO teammates via the optic and auditory modalities as before each signal is received at the rear of the brain then relayed through the thalamus once meaning is established the signals travel back through the thalamus to the front of the brain where the decision-making process occurs because Claudia is physically incapable of dealing with both streams of data at the same time her stress levels begin to elevate and her performance suffers sensing these increased physiological measurements the HM and begins to help reorder the appearance of data on Claudia's monitor in this case all texts and auditory data was minimized allowing her to focus her attention on the Cosa man street riots resulting in faster decision-making and thereby increasing her overall performance so that might be exactly what it looks like remember I said mitigation you know maybe I need some half dick stuff how the computer deals with its information but the goal today was to suggest you the next brain is really augmented cognition coupled with as higher level cognitive architectures and it's here and I want to show you some metrics these are some metrics right here decade ago using some great great things we had a number of research teams prove that they could do in performance remember what we cared about was performance whatever somebody was doing working memory for that bottleneck we got up to six hundred forty two percent improvement executive function attention sensory input there's lots of published papers on this over the last ten years but I wanted to give a flavor that this exists and we've shown that it works so human-computer symbiosis is within our grasp it's here you know I word it's it's coming whether we like it or not we might get distracted with some games brain pong we might get distracted with some other applications driving a cool little car you bought from the local electronics store but really what we need is I want to be able to walk down the street and you know I'm running a little bit early I'm a little early for a very important meeting my system knows from you know five years ago my best friend recommended a great restaurant around the corner my system knows that restaurant is open that system knows that I haven't eaten it knows it may be my physiological body needs some food more importantly maybe it also knows that what I've been studying because that's how I was studying for this meeting i have that i have not had consolidation take place yet there's this actual thing that happens in your brain after you like you've taken information your brain has to digest it so wow I've digested it but it needs you need some time to consolidate Michael hey I know Dylan needs to consolidate I know Dylan's a little bit stressed out I know Dylan's hungry I remember there's a great restaurant

somebody recommended five years ago I recommended huh so mom my sister couldn't say you're running early what why don't you go over here there's nobody in the restaurant right now get down have a sandwich and a coke and we'll be all set for your meeting now I can review some stuff so it's here that truly interactive symbiosis there's a movie the future of augmented cognition Alexander singer put it together for me highly recommend it showing you the future one thing I was really proud of that I've tried to put together is a practitioners guide because a lot of times people thought well this is just like crazy futuristic stuff no you can do it now we've had high school college kids you know look at awed metacognition again what do I mean that's augmented cognition that hole closed loop system even practitioner guides out there higher level cognitive architectures have some really good ones out there functional ones I think sore is a great one if we can couple that type of artificial intelligence it's very goal-directed where you could actually have a nice dialogue it knows how to use that cognitive information excellent there's actually even professional conferences that are occurring in this area and I should say sometimes we use words instead of augmented cognition you might hear neural organ aam Ock's you might hear operational neuroscience you know any of that you might hear brain this yes there is not a common term that people have really adopted I like augmented cognition because really what am I trying to do is awed met and perform that cognition which I hope does performance so we have some good ones there but we can really have this really the next brain-computer human symbiosis the Augmented cognition maximize human formance optimize that system and I think we're just going to need it think about how many devices how much sensing systems are out there let's use it in our own self and that's all i had today and I wanted you to take away with it that's real augmented cognition thank you very much you you Colgate University, Hamilton.