I would wager most of you listening have heard of Pavlov and his dogs. Does that name ring a bell? It is a famous experiment of conditioning training or reinforced learning. But why am I talking about conditioned responses on a space show? Well, what if a similar technique could apply to robotics in space? Yeah, wild, right? Want to know more? Let's dive in. [Music] This is T-Minus Deep Space. I'm Maria Varmazis. [Music] A U.S. Naval Research Laboratory research team successfully conducted the first reinforcement learning control of a free flyer in space this past May. I spoke with NRL space roboticist Dr. Samantha Shappen and NRL computer research scientist Dr. Kenneth Stewart about this demonstration and the future of autonomous robotics in space. Hey, my name is Sam Chippen. I'm a space roboticist here at the U.S. Naval Research Laboratory. I feel like I have the coolest job here because I get paid to play with robots. We actually get to send a robot to space and test something for the first time. I'm really excited to talk about that. My background is all about space robotics. I did undergrad research into grad school, getting my PhD, focusing on how we can do in-space assembly and servicing. Basically, how can we make robots autonomously assemble large structures in space and fix things in space, basically help astronauts have all the beautiful things we want in space. Now I feel really lucky to be here at NRL and getting to do really cool experimentation with our awesome research and development group where we get to really dig in and try what is really the latest in industry, figuring out what are the coolest concepts and actually applying them to the real problems we're trying to solve. That's so cool. Ken. I'm Ken Stewart and I'm a computer scientist here at NRL. I try to make robots smarter using AI machine learning. Before I joined here, I would have never thought that I would ever be doing space robotics. When I was, you know, for jobs at the end of my PhD, I would have never thought, you know, space robots would be the thing I would be, you know, getting smarter and learning and actually getting a robot to, you know, do something cool in space, like, really cool achievement. It is. And first, I want to say congratulations to you both and to the team. Because I want you both to tell me more about this actual amazing achievement. But since I know what it is, first congrats. It is really, really, really neat. So I'm just really thrilled that I get to speak to you both about what you all have achieved. So yeah, I won't keep the audience on Tenderhooks. Let's tell me first, what did you do? What was the experiment? Yeah, so the experiment that our Ape-Ary project focused on was how could we try using this thing called reinforcement learning to control a free flyer in space for the first time. So traditionally, robots in space are kind of done the most safe way possible, you know. Space is very expensive to send things into space. There's limited access. Not everyone gets to test in space because it's expensive. And so a lot of times for robotics, they kind of use the thing that they've done before that they know is going to work. So humans will tele-operate things and basically kind of control them or send slow commands. But we want to change that to be doing it autonomously. So what we did is actually the International Space Station has some different test facilities and one of them, NASA runs, is this Astro-B robot. So it's this really cute droid free flyer kind of like, you know, something like RTD2, but it flies around. And we were able to change how it moved. So instead of having the normal controller, we could try our, you know, cutting edge algorithms and test out and see if they were going to work. And on the day, we only had about five minutes to test. So we're really happy, you know, it worked for the first time. And this is the first time we think anyone has done it. So we feel so fortunate that we got to do that test, you know, with our really small team. It's basically Ken and I and then our co-workers Roxanna and Glenn. So, you know, our Scrappy team of four were able to pull this thing off in three months and do something no one's ever done. So we're really happy. That's awesome. Yeah, Scrappy team of four doing that is quite amazing. So yeah, again, congrats. Tell me a bit more about what it was that you did. If you can get any specifics about like what the goals were, what you achieved. Sam or Ken, if anyone wants to yield that one. So basically the goal is just to demonstrate autonomy and space on some levels. So like we are looking at reinforcement learning at the time, which reinforcement learning is basically like getting a robot to learn from its environment with rewards. So like, if you have a pet that like, like a dog, right? If you want to train your dog to sit, you know, if they sit, then you give them like food as a reward. So we're kind of like doing that with robots, you know, the SGB robot did what he wanted to do, which in this case was, you know, move to a, you know, correct position and orientation. Then we would give it a reward, which we can't give a robot food, at least not yet. I was like, what's the reward for a robot? Everyone asked that. It's like our most number one question. Like if you're taking a test and you know, you get an A, you feel really good. If you get like, you know, a failing grade, you feel bad. So we're doing that with the robot. Oh, so a good job. A thumbs up to the robot. Like you did great. It's a robot. Positive reinforcement. That's like, that's great. That's like a little sticker chart for a robot. It's a works. I mean, it's, yeah. So, I mean, so yeah, go ahead, Ken. I think you're in the middle of telling me about this. Yeah. So yeah. So that's, you know, how we trained it with reinforcement learning. And I mean, we, you know, only had a very short amount of time. So we only got to like, you know, getting the SGB to move to the correct position orientation. You wanted to do like, you know, if you send a command to it, it goes to the correct place, basically. But it did all autonomously, like figure it out on its own. And that's huge. And from my recollection of many ISS experiments, often it is only a short window that people have. So being able to achieve that in a short window is quite amazing. I think for a lot of us who dream of like the Star Trek future of what we want to see in space one day, this is such a cool step towards that. So it's like, oh my gosh. So, so Ken, you started telling me a bit about reinforcement learning control because I saw that phrase in the press release and I was like, okay, what is that? So you did explain that really nicely. I'm wondering, and again, this question could be both for both of you, either of you, where else could we see that being used in space robotics or maybe even robotics in general? I'm just very curious. I mean, I've never heard that phrase before. I'm not super familiar with robotics. So that might be why. Oh yeah. Yeah. Yeah. So it's kind of really cutting edge how people are using it now. So reinforcement learning is not like a new technique, but the way we're able to do it now is on such a larger scale. So basically, we've been able to use simulators to basically highly paralyze how we're testing. So instead of having things that used to take people months to train a reinforcement learning algorithm to be able to be effective, now we can do that in minutes. We can start a training in our simulation. It'll be running over hundreds and thousands of robots. They're doing all the same tasks slightly differently. So if I'm trying to pick up a pen, I can change the mass of the pen. I can change the friction of the pen. I can change where the center of mass is. And so that would allow me, if I was training a robot arm, to vary these parameters and make it so that the simulation is as varied and odd as normal life is. So historically in robotics, some of the pitfalls are you can simulate a lot, but then the gap between simulation to real testing is kind of where things break. And so that reinforcement learning and being able to now we're simulating on these really large scales and doing it very quickly allows you to iterate and create policies for these reinforcement learning algorithms that actually are able to be deployed in the real world. So we tested this on our robot in Space, the Astro-B. But previously, both Ken and I have been testing on different platforms. So I tested on robotic arms. Ken was doing some quadruped work. But we were able to take that expertise on different robots and then apply it to this new robot. And then that's why we were able to execute it so quickly because we already had the techniques and our group at NRL had been working up this expertise. And then we got a new robot. NASA was amazing to let us use their open source code, be able to find the easiest way to ingrate. And then we were able to test it out really quickly. But we think that's also really cool because if there's another type of robot that you're interested in, a different application, a different environment, we're able to show that we can successfully model environments so that the robots can work when they need to work. Because that's super hard. Something like Space, since it's so hard to get testing time there, you want to really have that high confidence that you simulated something and it's going to work the first time. Yeah, it makes me think about, ideally in simulated environments, it's sort of like a closed set, so to speak. But of course, in Space, we can't simulate everything. So adaptation is sort of the name of the game, right? And I would imagine this really lends itself well to that. Ken, is there anything you want to add? I want to make sure that I give you an opportunity if there's anything you want to add. Oh, sure. Thanks. I guess, you know, I've talked a lot about using a simulator and simulation environment. I mean, what's kind of really neat about all these machine learning and AI technologies, including simulators, that they're really based upon the success in the video game industry. The simulator is very GPUs, graphical processing units. They're made for video games and now they're used for AI. And like these simulators are basically game engines, except now it's adapted for scientific work where we can, you know, very highly realistic physics, right? Whereas like people are trying to make their video games much more realistic. Now we can adapt that and make our simulation environments much more realistic for like, you know, training robots for all kinds of things, which is pretty cool and exciting. It really is. It really is. This is just the beginning of things. So where do you see this going? Specifically within space, but maybe even beyond that. Yeah. I mean, so my dream is, you know, I love Star Trek. I love Star Wars. How do we get the most autonomous robots that can do everything? So, you know, right now in space, you know, it's expensive to get things up there. Astronaut time is very precious. And it's also dangerous for astronauts to do certain things. Say you want to service a telescope that's in a really far away orbit. You know, that's remote. It takes a long time to get there. Can we have robots do those kind of tasks? So I kind of view it as, can we give robots the tools with autonomous methods to be able to handle all of the type of uncertainty you're going to get in space? So, you know, I dream of really, really large space structures like space telescopes or the next great space station. And how could we have robots do the assembly? Because, you know, right now in space, the International Space Station, that was assembled with, I guess, a team of robots and humans, but it was again, you know, kind of teleoperated humans at the end of a Canadarm, like attaching things together. But, you know, I dream that we could, you know, send out a team of robots and then they could execute on whatever the tasks we want are and make sure that we're keeping up our in space ecosystem and things are, you know, thriving for a really long time. But I also think, you know, these same type of, you know, desires that we want for being able to have robots complete, really complex tasks that are, you know, highly variable, changing throughout the time that they're completing it, you know, apply to any other type of application you want, you know, either on the surface, in water. So, I love space, but I think it can apply basically anywhere you want robots to do, you know, complex tasks that we don't need humans to do. We can have the robots do it. So, yeah, I think, you know, in addition to complex tasks, like the real big push right now in autonomy for machine learning and robotics is like trying to get robots to do many varied tasks or to, you know, generalize to a greater number of, like, capabilities. Because historically, like, you know, a lot of machine learning and robotics things, you know, they're trained on data to be very specifically good at one thing or like one type of task. And so, you know, it's really exciting is like trying to figure out, you know, how you can get a robot to function, you know, if it's space, like, you know, get a robot to do some task in the library and then do something different on the ground, for example, and be able to, you know, plan and figure out and like adapt, like, based on conditions or like, you know, if it's on land, like having like a robot be able to do like surgery or medical things, like on the fly and like figure out, like, you know, how to treat someone, for example, like, because, you know, there's so many variables and, you know, everyone's different. And industry right now is super focused on things like kitchen tasks. So you'll see lots of videos of robots, you know, picking up a cup, moving it somewhere else. And we're kind of trying to figure out how do we take that like wealth of work that people are doing in industry for very specific tasks that are obviously, you know, important to our daily lives, but don't exactly solve the same type of problems that we care about here at the Naval Research Laboratory. So we're very much, you know, how do we make sure we're not, you know, we're including whatever is the current state of the art, but we're finding the ways to adapt it to the use cases that we have, which have a lot less data for them. So, you know, there's not as many people trying to solve the exact same problems we are. So how do we adapt things that have a ton of data that everyone's, you know, kind of, you know, universities and industry are on the same team working on solving those kind of, you know, household problems? And how do we apply those same type of techniques to these very specific problems? We'll be right back. I'm wondering again, as somebody who's really not familiar with robotics aside from whatever I hear in pop culture and maybe a few headlines, I think it might be also important to delineate what's possible versus what realistically is not super possible within the next five, ten years. Like what is more sort of like long term, a long term goal? Because I think sometimes there's a hype cycle, at least I know in the space industry where people go, "I don't know how realistic that is sooner rather than later." So like what's realistic within the next five to ten years versus way longer, especially within the realm of space? Like I'm thinking ISAM type stuff. Yeah. So I think the funny thing is for space specifically, it's not that we couldn't do a lot of this stuff right now. It's that people don't want to take that extra risk of trying it a new way. And so I think it's more of a culture of not wanting to take risks, doing stuff the same way we have done it. And that's kind of what's slowing the progress. In addition to things like, it's kind of funny, you see robotics on the ground, you see all the amazing stuff we can do. And then most people don't realize that actually the things you have at your disposal in space are a lot different. So there's not as much processing usually or compute. And that's again because of trying to do risk aversion, so flying hardware that's already been flown. So actually as an example, on Mars we have the rovers and their processing is actually much simpler. The highest processing on Mars right now is from the quadcopter that they flew, ingenuity. And that's because it wasn't the critical path. It was kind of an extra test and they flew a Snapdragon processor. And so that's like they ended up using that for some other tests because it was there. And so that's what we're trying to do is how do we get these smaller tests that aren't as big of a deal. People aren't worried it's an extra test. Like with the AstroBe it was a scientific platform. They had it there for people to test. And so that's why they were fine with us testing this algorithm that someone else might not be okay with us testing because they don't want it to end whatever their major testing cycle is. So we're trying to find opportunities where we kind of buy down the risk by saying, "Hey, we've tested this smaller component. It's not everything. Our AstroBe test did a very specific goal of moving from A to B." And that's just one aspect of space robotics. But how can we show people autonomy is not scary. We can show that it's going to work time and time again. We can simulate it. We can test it on the ground, on our granite tables, and then we can show it working in space. So we're really trying to prove to people that our algorithms are robust and can work when they really need it to. That's fascinating. Anything to add to that, Ken? Yeah, I mean, like Sam was saying, you know, first space specifically partly is an economic problem, but we've been talking to people like at NASA, for example, and it seems like people are more open, especially I think with the iBoom too, like putting more compute in space such as, you know, PPUs and things like that. So I think like, because, you know, people are much more accepting of it and because these algorithms are being so rapidly developed, like, I could see like space having a lot more autonomy, like maybe some, you know, basic autonomous, you know, ISAM demonstrations of like, you know, robot arm, like doing some very basic assembly, like maybe we, you know, see robots being able to talk with each other, like to a station autonomously without like someone having to control it manually, or like, you know, having like observation satellites that can, you know, autonomously take data or like take data at specific times or they're always recording data, things like that. Yeah, I hope that like in the next five to ten years, it does shift from not just using autonomy when you must, like you simply have to and more of a shift to using it instead of having humans more in the loop as we can, you know, prove that it's, you know, something you can rely on. That is a fascinating distinction, honestly. And I anecdotally, I'm just wondering, this is just a comment, not a question. As I know, I've had a lot of conversations with people about edge computing in space and as that advances, I wonder also how that will be interactive with what you all are doing. So that's just more of a comment. It's going to be very interesting to watch that happen. I'm curious about the research that you all are doing and how it supports civil space as well as defense operations and what you all think about that. Yeah, I mean, I mean, our research topic is, you know, focused on like robotics and machine learning and trying to give robots more autonomy, right, make them smarter for like different applications that people are interested in. I mean, a lot of people in our lab look at like ISAM tasks and, you know, SAM is especially excited about ISAM, you know, trying to figure out how to make robots do simple things, but we're also part of the Navy. So we also do things like Naval Ship Maintenance tasks. Or one thing that we've been looking at, like, you know, a lot of people have been talking about drones. So there's people starting to get into that space as well. And what's really cool about, you know, machine learning algorithms, that like the algorithms themselves are like fairly agnostic to the task if you have the data and problems set up correctly. So like as we advance these methods, we can probably find lots of different spaces, including space to apply them to. Yeah, I think, I guess I just focus on ISAM, SAM and ISAM, but yeah. So for in-space assembly and servicing, I think the cool thing is that once we get some more of these capabilities as kind of, you know, proven out, we have industry testing, really cool things, government testing, really cool things. And we show that these kind of this catalog of ability that we have, that we can do robotic things. We can, you know, do things like refueling or do things like adding on a new payload or switching out a payload, that we can show that, you know, this is just, you know, how we think, you know, you can do space operations and keep it, you know, so that we're not sending up a satellite having it having one thing fail after 15 years and then having it completely be, you know, jettisoned into a graveyard orbit instead of being able to just, you know, fix maybe whatever that is or find a new way to use it and repurpose it and kind of make the space ecosystem, you know, last longer and have us be able to do even more awesome science and really cool things. So yeah, I hope that we get to do even more things with robots in space. Amen to that. Anything that you would like to add that you would like the audience to know anything, any reflections about this mission or if there's anything I missed that we should cover by all means? Yeah, so obviously this is, you know, really exciting opportunity to be able to actually test in space and a microgravity and light-mine ISS. We're hoping to test in, you know, space space or like, you know, outside of the space station, you know, actually like in orbit around Earth. That's what we're looking at next is applying these type of things to that domain. Yeah, because we're the DOD, we're also hoping to, you know, apply robotics to problems like, you know, search and rescue and like, battlefield medicine. Like I mentioned, Navy Shipmating, you know, there's lots of exciting applications, I think, for robots, especially in this exciting time where people are now super interested because of the success the industry has shown and, you know, things like chat GPT. So we're just hoping to keep making more and more cool things. Yeah, I just feel so lucky that we get to work with such a cool team here at NRL. Like our group is just full of these awesome roboticists coming from kind of different backgrounds and different viewpoints. And so I think it's really fortunate that we're able to, you know, apply our expertise to these really fun problems. And we're always open to new fun problems. So, you know, this was a specific example of us having access to a cool robot on the ISS and getting to see if we could control it in an interesting way. And we keep trying to find new ways that we can, you know, push what is the state of the art so that we can push the final frontier and, you know, make robots be able to do even more cool things. And yeah, so it's, I feel really lucky we got to do this and hopefully we'll have even more cool stuff to talk to you in the future. That's T-minus deep space brought to you by N2K Cyberwire. We would love to know what you think of our podcast. Your feedback ensures we deliver the insights that keep you a step ahead in the rapidly changing space industry. If you like our show, please share a rating and review in your podcast app, or you can send us an email to space@n2k.com. We're proud that N2K Cyberwire is part of the daily routine of the most influential leaders and operators in the public and private sector. From the Fortune 500 to many of the world's preeminent intelligence and law enforcement agencies, N2K helps space and cybersecurity professionals grow, learn, and stay informed. As the nexus for discovering connection, we bring you the people, the technology, and the ideas shaping the future of secure innovation. Learn how at N2K.com. N2K's senior producer is Alice Carruth. Our producer is Liz Stokes. We are mixed by Elliott Peltzman and Tre Hester, with original music by Elliott Peltzman. Our executive producer is Jennifer Iben. Peter Kilpe is our publisher, and I am your host, Maria Varmazis. Thank you for listening. We'll see you next time. [Music] [Music] (gentle music) [BLANK_AUDIO]
