Lets Retire NTLM

Hello. Who likes NTLM? Nobody. So quite a few people apparently wanna get rid of NTLM. Which version? All of them. So I'm Linda Taylor. I am a software engineer at Microsoft. I've been around for quite some time, and in the past year, I moved into authentication from Active Directory, and I was the lucky person to inherit NTLM in all its beauty, including its deprecation. So I'm here today to help you get rid of NTLM. Here's what I'm going be talking about in the next forty five minutes or so, if anyone can spot a theme of a movie. First, we're gonna talk about where NTLM is today. I'll share some information, what we see in our telemetry, how NTLM is being used, and why we are here. Then I'll share some information that we have been getting from our customers around the blockers that they're facing and the lessons they've learned. We'll do a bit of a dive into the tools that we have available today already for people to get rid of NTLM, and then we'll face some of the hardest challenges. So we'll deep dive into some of the hardest problems, you know, where the difficulties lie, and I'll end with the confronting the future face to face of what it may look like without NTLM. Alright, so let's start with why we're here and where NTLM still lives. You might not be surprised to see this, but about ten years ago, we used to get less than 20 MSRC cases that involved NTLM, so that's like one a month. And today, there's over a 100 a year, so that's like one week now. And so that's the reason we're here. There's a couple of examples I've put up here of some very public NTLM attacks that have happened in the past, but it's just accelerated and it's increasing as you can imagine. So it's really important that people start thinking about what they can do to reduce the amount of NTLM that's in your environment. So I thought I'd wake you up a little bit because everyone's been sat down for an hour. So let's do a little quiz. And I want you to say if you think this is true or false. NTLM v one is removed in server twenty twenty five. And when you add a 2025 DC, you might face some problems with your legacy apps. Who thinks that's true? One person, couple of people. No. That's actually false. It's a question we get asked quite a lot. So, NTLM v one is disabled for outgoing traffic in 2025. So, a 2025 DC won't try and use NTLM to talk to a remote server, but it won't accept. Sorry. But it will accept NTLM v one if it's coming from an app. So you can install twenty twenty five DCs and nothing will be broken in terms of NTLM. Second question, who thinks NTLM is gonna be... Or who fears NTLM might be disabled in a future Windows update? I just wanna iron that out. That is not to the plan. You can go home happy, but you have work to do, and we'll talk about that. And then number three, most organizations are using more NTLM than they think. What do you think about that? Yeah. True. Okay. Awesome. So now let's have a look at why NTLM is being used and what we can actually see from the telemetry that we get from customers out there and where the big usages are. Number one might not surprise you, local accounts. And that's obvious. Right? There's no other thing than NTLM for local accounts. The second one though might surprise you, it's actually people using IP addresses. And I'll dive into that one in a bit more detail later on and what solutions exist and stuff like that. Number three is when you can't find a DC. You might use NTLM. Number four is unknown SPN. Who knows what an SPN is? I hate all of you. Right. So an SPN or a service principal name is required for Kerberos to work. So if that SPN is not registered in AD, we will fall back to NTLM. And that might happen silently, so this is why people may not know that NTLM is being used because stuff just works. And then finally, you... This might surprise you as well. Quite a small amount is actually hard coded NTLM. You might think hard coded NTLM is our biggest problem, but actually isn't. But it's there. So from that, we can see that most usage is actually not an explicit choice of NTLM. It's that somewhere there's a missing precondition for Kerberos to work. Okay. A couple of months ago, our lovely PM, Mariam, and me put together a little poll or medium sized poll because we wanted to understand where customers were on their journey in deprecating NTLM or reducing NTLM. Has anyone here got rid of NTLM? Steve. Right. Because he's passed it on to us, on to me. So... But... Yeah. So there aren't many customers that have actually managed to completely eliminate NTLM, but people are on journeys to disable it. So what did they tell us? Well, 84 and a half percent of people are working, willing to work, or asking for help to disable NTLM, and that's pretty amazing. That's good. That means we're we're on a good path. So most of them are actively doing it. A good bunch of people wanted to do it, but they needed help. Some people need more time. This next one I don't like. Some people actually have no current plans. So that's quite worrying. Anyone here have no plans? Clearly not because you're here. Right? So you're you're in that good percentage. There were a few people that also said they were blocked, and and we're helping those people. So the blockers were usually confidence, evidence that there's gonna be a safe rollout because, obviously, no one likes to break things when they disable NTLM. So I wanted to share what themes came out of all of that feedback and some verbatims that people might relate to as well. So number one reason was the fear of the unknown. So the thing that's gonna break or stop working that they haven't thought of. Number two, people... Some people said they had lack of visibility, so they had they had issues trying to understand which systems are using NTLM instead of Kerberos, and we do have some tools for that that I'll share with you in the next section. Number three, legacy application debt. Anyone here relate to that? Probably everybody. Yeah. That's a big one, unfortunately. Number four, for some people, business impact outweighs security. Unfortunate and sad, but that is reality in the real life. So that's what they told us. And then final... The final bucket was of people who actually didn't know where to begin. They might have been overwhelmed with the amount of events that they're seeing or the amount of devices they have to deal with. So 174 customers that was, all from different industries that we spoke to, but they all had the same story. And so when we bucketize those themes, they come out as two different blockers. So they were either organizational and technical. So third party products, Kerberos complexity, or legacy apps were the technical blockers, and then outages, business risk, things like documentation needs and guidance and process is often blocking for people. Okay. Chapter three. Let's get geared up to get rid of NTLM. I'll start with what's actually been available for quite some time. Is anyone using... Has has anyone got 2008 still? And I won't be surprised. There you go. So starting with 2008, there is actually some auditing that's already available to try and help people see when NTLM is being used, and that is disabled by default, so you have to enable it in order to start seeing it. It's not amazing, which is why we've made some improvements, but it does tell you when NTLM was used. The good news is that we do have some enhanced NTLM auditing that we shipped with Windows eleven twenty four h two and in server twenty twenty five. So, this auditing supersedes the old events. It's already on by default, so you don't need to do anything at all in terms of enabling it. And the most key information about that is actually showed us why NTLM was used. So in that little purple box there, you can see a reason ID and some explanation of the reason. And then there's also a second event where we can tell... It's a warning event that tells us if there... If there's any security that was also downgraded. We do have some good documentation about this. Is anyone here using this? I hope so. And then the nice thing I wanted to tell you is that this auditing is currently in preview, and it is coming soon to 2022 and 2019. And soon, I mean, next month. So hopefully, that will be of great help to those with legacy operating systems. Okay. For the geeks in the audience, because I know this conference is technical and so am I, I just thought I'd do a little deep dive into how this auditing actually works under the covers. So we've got the l s s process and a negotiate package, is the one that is invoked. If you do authentication, then you choose negotiate. MSV one zero is where NTLM lives. That is the actual NTLM binary. And the first thing it will try is always gonna be Kerberos. And in this case or in this example, you can see we failed with k d c error s principle unknown. Who's seen that before? It's an old friend. Yeah. It's it's one of the common issues. And so when that happens, then we actually save this error code now into the LSA memory and we fall back to NTLM. And on the client side, the client does complete the authenticate message because it needs that to get all the information for the logging. It will check if auditing is enabled, and it will also pause the message and the AAV pairs inside it to then be able to log some of that information, and it resolves the process, so you'll now get the process name rather than a useless PID, and it will tell you the target and then the reason ID, which is the thing that I pointed out in the previous event. And then it will create a nice little auditing context with all of that useful information. It's important to say that this is this is the client event that I showed earlier. Okay. So I thought it'd be nice and make a little auditing event map. I'm not gonna go through all of this because there's a lot of text, but in case you wanna snap it or anything, this is an overview of every event. The light blue ones are the legacy events, and then the purple ones are the new ones, and explain what each event tells you and where it's logged. So whether that's on the client, on the server, or on the domain controller. And when you are doing this logging, you're gonna need to correlate between all three in order to understand the bigger picture. Another thing I made for you is a map of why Windows used NTLM. Again, I won't go through all of this, but this is every possible reason ID that you might see in that event log with an explanation of what it means and possible ideas of how to fix it. So hopefully, that might be a helpful hand. Okay. So now we know where NTLM lives, let's think about how we can block it. This is the one that most of you would be using if you've started. Who started blocking NTLM? Not too many people, but a few hands. Good. So currently, we have these policies under the security settings, local policies, security options. And so there's seven different settings there that you would need to set and manage depending on where you wanna block, and each one of those has a allow, deny, or deny all. And there's also an auditing setting there and some allow lists. So in the allow list, you can configure server names and such that you still wanna allow in case they don't support Kerberos. The legacy policies can be set or they can target the client. So on the client outgoing NTLM traffic, on the server incoming NTLM traffic, and then you've also got the domain controller so you can prevent it from actually saying yes to NTLM. Two different exception lists. Are you tired yet? All the different settings. And the important thing I wanted to mention is that these particular policies do have the legacy tattooing behavior. So if you set an allow list, it gets stored in the registry. And once you come around to disabling or to getting rid or clearing that allow list, make sure that you don't just set the policy to not set because that will leave it in the registry. So it's important to actually clear it first. I actually did that myself and spent some time debugging why the hell it was still using NTLM. So I learned the hard way. Alright. The next thing I wanna mention, of course, again, we've got something new and shiny. Currently in Windows Insiders is a new set of NTLM blocking policies. The difference with these ones is that it's actually everything in one place. You no longer need seven different policies. It's all in that one UI. It currently has its own NTLM node under admin templates, which means no more tattooing. Under systems, you'll see NTLM, and then you'll see this policy there. So if you were playing with insider builds, go and try it out. It has four different settings, so you can block NTLM for domain accounts there. You can block it for domain controllers, or you can block it for... Oh, there's a machine binding, which I'll go into in a little while, but it's kind of like an allow list or a restriction. You can also set the auditing, although it's enabled by default. And then there's the block all, which overrides all the other settings. And there is still an allow list, and I'll I'll go through that again in a second. So easier to manage all in one place, ability to block NTLM. Again, I like tables and maps where things are all in one place. So here's a list of the registry keys set by those settings. Their values can be between zero, one, and two for off, audit, or block, each one of those, and then a quick scope of where they apply. Okay. Sorry. What was that? Windows 2025 or across the board? So the question is Windows twenty twenty five or across the board currently future Windows insiders only for now. Okay. Going back to the policy policy engine, engine, again, being a technical person, I thought you might appreciate some information about how that actually works. The engine evaluates the new engine. This is... Evaluates the account, the caller, and the target, and then it evaluates the settings that I just showed in an order where the block all policy always overrides everything else, and the strongest result always wins. So you're probably already thinking, well, what if I've got the new policies set and then I'm gonna deploy... Sorry, the old policies set and then I'm gonna deploy new servers. What happens? So quite a simple rule of thumb is that if you set the new policies, they will override the old ones. They are mutually exclusive, so you don't have to deal with trying to figure out like what took effect. So the new one takes precedence. And I also wanted to share some examples of the enhanced allow list for those of you that wanna try this out in the insiders build. So you can just enter like a SPN there and say, I wanna allow this SPN cause I know Kerberos doesn't work for that. Or you can be even more granular and we have a tag that you can add. So you can say, for domain SSO, allow this SPN. And that way, it will only allow it for that one policy. And similarly, you can do the same for DC auth policy and for the machine binding. So you can be very granular about what you allow. But an exception on the allow list should also be tracked and be a backlog item, or maybe a high priority backlog if that's a thing. And it should always be temporary. It should have evidence why it's there. It needs to have an owner, so somebody who owns it from a technical perspective, from a business perspective, and should have as narrow scope as possible. So don't exempt an entire server. It's better to exempt a few SPNs only. It needs to have a review date because otherwise... And an expiry date because otherwise, it's gonna become an unmanaged list. Somebody will leave the company, and it will get forgotten, and there you've got a vulnerable service. So be careful with the exceptions. They're there to help people reduce NTLM usage as soon as possible, but they should go away in time. Okay. So we had auditing. We can now block NTLM. What other tools are there to help people with this mission? So beyond those, there are learn docs, there are blogs and some community guidance that we've already published, and we are also working on some improvements in that area. So watch that space. Some ideas I had, you may wanna create, for example, PowerShell scripts that will analyze the event log, get you the top 10 SPNs that are using NTLM so you can then go and address those. And something that you probably will want is a dashboard where you can watch your NTLM. Those can be done with log analytics views in Azure, for example. I know that some of our own Microsoft field engineering helps with those. So talk to your CSAM if you're interested. And when you're stuck, we have ntlm@microsoft.com. We always welcome feedback to this alias, it goes straight to people like myself, Steven, Marion, and we try to answer you as much as possible. But I'll also say that if there is an issue that you really need more than, like, a five minute help with or if it's more than a feedback, please use the CSS channels so that somebody can actually help you look into logs and help you with the issue at hand. Just wanna set expectations. We can't really do that on the ntlm at microsoft.com. Okay. So once you've got a plan, how do you roll it out safely? One of the things people were concerned with is having an outage. So here's a plan that we would highly recommend that customers use. So deploying it in rings, having ring zero be your domain controllers, your crown jewels, the really important machines, the certificate authorities, those kind of machines. Then you'd have a ring one, which is business critical servers, exchange servers, file servers, database servers, other important servers, and then your general workforce, and anything else. Now, as if rings weren't enough, what you really wanna do for every ring is also do it in phases. Obviously, first, want to discover what the NTLM usage is, remediate the problem. So if it's a missing SPN, go and add the SPN, test that it works, and then pilot block a small amount of machines, and then fully block. I skipped that very quickly, but in the end, in an ideal world, fully block. So that's the tooling and the approach. And now I wanna have a look at what we know the hardest NTLM challenges are and go over some of those buckets of telemetry and the suggested solutions for those. So you can have your usual suspects, things like the legacy apps that we talked about, third party products, appliances, hard coded NTLM, all the things that we already expect that are gonna be there. But what about the unexpected cases? So those are gonna be the tough ones, the things that out the woodwork when you enable that auditing and you start looking at it. Things that silently worked before and just used NTLM, like the IP addresses, SPN unknown, crossed forest edge cases isn't interesting. How many of you have external trusts? Quite a few. I would love to talk to you about those because those are gonna use NTLM, so we we we need to think about those. Then you've got legacy management tools that might be in the environment, again, might not be visible until you start auditing them, But now you've got the process names, and of course, the work group and the disconnected environments that we saw in the telemetry earlier, and then other scenarios that might have just worked previously. Okay. So what do you do about those? First of all, let's talk about the IP addresses. Anyone here guilty of IP address? Who wants to admit? There's a few people. Okay. So when I first started looking at this problem, I saw the telemetry and it was like a, you know, huge number of IP addresses. So I was like, okay, but why? And I wanted to know why people were using IP addresses. So again, I did what I love is a poll, and some of you answered, quite a few of you answered, and I got to learn why people are using IP addresses. A third of that was actually hard coded in applications or scripts. And for those, if it's a static IP address, there's actually already solution that you could deploy if you can't use a name instead. So you can register that IP address as an SPN AD, and on the client side, you need to set specific reg key, and Kerberos will try and now use that SPN and will stop using NTLM. So that's something that people can already do, and I think it's just not well enough known perhaps. Another third of the people said that it's because there's no DNS, so they need to type the IP address, otherwise, they can't find the target machine. That's fair enough, and there's a feature called I a curb that Steve and Marion will be talking about at 03:00, if you hopefully will come back, that will get rid of that particular dependency. And then the thing that did kind of surprise me was the other third was actually other reasons for using IP address. Things like... Some people said they found it easier to type an IP address than the name because their naming convention was so bad. Like, so those are gonna come down to education. And some people actually said they didn't realize it used NTLM, that they were lowering the security. So those are gonna be like educational work to do. Okay. Enough about IP addresses. SPN not found. Who's troubleshooted SPN not found? Probably many people because that is an issue that I myself worked on when I was in CSS many years ago. It was quite a popular Kerberos issue. So what actually happens when an SPN is missing is you're gonna see the KDC error SPN unknown. Negotiate falls back to NTLM, but nobody notices until now when you don't want NTLM anymore. And so in those particular cases, you'll see in the event log what the SPN was. If you use the new enhanced logging, it will tell you which SPN it was looking for. And then, you need to investigate the reasons of why that SPN is missing. So, some of those reasons could be it just wasn't registered, or it failed to register. The application didn't have the permissions to automatically do it, and it just never happened. I've seen that before. Other cases are duplicate SPNs or SPNs on the wrong account, where, again, Kerberos won't work for that reason, things like that. So, the fix for that looks very simple, just register the SPN, and hopefully it all starts working. If you do find cases where the solutions that we've documented are not helpful, I've put a link there. It's quite small. Please do email ntlm@microsoft.com because even though we believe we understand what the problems are from the telemetry and stuff, I'm sure there's still going to be corner cases that we don't know about. So we would love your feedback. I do want to talk about a special case of SPN unknown, and that is DFS based namespaces. In that particular case, what happens is is that you have a DFS setup and the client tries to go to slash slash domain name slash share. And it needs to talk to a domain controller for that to understand which actual end server to contact. So it'll get the TGT, and then it will try and get a TGS for... It will do a TGS request for siftsbackslashdomainname.com, like contoso.com. And you'll see that fail with s b n unknown because that s b n doesn't exist, and it will go to NTLM. Now that particular scenario, unfortunately, isn't something that you can solve by registering that SBN. So today, you can't register it, and in fact, if you try and register it, it will just make things worse, it will make it completely fail. And that's because every DC has to be able to encrypt a ticket with some kind of an identity, and then another DC must be able to decrypt it. So what you can do today is actually just exempt that SPN from your policy for now, and in the future, we are working on a solution for that so that you won't need to do that anymore. So keep an eye out for that. Hard coded NTLM. Do people here own applications, in house applications, scripts, things like that? Okay. I thought I'd share some examples. This is an exercise we've actually done extensively across the Windows code based in Microsoft is to find hard coded NTLM and eliminate it. So I'll show you some examples of what it looks like so that you can also use them to do your code searches. So there can be a number of different surfaces, so SSPI, RPC can be hard coded, DCOM, and you can simply search for the word NTLM. So the purple on your left is bad and the blue is better. So find NTLM or find the constant RPC, c auth, n win n t, and look to look to replace those with negotiate. Now, it sounds simple, it's actually not that simple because when you replace with negotiate, you must ensure that there is that SPN. Otherwise, once again, it's gonna fall back to NTLM. So there's some examples there, some rules, and a checklist. So search for the package, make sure that if you're gonna use negotiate, that there is an identity that is gonna make cover us work. Check all the different transports, RPC, SMB, HTTP, all of those. Ideally, don't have a fallback path. I know it feels nice just in case, but just use Kerberos ideally. And, of course, validation is important to see that it's working with Kerberos after the change. Okay. Just a quick mention of the... There are some other known dependencies that is... We all know are gonna take time. So things like legacy operating systems and apps are gonna take a long time to phase out. So phase those out with time. There's a particular... There's another issue to mention is that in legacy operating systems, the net log on service uses NTLM to forward SID lookups, for example. So if you have old OSs and new ones, past 2025, it can do Kerberos. But before that, unfortunately, still use NTLM. So that's something to keep in mind. And trusts, I mentioned earlier that quite a few of you are using and I would love to talk to. Alright. Ten minutes left. So to wrap this up, let's talk about what the road to zero NTLM might look like. As you already know, we didn't release that in a Windows update, so no no nasty surprises there. The main message I wanna say is that retiring NTLM is going to be a team effort. Microsoft doesn't have a magic wand. We can't reach your environments, so we're gonna have to work together where we're getting rid of it and we're doing our part, and then customers need to also do some work in their environments. So to list the challenges. So for the legacy in house apps, what you can do is find the hard coded NTLM in the apps that you own, use my examples, and switch to negotiate. On the Microsoft side, we're also working on driving NTLM hardcodes to NTLM out of Windows, so you don't have to worry about those. Third party products, you can use the new auditing to prove which process used NTLM and capture that audit evidence to help you drive those vendors if needed. IP addresses, Move to names. Please. Register the IPSPN where it makes sense. If it's a dynamic IP address, don't do that because obviously it's gonna change and it's not gonna work after however long. And then on our side, Microsoft is working on I a curve that you can learn more about at 03:00. SBN unknowns, again, you can exempt the global SBN that I mentioned for the DFS namespaces if that's a problem that you are seeing, but for the rest, find the missing SBNs, register them where possible, and see that Kerberos now works. And on our side, we're working on a solution for the DFS namespace. Other dependencies that we haven't talked about or you don't find anything documented, please send us an email to ntlm@microsoft.com. Easy to remember. One more issue I wanted to mention that quite a lot of people have reported quite often is their AD certificate services using NTLM. That is getting fixed. And if I haven't said it enough, ntlm@microsoft.com. Alright. So finally, I wanna revisit those telemetry buckets and just point out what solutions we have that are mapped to those. So local accounts and there's no DC. There's two different feature scenarios that you can learn more about at 03:00 from Steve and Marion. IP addresses, we've got try IP SPN or alternatively, configuration changes to names. Unknown SPNs register the missing SPNs, hunt them down, get rid of the NTLM, and exempt the global SPN where necessary. For the hard coded NTLM, I mean, straightforward. Sounds straightforward, but it's way harder. I'm just making it sound easy. You're gonna eventually need to remove the hard coded NTLM. We do have a roadmap that was published a little while ago by the lovely Mariam, and I've just kind of illustrated it slightly differently on the slide, but I just wanted to kind of give you a picture of where we're heading, where we've been. So about a year ago, all this enhanced auditing came about, new policies are coming. We're kind of in the middle of the journey towards a Windows free of NTLM where we're like we're right amongst the mud right now, fighting all the bad problems, finding where NTLM lives, and eliminating it. And the future, we envision and will be that eventually we'd like to ship a version of Windows with NTLM disabled. A reminder of the three principles to keep you going. So, install every update. Probably don't have to really say that, but we are shipping constantly back ports of things to help you out with the NTLM journey, features getting enabled, things like that. So keep updating. Enable pairing and validate pairing, and then expand gradually to widen scope. Anyone guess the movie yet? Yep. So your checklist, because again, I like checklists and tables, discover NTLM, prioritize by tier and by account. So prioritize the DCs, the things that matter the most, machine accounts, high... Highly valuable servers. Make the repairs, do the testing, and then finally, when you feel comfortable, block with some exceptions. And that concludes my NT elimination today. We know where NTLM is. We do have the tools to remove it, and we have a road map. So just need to work together to make as much progress as possible. Thank you very much. Just don't ask me any hard questions. Thank you. Well, you you know me. You know me, Linda. Great talk. Thanks thanks for that. In the recommendations for missing line of sight to to DC, you didn't mention KDC proxy. Is there is there a reason it's not a recommended solution? No. I actually forgot about it. So thank you very much. That's another idea. That's something that currently exists as KDC proxy. You may want to have a look at, but Steve might have some more ideas as to why. It's not on by default. Okay. Thanks. Just a quick question. Just wanna verify. If we have some ADLDS instances, is there anything we need to do for those in particular, or that just won't ADLDS? Yes. There is currently an issue where it's using NTLM. If you... I'll dig it out after the session, but it... I think it's if you use specific type of account, it may use NTLM. But it it would be the same deal, audit, and you'll see the process in the in the audit log. Any other questions? I'm happy to speak to anyone throughout the day as well. If we happen to be lucky enough to have external domain trusts, obviously, I'm gonna say obviously because I'm not sure, we wouldn't start disabling NTLM on the domain controllers. Right? Because then those proxy requests would all fail? That's right. So you'd need to put the exceptions in place for that for now. Thank you. Can't you incorporate, basically NTLM eliminator into the SentinelOne and record literally every workstation that has the agent, if they're running anything NTLM and somehow report back to the console. Right? In ways, you can already do that with auditing. Right? If if you if you have machines that are joined to Azure and you're you're sucking out the auditing or any other way that you're collecting audit logs with forwarding or something like that, you can already mine that data to to find the processes that are using NTLM. Like, earlier I mentioned, like, you could also use a PowerShell script to get you, like, the top 10 NTLM events in your machine, things like that. But sure, there's an opportunity for someone to create an NT eliminator, a terminator, Something as violent as possible to get rid of NTLM. When you talked about ADCS is in flight, can you provide any clarification or timeline? I don't have that off the top of my head, but if you catch me offline, I'll look it up for you because I don't actually know if it's already shipped or not. That's it. No more questions? Get an extra minute's break. Thank you very much.