Episode 283: The State of the Update

Meter is the easiest way for businesses to get internet, networking, and WiFi. Our full-stack approach combines hardware, software, and operations so that any company can seamlessly run on a reliable and modern network.

• Streamlined installation: We take on the complexities to make designing and deployments easy, fast, and stress-free. We manage the entire installation process, and provide ongoing maintenance and support.
• Network hardware, security & management: We design and build our own controllers, switches, and wireless access points. After the network is deployed, review your speed, usage, and security in one unified dashboard. No need to hire vendors in every location or have IT teams fiddle with manual configurations — everything is automated with our software.
• Simple pricing: Pay one monthly rate with no up-front costs for installation, configuration, or hardware.

---

James Smith:
This week’s episode of the Mac Admins podcast is brought to you by Kandji. Automation in IT is a hot topic and for good reason. Automating repetitive tasks frees you to focus your skills on more strategic projects that move the needle for your organization. Kandji, the Apple device management and security platform features over 150 pre-built automations to multiply your effectiveness and impact daily. To see how to take the repetition out of your to-do list, visit kandji.io. That’s K-A-N-D-J-I.io.

Tom Bridge:
Hello and welcome to the Mac Admins Podcast. Emily, welcome back. It’s great to see you in your office.

Emily Kausalik-Whittle:
What’s up. I’m actually not in my office. I’m still in my kitchen, but we’re really close. We’re really close. For those that don’t know, my house has been in a … I’m on, what, week 17 of a seven-to-eight-week renovation.

Tom Bridge:
I am told that is the approximate correct timeline.

Emily Kausalik-Whittle:
Yeah.

Tom Bridge:
My deepest condolences for that whole adventure.

Emily Kausalik-Whittle:
Yeah. It’s tested my patience. But anyway, how are you today, Mr. Bridge?

Tom Bridge:
I’m fantastic. Fall is here in Washington and it’s beautiful. I had a great lunch or breakfast this morning with Luis Giraldo of ScalePad formerly OG and IT Glue and things along those lines. It was fantastic to see him. And he was in town for DattoCon. Ocon where I was booth candy for a couple of days going, doing demos and stuff like that. I’d forgotten what it’s like to be on a trade show floor for 14 hours. And my knees had also forgotten that and they are reminding me that that is a bad plan. But I think that, yeah, things are starting to come back again. I think that we’re starting to see events happen more often.

Tom Bridge:
We’re here, what, about two weeks ahead of Jaynac at this point. And so, maybe is life starting to return a little bit to normal?

Emily Kausalik-Whittle:
A little bit.

Tom Bridge:
We’ll see.

Emily Kausalik-Whittle:
And Dr. Schul’s insert of stock going up as people are like, I need to be on my feet and I need to be comfortable. I am ordering a few new sole inserts before Jaynac because I assume I will be on my feet a lot there. So, yeah, starting to feel like real life is coming back.

Tom Bridge:
Yeah. But we’ve got a great guest this week. Kevin White, welcome back to the podcast. It’s so fantastic to see you again. How are you?

Kevin M. White:
I am doing well. Thanks for having me on my third, I think, this is my third. If you guys keep tracking …

Emily Kausalik-Whittle:
I think that sounds right.

Kevin M. White:
Yeah. So the first time we talked about something I used to do and the second time we talked about something I had nothing to do with. So I thought this time I’d actually show up and talk about something I did do something about. So, that’s exciting. Right?

Tom Bridge:
And we’re here to talk a little bit about software updates. We know that there might be the trickiest part of being a Mac Admin right now and there’s lots of different approaches and a whole bunch of edge and corner cases. So we wanted to talk to somebody who had a huge amount of experience working with that software update binary. And so, welcome back to the podcast.

Kevin M. White:
Right.

Tom Bridge:
I think it’s important for admins to understand something. So, how does software update work at the command line in Big Sur later? And maybe how has that changed recently?

Kevin M. White:
Well, I think we could even take a higher level than that. So, with Apple, it’s interesting because they do so much work to obviously improve the operating systems from a security perspective. And they’ve done some really cool things, how it’s all signed as one big blob that you can’t break and that’s really cool. But then that ends up making it hard from a network perspective because now we have to talk to all these Apple servers. And so, it’s kind of like when Apple implements something new, it’s great that they’re trying to make improvements. But then, from an enterprise standpoint it usually makes things more difficult.

Kevin M. White:
And so, that’s one example is that from a networking perspective, we can’t just throw a package on the Mac and update it anymore. The Mac has to talk to Apple. And so then, that means that our workflows are all now based on, again, we can’t just push a package. We basically have to tell the client, the endpoint, to update itself. And for a long, long time, and it’s still there, there’s a binary on macOS 10 called software update. And that’s what a lot of third party management tools used to enforce that update locally.

Kevin M. White:
And even if things were working perfectly, there’s still a lot of limitations there. For example, on Apple silicon, and this would’ve been a big roadblock for a lot of deployment mechanisms and management tools. You have to authenticate that in that upgrade or that update outside of pseudo, outside of root. Route is not enough. It’s kind of like an iPhone if you think about it, you can tell an iPhone to update, but it’s not going to update until you provide the pin code to unlock the drive.

Kevin M. White:
Well, Apple silicon Macs are just like this too now. So we now have this new requirement for Apple silicon Macs. Even if we have awesome automation and management tools, somebody’s got to authenticate that locally on the computer or something, in the case of a management tool that actually knows how to do the right thing.

Tom Bridge:
Yeah. Using the bootstrap token or something equivalent.

Kevin M. White:
Yeah, exactly. And so there’s that complication thrown into the mix. And again, even if we had a good solution for that, Apple’s idea of what an enterprise institution needs, as far as managing those updates, the built-in mechanisms really just don’t have the feature set that some institutions require when it comes to having deadlines. And then also going along with a deadline, you’re forcing the user to do something they probably don’t want to do, which is to restart.

Kevin M. White:
So then, a lot of these mechanisms that exist on other platforms have these deferral mechanisms where we can have these rules about, I want them to be able to defer for so many days and this, that, and the other. And macOS 10 has very little controls for that. We basically have the built-in software updater and we can push down a profile to turn that on or off, depending on our needs. And then have, they call it a deferral mechanism, but it’s really an ignore the update mechanism where you can push down in the restrictions payload, ignoring the update for up to, I think, 90 days.

Kevin M. White:
But that really just makes the Mac pretend like there is no update. The Mac is kind of unaware that an update even exists. So there’s no messaging to the end user that, actually, I think if you go into system settings or preferences, it does say that updates are being managed. And then finally …

Tom Bridge:
It says that you have the latest update approved by your administrator, which I think is really fascinating language in that case, especially when software update maybe loses its mind a little bit.

Kevin M. White:
And this is not Apple’s choice. This is your administrator’s choice. Apple is not doing this to you.

Kevin M. White:
And then finally, we did recently get with the MDM push commands. If your MDM can even do those, the option to add another up to seven days after that. Honestly, I don’t use that feature because of the thing we’re going to talk about later, but that’s it, right? That’s all Apple really gives you. And then at the end of it all they do, we do have a nice new push command where we can send down an MDM push. Again, if your MDM supports it.

Kevin M. White:
Unfortunately when that even works, it just sort of restarts the Mac on the end user and then it doesn’t tell them anything. It’s kind of like, screw you, we’re updating now. Thanks, bye. And so even …

Tom Bridge:
Yeah, I hope you didn’t need that work you were doing.

Kevin M. White:
Yeah. I hope you didn’t need that. And so even network issues aside, flakiness of the software update engine aside, Apple literally just doesn’t give us good tools for managing these updates compared to what most enterprises are used to for things for Windows or even Unix systems where you can have scheduled down times and all this other kind of stuff.

Kevin M. White:
And so, yeah, updates are a pain even when Apple wants us to do them all the time, they’re a pain. And so, that’s what I’ve been working on and dealing with in my world.

Emily Kausalik-Whittle:
Bring back combo updates. Gosh, RIP.

Tom Bridge:
Yeah. RIP to the combo updates. I feel like that’s a very true, true statement. I think if we start to think a little bit about software update and how it’s changed over the years, it used to be a pretty reliable command line binary. Hey, what do you got? Can you do some things for me? Do those things at the end. In recent years, that has not been the case as much. We’ve certainly seen cases where it’s like, hey, I should have an update, but I don’t. And software update needs to be launch CTL killed by the system in order to see an update. Or it needs to be killed and then restarted, the whole operating system restarted at that point before it can see the updates that are due for it.

Tom Bridge:
It feels like Apple’s current tool for software update mechanisms is maybe not up to the job.

Kevin M. White:
I fully agree. Even the functions it used to have weren’t enough. And then, now that we’ve gotten into later OSs, there are, for instance, macOS 11, on macOS 11 on Apple silicon, you couldn’t give it a password. So, even if you were running it locally via root, you had to use an expect command line tool to automate that process. And then they added it to iOS 12 or -macOS 12, where we could hand in an authentication account so that we could write some sort of script to automate away that software update process.

Kevin M. White:
And then Apple broke the software update process if you were running it via an agent. And so, that’s a fun thing I discovered is that, and actually it all boils down to the notification system. What happens is on macOS 12 when software update starts the update process, it tries to send a notification to the end user that something’s happening.

Kevin M. White:
And if that’s run via a root system service that crashes and hangs software update, because the notification system is like, there’s no notifications for root. What are you talking about? And so …

Emily Kausalik-Whittle:
No.

Kevin M. White:
And so, for macOS 12, there is a lot of workarounds you have to hand hold around that. And supposedly though I was talking to someone, they were getting a little bit deep in the weed. But supposedly there’s some changes in Ventura. That’s going to make that a little more reliable because it’s been a post on Apple’s dev forms for this for months now that, yeah, if software updates called via an agent instead of via the user, it will hang trying to download and prepare. And that’s fun.

Kevin M. White:
And so, what we’re all getting at here, what we’re beating around the bush to is, so I’ve released a product or project called Super that tries to get around all these limitations as best it can and provide actual usable workflow for enterprise customers who need A, more flexibility with their update workflow, where they can have all kinds of fancy deferrals and deadline mechanisms.

Kevin M. White:
And then also, let’s babysit the software update process if we have to restart it, we have to try again, we have to do all this kind of silly stuff to make it behave properly. And then lastly, on Apple silicon Macs, let’s automate some of that authentication so that the user literally doesn’t have to put their password in to actually get those Apple silicon systems to run the updates. And so it tries to do all these things all in one as … It is ultimately a script, but it does has a launch agent component that installs, and it also has a notification component based on the great work by the folks over at IBM that uses IBM notifier for its dialogues.

Kevin M. White:
And again, just back to Apple, Apple has the notifications system, but guess what? They can ignore it. Yes. Right? It’s like, you’re going to focus, you’re like, there is no notifications, right?

Emily Kausalik-Whittle:
A do not disturb, but you don’t see anything.

Tom Bridge:
Yeah, that’s right.

Kevin M. White:
You don’t see them ever. And then even, I would argue that a fair amount of … You got a 40 inch screen, you’re not going to see that notification even if it’s working.

Tom Bridge:
Yeah.

Emily Kausalik-Whittle:
Or what we run into is people that have multiple monitors and maybe the notification pops up on one that they don’t look at regularly or whatever.

Kevin M. White:
Yeah.

Emily Kausalik-Whittle:
Can you just speak a little bit too, I know you talked a bit high level about what your project is aiming to do, right. Can you selfishly from an enterprise perspective or a government or education organization perspective talk specifically on what challenges Super is looking to address very specifically?

Kevin M. White:
So it’s funny, Super, or actually the full name is Superman, which is an acronym for Software Update Policy Enforcement With Recursive Messaging And Notifications.

Emily Kausalik-Whittle:
Wow.

Tom Bridge:
Which I love, and that is genius.

Kevin M. White:
You’re welcome.

Emily Kausalik-Whittle:
That’s a beautiful thing.

Kevin M. White:
It wasn’t just me. Let me just put you this way. The customer that was the primary genesis of this also really loves big projects with acronyms. So we’re like, well, we got to make to …

Emily Kausalik-Whittle:
You gave him a great one.

Kevin M. White:
We got to make an acronym of this thing. I mean, come on. And then obviously Superman is too long to type into the command line. So the script is just literally Super, S-U-P-E-R, which is also just super fun to type into the command line, because it’s like pseudo super update the computer. And then, can also refer to it in the first person, Super does this or the Super config profile or the super jam policy. So it’s kind of fun. So yeah, all around win on the name, but basically it is just fundamentally … So everybody understands, it is just a big, long bash script that will install, like I said, IBM notifier. And it will also potentially spin up a background agent so that it can pop up on the regular for various things.

Kevin M. White:
But the idea is that it’s obviously controlling software update the command line tool. It also has the ability to update, and this is limited right now to Jamf Pro because Jamf Pro is the only API I know of that you can send the MDM based software update commands. It will control that mechanism too. And the result of that is you have a admin controllable command line tool that you can basically say, okay, listen, I want you to download the update, get it ready. And then, based on this deadline or deferral criteria, I want you to bother the user until it gets done.

Tom Bridge:
Okay. And what happens when they exceed the deadline? Because we all know that there’s one of them out there.

Emily Kausalik-Whittle:
Just one.

Tom Bridge:
They’re going to find the way to, well, I mean 1%, right? I feel like it’s 1%. I feel like if you’ve got a 72 hour update period, 85% are going to upgrade in the first 48 hours of that. The other 10% minus this rando group are going to update within that last 24 hours. And then, you’ve got the scoff laws, I like to call them who just refuse.

Kevin M. White:
Yeah, absolutely.

Tom Bridge:
Such time as they’re forced.

Kevin M. White:
Yeah, absolutely. So what happens when you start on one of these projects is a little thing called feature creep and then you end up having, oh yeah, I have nine types of deferral deadlines. So, let me try to boil it down for you. Okay. So there’s three types of deadlines. The focus deadline is when Super stops paying attention to your focus mode. So, that’s kind of cool. We can basically say, hey, after X, I don’t care if they’re in focus or do not disturb anymore, go ahead and start bothering them to do this job. So that’s one kind of deadline.

Kevin M. White:
And by the way, all these deadlines I’m going to talk about, you can mix and match and make your life crazy if you want to. But the idea is I wanted to give you the flexibility to do whatever. So focus deadlines, right, when do I stop? When does Super stop paying attention to the fact that the user doesn’t want to be bothered or I don’t know, they’re in a Zoom, right? Recording a podcast.

Kevin M. White:
Second type of deadline is a soft deadline. And what that means is at that point, you’re going to give them one more dialogue box where they have to click okay. And there’s not like a defer, not till later they have to do it. They have to click okay and then it goes. And then, what I call the hard deadline, which is really just a slight variation on that, where I don’t even give them a dialogue to hit okay, it just goes and it just says notification, your Mac is about to restart.

Kevin M. White:
So those are the three types of deadlines. And then from a timing perspective, we have three kind of time horizons on top of that. So, we have just the count. What I mean by that is, let’s say you say, I only want them to defer a total number of 10 times. That’s a count based deadline. So you can say, okay, after 10 of them going, no later, no later, no later they don’t get it anymore. That’s a count deadline.

Kevin M. White:
And then we also have a days deadline. So days just literally means the number of days since we found out we got to update. So right. Super runs. It figures out, oh, there’s a restart. I got to do. That’s day zero. How many days after day zero. And this is kind of like Apple’s built in thing sort of right where it’s like X number of days passed. And then finally, there’s for the kind of hardcore it’s like on this day, at the end of the month, it doesn’t matter, it has to happen.

Kevin M. White:
And what’s cool about super and what I spend a lot of time actually working on is you can mix and match. So you can say, okay, after two days, the focus is done and we have to actually start showing dialogues. And then that they can defer five times. But then that has a deadline. So you could actually mix and match different types and that was a lot of code. And unfortunately it did make the product a little complex because basically you can keep throwing options at Super and have this gigantic string of keys. And by the way, Super does also support a config profile. So if you wanted to throw all that junk in a config profile, you can do that too deployed via your MDM.

Kevin M. White:
And so it’s funny, it’s got so many choices. A lot of people got really confused by all the choices. And so, for those of you who are new to Super start with no choices, then add them. You don’t have to add, you don’t have to turn on every bell and whistle when you start with the product. Just start with nothing. In fact, it has a test mode. You can play with it in test mode where it doesn’t actually do any of the updates. It just pretends it throws dialogue boxes up and all that good stuff.

Kevin M. White:
And that’s when you can try your deadlines. You can say, okay, what happens if I have a deferral of two? Okay, cool. Do it again and again. And so, you can play with all that in the settings. Again, if you’re new to Super, I cannot encourage this enough, read the Wiki. I spend a lot of time writing that Wiki. I’m going to probably spend many more hours editing that Wiki. And there’s even literally …

Tom Bridge:
It’s a good Wiki.

Emily Kausalik-Whittle:
And there’s a getting started section. I mean, it literally says getting started, do this. Amazing how many people don’t look at that. I need help getting started. Well, have you read the getting started Wiki?

Tom Bridge:
I was going to say, part of me says, you put that right on the homepage. Don’t bother, put in the interaction section and just say, if you want to get started, get started.

Kevin M. White:
Yeah. Get started. So, basically even outside of just Apple’s software update mechanism magic before that happens, it gives you a lot of flexibility into enforcing when it does actually happen, which is it’s cool. I wish other products could do this, but they don’t. So here we are.

Tom Bridge:
Yeah. Deploying, managing and protecting apple devices at work shouldn’t be difficult to require several solutions. Mosyle is the only Apple unified platform for business by combining enhanced device management, endpoint security, internet privacy and security, single sign on and enhanced and apps management into a single Apple only platform. Businesses can now easily and automatically deploy, manage and protect their Apple devices with one solution and at an affordable price with a solution for every business size and the best support in the market.

Tom Bridge:
Request your free account today and see firsthand why Mosyle is more than an Apple MDM. mosyle is everything you need to work with Apple. To learn more, visit business.mosyle.com. That’s business.M-O-S-Y-L-E.com.

Tom Bridge:
You mentioned that this will work with and without MDM on the backside. It’s intended for use in both situations.

Kevin M. White:
Yeah. So obviously Intel Max don’t have this new requirement we talked about. So. Super doesn’t need anything else except to run as root. So even if you don’t have an MDM, let’s just say that you’re doing something where you make custom packages through something like Monkey or whatever, as long as you just run Superman from a temp directory. So in other words, you could make a package that just throws Super in a temp directory, literally like slash temp. And then, the post fight script that just calls super, it will install itself in all its dependencies and start going.

Kevin M. White:
And again, on Intel, we don’t need any additional authentication because as far as I can tell with Ventura and I have been doing something like that does not need any additional authentication outside of root access. So, that’s good to go. It’s when you get to Apple Silicon, you have to start making choices about how to handle that authentication. And Super does offer two local authentication mechanisms for running the update without an MDM.

Kevin M. White:
And then, the third method is via MDM. But right now, again, that’s Jamf Pro only. And so, the two mechanisms for local authentication with Super. And again, I encourage you to read the Wiki, because these things have their own sort of fun things going on. Basically with macOS and Apple silicon, someone who is a volume owner needs to authenticate the upgrade or the update, right? So what does it mean to be a volume owner? It basically means you can unlock the drive.

Tom Bridge:
Yup.

Kevin M. White:
Interestingly, you don’t have to have login rights or even admin rights to do this.

Tom Bridge:
Correct.

Kevin M. White:
So I’ve done one, well, one mode is you give Super, you literally tell Super what your local admin is. Your regular, let’s say you’re in a lab environment where you have your standard lab admin on all the computers. You literally give that username and password to Super when you install it, right? I save it to the key chain, because where else are you going to save a secret like this, right? And then Super will use that to authenticate the update mechanism, right? But for people who don’t like that because they don’t want to have the admin account hanging out in the system keychain, for a workflow where you give me an admin, but then I use that admin to create a service account that then will update locally.

Kevin M. White:
That’s service account. It is saved to the key chain, because again, that’s where you got to save secrets. But the cool thing about that account is it can update Super, but you can’t log in with it. So even if somebody figures out that that account is there and can see it, they can’t log in with it because it has no login credentials. It has no home folder or anything like that. The caveat to that is, and this is true for both account types, if your Mac is file vaulted, you will see these accounts at the file vault unlock screen.

Tom Bridge:
Correct. Yep. That may cause questions of their very own.

Kevin M. White:
Feedback@apple.com. How do I update a Mac? How do I update Apple silicon without having file vault keys? There’s no way. And the answer to that is the bootstrap token. And guess what happens with the Jamf API? It uses the bootstrap token to send the MDM push to tell the computer to update. And then in that case, Super does actually need to cash credentials because what it does when it’s in the MDM mode is it still does all the things that can do locally via software update, like check for the update. It’ll actually pre-install restart updates completely silently. And this is on all platforms at all times.

Kevin M. White:
But when it comes time to push down the download preparation and install of the update, it does that through Jamf Pro’s MDM command. So it literally goes into Jamf Pro and presses that button in your Jamf Pro inventory that says, download the update, literally presses that button. And then it watches the update download, it watches it progress.

Kevin M. White:
And once it knows it’s ready, it then starts bothering the user. Hey, it’s time. Let’s do it. And then when the user clicks go, it actually sends another MDM command to Jamf to say, okay, install that thing. And it does that again over MDM. But that does mean that it is caching a Jamf API user locally to the computer because the computer has to know literally when does the user click, okay. And that is in the key chain because where else are you going to store the secret?

Kevin M. White:
And I do document, hey, how do you make this account as safe as possible? So there is extensive documentation on how to limit that account. So all it can do is literally press that button, so, yeah.

Tom Bridge:
Awesome. And so, what will end users see as they go through a version of this? And so essentially in the default settings, what happens?

Kevin M. White:
And of course I’m just pointing out the Wiki because podcasts aren’t exactly a visual medium. So, if you would like to go look to see, just look at what they look like, go to the Wiki and then you can look at the different deferment options. And then basically the whole section that says deferment deadlines, you can see what each kind of dialogue looks like. But I always try to keep the dialogue as minimalist as possible because I find the more words you give to the user in a dialogue, the less they are to read it, right?

Emily Kausalik-Whittle:
Yup.

Kevin M. White:
And so, the default dialogue is basically you got to restart. If there are any deadlines or deferral options that says, can defer three times or there’s a deadline on this date and in two buttons restart or defer. And that’s it.

Kevin M. White:
And then obviously, as I mentioned before, there’s a soft deadline option. And that one is basically, it just says, you got to restart. You’re out of deferments for whatever reason date or whatever restart now. And again, it’s important to know that those dialogues don’t appear until we know that any non-restart installs are done. So that again, kind of backing up in the workflow, Super automatically installs anything that doesn’t require a restart before anything happens to the end user.

Kevin M. White:
So, what is it like, command line tools, xo command line tools, Safari, anything that doesn’t need to restart the computer, it’s just going to install that in the background. And the cool thing is, even on Apple silicon, you can do that without authentication so that doesn’t even need authentication on Apple silicon either. Then once it’s done installing, I guess non-res restart updates is the best way to call it that, it triggers the event to download the macOS update and on modern Macs, it actually prepares it.

Kevin M. White:
So if you’ve ever watched the process in detail on modern Macs, the download is only a third of the time. Then, it spends two thirds of the time total. So if it takes 30 minutes, it’s like a 10 minute download. And then for 20 minutes, it preps that installation. And the idea is that when the computer does restart, it’s much faster because it’s done most of the work beforehand.

Kevin M. White:
So again, Super watches that process. And then once that process is done, that’s when Super starts bothering the user. Because the whole point is, we only want to tell the user when we need to restart, when we actually freaking need to restart, not restart and download. It’s like, well let’s restart now. And that’s when all your deferral logic kicks in. That’s when it’s like, are we going to let them sit there for three days in this state? What are we going to do?

Kevin M. White:
And then obviously, if they click restart, we trigger the restart right then and there. On an Apple silicon system, they get a dialogue box, honestly, for 10 seconds that says, cool, thanks for that. We’re restarting this computer. Because at that point, basically what happens is when the user clicks restart and it’s local, I tell the software update process, okay, cool. Restart the computer now that you’ve done all the downloading and preparing. And at that point it’s how fast does your computer restart, right?

Kevin M. White:
So, I do throw a dialogue up, but they may only see that dialogue for about five seconds before that computer’s like, cool. I’m out. Let’s go. And so that’s one of the downsides is that if you do all this work beforehand to reduce the amount of time the user has to wait on modern Macs, by the time the user clicks restart, it goes. But the good news is two or three minutes later on an Apple silicon Mac it’s up and running again, which is really cool.

Kevin M. White:
Now throwing a monkey wrench in real quick, those dialogues take a little bit longer on the Jamf Pro API because that MDM push down the flow for the MDM update command is not a one way push. There’s actually a communication that goes on between the client and the MDM. And there’s about a five minute delay before anything interesting happens. And so, you get a slightly different dialogue that basically when the user says, restart, you get a dialogue that basically says, the computer’s going to reboot in five minutes. Because at least with Jamf Pro, maybe other MDMs will handle this differently, but with Jamf Pro there’s sort of a mandatory five minutes.

Kevin M. White:
And again, this has nothing to do with your network not working your push being weird today. Even if everything’s working perfectly, it takes about five minutes because well we’re on a nerdy podcast. Here’s what happens. When you send that MDM command, okay. So you send that MDM command to a Mac. Actually, it doesn’t matter which MDM it is, anyone that supports this. It says, hey computer, go check for updates. I’ll be back in a couple minutes. And then …

Emily Kausalik-Whittle:
See how you did …

Kevin M. White:
See how you did. Exactly.

Emily Kausalik-Whittle:
Yeah.

Kevin M. White:
And I do believe that that timer is MDM vendor choice. They can choose well, how long is this going to take? 20 minutes? Who knows? Jamf happens to take five. It says, okay, cool. I’m going to check back in five minutes and I’m going to say, did you get any updates? And then, if the computer says, yes I did. Then Jamf goes, cool. Install them now.

Tom Bridge:
Yep.

Kevin M. White:
I don’t know if you can. And this is just now we’re talking way outside the box. If third party MDMs, although I don’t think you can do this, just send the command to go without asking. I don’t think you can do that. I think with Apple, you have to say, please go check. Do you have anything? Yeah. You got to …

Tom Bridge:
Yeah. You got a list of available OS updates, because you’re supposed to send the list OS. But it’s not a hundred percent clear to me how recent the results of that have to be. My understanding is that needs to happen. It probably shouldn’t be more than 24 hours old. And so for example, if you’ve got a check in that happens once a day where you’re getting a list of OS updates available to that device, you probably can use that to execute from the spot.

Tom Bridge:
And so essentially, that could be your trigger point or at least that’s how I read the specification. I’m guaranteeing you, someone out there knows more than me.

Kevin M. White:
Well, if you use Jamf Pro and you look at your Super logs, then you’ll see that five minute timer in there. And it’s assuming your network is happy today and MDM pushes are working right, it is literally just five minutes and then magic happens. That’s actually fairly reliable. In fact, it’s kind of interesting in all my testing, the most reliable method to update Apple silicon computers is with the MDM push it.

Kevin M. White:
Ultimately, the OS is being updated by the same mechanism. But what starts it is different with the MDM flow. And that tends to be a lot more reliable in my testing than the local flow, unfortunately. It just is what it is. Maybe it’ll get better. We’ll have to see in Ventura. And again, I have literally hundreds of lines of code in Super to deal with the local software update process being a pain.

Kevin M. White:
But if you’re trying out Super for the first time, if you have Jamf Pro the best way to do it is the MDM. It wouldn’t surprise me if at some point going forward, that’s our only choice. You want to manage software updates, you got to do it over MDM. As we go through this with Apple, it seems like they’re kind of enforcing stuff via MDM. So it wouldn’t surprise me.

Tom Bridge:
Yeah, absolutely. This week’s episode of the Mac Admins Podcast is brought to you by Black Glove. Black Glove is about to be your new favorite IT partner. They provide ongoing expert support and rapid deployment services for your current, new or refreshed Apple fleets. But what they’re really providing is complete peace of mind that your technology is safe, secure, and operating at its full potential. So, no more quick and expensive calls to the geek squad or Apple support. Black Glove’s strategies and fixes are from the hands and minds of former apple engineers. So not only is the expertise of this team unmatched, but their services are affordable and easy to get started too.

Tom Bridge:
Fortune 500 companies and small budding businesses alike are working with Black Glove to ensure their Apple technology is doing exactly what they need to. Whether it’s helping manage your remote teams’ devices, transitioning your device management system, onboarding new employees or casing tagging and tracking your devices, Black Glove can handle it all.

Tom Bridge:
They’re also just really great people to work with. In fact, mention this podcast when you reach out to them and the Black Glove team will sponsor the next generation of Mac Admins through our Mac Admins foundation. You can learn more and get started at blackglove.com. That’s B-L-A-C-K-G-L-O-V-E.com. And while you’re at it, ask them why they’re called Black Glove. It’s a clever nod to how white glove services just don’t cut it for IT.

Tom Bridge:
The more we think about it, the more intricate that dance is, right? And if you are doing it via an MDM that has a bootstrap token, that’s going to be the best way because then you don’t have to prompt the user for their password. You don’t have to get their approval. You can just execute on the process or better yet design a better mousetrap. And as an MDM provider. provide a way for the user to kick that off. And then, they control the whole of the thing.

Kevin M. White:
And also, I should mention that if you are on Apple silicon and you don’t provide Super with any authentication credentials, it acts like Nudge. Yeah.

Tom Bridge:
That’s what it does. I don’t want to draw comparisons whether or not warranted, because this seems to use a couple of different mechanisms, but what’s the difference between Super and Nudge? And it really seems to come down to that MDM component.

Kevin M. White:
Well, one thing, Super always tries to install non-res restart updates. So that’s a little bit different. So that will always happen.

Tom Bridge:
How common are those updates?

Kevin M. White:
Well in all my testing on all my computers I used to test, I put them in to make sure that it works. But things that you surprise you like, oh look, the symbols app needs an update. I don’t know if you know the symbols app. It lets you look at all the symbols in the OS for picking out icons and stuff. That thing somehow gets into software updates mechanism. You can update the symbols app. Xcode command line tools. Let’s see. Safari update sneaks in there. Yeah, every once in a while, you’ll run into something like that. I mean, just recently the Safari security updates for the older OS was in there. And so that was something that can get installed silent in the background.

Kevin M. White:
Obviously if you’re running safari and that happens, it kind of sucks, but you know what? They didn’t have to restart. Then after that, it will actually try to download and prepare the update. So I know Nudge doesn’t do that either because it doesn’t have credentials. And the whole idea is, again, by the time the user gets to those dialogues and they’re being pestered by the system to update, it should restart as quickly as possible. Whereas if you don’t give Super authentication credentials or you’re using something like Nudge and the system hasn’t pre downloaded it, because you can push down the profile to pre-download.

Kevin M. White:
When you redirect the user to the system preferences to hit that button, it could take it an hour before it restarts depending on what state that software update mechanism is in. And so, yeah, the big thing is Super can do, by default, tries to do it all without having the user have to go to system preferences or in the case of Ventura software settings or system settings if I got that right.

Tom Bridge:
System settings. Yup.

Kevin M. White:
System settings.

Tom Bridge:
Well, you brought up my favorite topic right now, which is macOS Ventura. And software update or excuse me, software upgrade going from the version that we are on now, Monterey to Ventura seems to be a little bit different this year than in previous years.

Kevin M. White:
So full disclosure, and this is in the documentation, currently Super does not do upgrades. It does updates. But you can probably guess what my next big push is for the next version of macOS is to support upgrades to Ventura. Now, it’s interesting. Emily, you said bring back the combo update. Couldn’t I just download the installer? I mean, isn’t the macOS upgrade a combo update?

Emily Kausalik-Whittle:
Sure. But it’s like 13 gigs instead of two or whatever.

Tom Bridge:
Yeah.

Kevin M. White:
If only there was a command line tool that could automatically download that for you and then monitor that and then bother the user when it’s actually ready to go. So yeah, that’s something I’m definitely working on right now with Super. It’s not out yet, but my goal is to have stuff very soon here. Obviously in time for Ventura where Super will either, your dealer choice, is download the Mac app store app in the background. I’m going to leverage some existing art for that. I’m probably going to leverage erase install code. If you don’t know what erase install is. It’s another amazing GitHub project that automates the download and preparation of the upgrader.

Kevin M. White:
And then also, I’m going to look into the MDM flows, because again, the only API right now is Jamf Pro, but that doesn’t mean I have to use something that again that doesn’t preclude me from using other MDMs. It’s just right now, that’s what I’m focused on. But via Jamf Pro you can send down an MDM push and do that. Yeah.

Tom Bridge:
You can do it. All you need to do is specify the version.

Kevin M. White:
Yeah.

Tom Bridge:
I’m aware of the MDM commands. And so that’ll also let you do this with standard users. So that right now, the main difference between a minor update and a major is that major updates require admin credentials where minor updates do not.

Kevin M. White:
Which is a thing, whatever.

Tom Bridge:
Which is a whole thing. But thinking about it from the perspective of an admin, hey, I want to make sure that you’re on the most recent version of the operating system. For people that are taking away admin rights from end users, you need a way to do that. And being able to do that with the MDM commands, where from the back end of the operating system, you can essentially, an admin can go hit that switch on your behalf or a user can trigger an API to do that. Then you can keep your users from needing admin for major upgrades.

Tom Bridge:
What I was dancing around there at the beginning of our question is that there appears to be a change in macOS Ventura regarding the major upgrade process. So that if you are on macOS, I think it’s 12.4, it might be 12.3 later, you will see, or you will get an installer for macOS for Ventura that is not the standard installer application bundle, but instead is delivered in through the same software update mechanism that you might get a minor update.

Tom Bridge:
And at least as of the most recent beta, which is at the time of recording is beta seven. There are some strange behaviors on Ventura systems, or excuse me, on Monterey systems with the Ventura update that may cause if you’ve got a major upgrade policy delay through the MDM profile, you might still see the Ventura app because currently macOS sees that as a minor update because it’s part of that maze you feed.

Kevin M. White:
Wow. Okay. That’s fun.

Tom Bridge:
File feedback. I was going to say consult apple seed dash private on the Mac Admin Slack. And I believe all credit due here to Joe Chokote from Meta, who’s been doing a lot of the research associated with this. But there are a couple of doosies out there still in Ventura testing that had me a little nervous on organizations being able to properly delay the upgrade to the next major operating system in case some of your tools aren’t ready yet.

Kevin M. White:
And …

Tom Bridge:
So, yeah, that’s a little terrifying nugget for your brain right now. Sorry for everybody who’s hearing this on a Monday.

Emily Kausalik-Whittle:
And there’s me, and I’m like, we want people on it immediately, but I know that that is just not true for everybody, which makes it very tricky.

Tom Bridge:
Yeah. I think that’s absolutely right. So many different organizations are like, yeah, I support the latest version of the operating system including day one of whatever the latest operating system is. As opposed to organizations who maybe have some key tools that are, shall we say, you’ll forgive me for using this euphemism, enterprise software that maybe aren’t tested against the beta period. Or maybe they don’t even start testing until such time as the final release has been released. I’m not sure why you buy software from those people. I really am not. But that’s a whole another show topic right there.

Emily Kausalik-Whittle:
So, I have a potentially selfish question to ask you about Superman, which is, what if you want to try to update unattended in places like labs or conference room hardware that runs on Mac minis or situations like that to super help with those scenarios or not really just curious too.

Kevin M. White:
Yeah, it absolutely does. In fact, it detects if there’s no one logged in, it just goes, it just does it. It downloads the update and goes and it does that. So it always knows if there’s someone logged in because any good script that messes with the user just figures out, hey, who’s the user right now. And so, it will automatically, if no one’s logged in, so it’s lab situation, it will automatically, like when you trigger the Super command line to go, it will immediately start going into the flow. Because it knows there’s nobody to ask, I’m just going to go ahead and do it. Yeah.

Kevin M. White:
And in that case, all the deferrals and deadlines are just ignored because they’re like, well, there’s nobody here. I’m just going to freaking do the job. And the computer gets restarted. Same rules apply though. Apple silicon, you still got to give it update credentials of some kind, whether that be Jamf Pro MDM or some kind of local. I would actually imagine labs, I don’t run a lab, but I can imagine someone who does, they probably have a default admin and that might be the easier mechanism for them anyhow, because there’s already an admin here on this computer. Let’s just give Super that account because that’s what we would be logging into it with as anyhow. And then they don’t have to worry about the MDM flow kind of getting in the way.

Kevin M. White:
Because when software update is working, it is the faster flow because we don’t have to wait for all that MDM traffic to go back and forth. And I do actually have a couple of ideas for how we can make it better for lab environments as well too like actually forcing a log out. There are some other things that we could do there. And I guess if you guys are, we could start the part of the show where I talk about other things that don’t work or features we should probably add.

Tom Bridge:
Let’s start with what doesn’t work.

Kevin M. White:
What doesn’t work. One thing that is a real monkey wrench is studio display for more updates.

Tom Bridge:
Yeah. Those are fun. I’m waiting on one of those right now, because I’ve been afraid to hit the button.

Kevin M. White:
Software update has no idea that that is happening. There are some other mystery mechanism that shows up in software update in the GUI but all the things I’m aware of didn’t know like the MDM push knows nothing about it. Software command knows nothing about it. So yeah. Find beyond the Super Mac Admins If you know anything about this mystery box, that is the studio display firmware updates. And we’ll try to sort that one out, but can’t do that …

Tom Bridge:
Yeah.

Kevin M. White:
… because I don’t know if there is a way to do it without clicking that button. The other one that’s sort of fun and we’re also talking about external physical things is you got to be plugged into power to run a system update. And so, how does that get handled? And right now, Super doesn’t really do anything about that full disclosure. I am definitely looking into ways to, do you lock the screen?

Kevin M. White:
Not just a dialogue saying, can you please plug in? Because if you are a customer who has a deadline, you don’t just want, please plug in, right? You want to take over the screen black screen of death plug in now or you get nothing. Because you can only do so much to enforce a user behavior, right? Go plug this Mac in. It has to restart. Now it’s funny, even when you send down the Apple MDM command to say restart this computer, if it’s not plugged in, nope, it just doesn’t do it. And there’s no way around it except to … And the best part is of course it doesn’t tell the MDM that that’s the case. So your MDM has no idea that it’s sent the command. It’s shouting into the dark and the computer just doesn’t update because it needs to be plugged into update.

Kevin M. White:
So that’s a big one right there that we need to work on. And of course, full upgrades, yeah, upgrades are upgrades are coming. That’s the next thing on my list. But for all of those of you who are excited about that and winning, keep in mind that Super works back to 10.14, 10. 13. So just imagine testing that many versions of macOS upgrading to a new OS and how long it takes someone who’s writing a script to do that and make it reliable, right?

Kevin M. White:
Because ideally, hopefully, I mean, I’m trying to encourage all of your devices get up to Ventura regardless of age or as late as they can. And there’s another monkey wrench. It’s like, what if some of your devices can’t run Ventura, you got to accommodate for that. So honestly, upgrading Macs, when you have a bunch of them is a pain, there’s a lot of variables that you got to deal with because you can’t just blindly shoot out a command to upgrade everything and hope for the best because stuff happens, right?

Tom Bridge:
Yep. That’s for sure. So I think, well, last question before we start to wrap up here is where do you see this going? And what kind of things are you hoping for from Apple?

Kevin M. White:
I think that they’re probably, well obviously, we have MDM2 coming out eventually whenever that happens.

Tom Bridge:
Well we kind of have declared …

Kevin M. White:
Yeah. That’s what I mean. Sorry. Yeah. Declarative MDM, MDM2, whatever you want. I guess it is technically called declarative MDM. I’ll go with that. So declarative MDM, and we don’t quite know the update story there. But I would hope that even in the short term, Apple makes the software update mechanism more reliable. There’s a great quote from a movie called Anchorman, 70% of the time it works every time. And I feel like that applies to the software update mechanism, because I have customers that have hundreds of Macs and they use it and they’re like, yeah, about 70, 80% of the time it works. And sometimes it just doesn’t.

Kevin M. White:
And to be clear, there is a ton of logging in Super. I have a whole section of the Wiki about how you can look at the logs. I put the MDM demon and I put the software update agent in a logging mode where I’m spitting out the details of the logs and I’m watching it and sometimes it just stops and it just sits there.

Kevin M. White:
It’s just like, okay, I guess it’s not going to go today. And I’m probably going to have to ultimately engineer, and I’ve been trying to put this off, a timeout mechanism in Super that just goes, software updates just bailed. We’re going to stop. And we’re going to try again and in an hour because it just kind of stopped right now. And there’s nothing I can do about that because you can only hit the software update process so hard with a hammer. If it doesn’t feel like updating today, it doesn’t.

Kevin M. White:
Like Tom Hood mentioned that sometimes it doesn’t find updates. I already have a code in Super that goes, oh, we didn’t find updates really? And then I clear some stuff out and I do it again. Right? It’s like, no, yeah.

Emily Kausalik-Whittle:
It’s that kickstart. It’s that kick start command.

Tom Bridge:
Yeah.

Emily Kausalik-Whittle:
At Jamf, we just now proactively push out a kickstart ahead of updating. We’re using Nudge right now. Our Nudge push just because we ran into so many people who go well, Nudge says I need 1251, but when I go to system preferences, it says I have no updates available. So what am I supposed to do? So we try to be really proactive and we just do that.

Kevin M. White:
Actually, I also use the simulated MDM command beforehand, too, where you tell the … I wrote a script, so I don’t have to remember these details. You basically tell the MDM agent, hey, pretend like somebody just asks you what updates are available. And then I do that. I just wait for any return. And then I go tell software update to go check. So yeah, I’ve already architected a lot of it because you have to. It’s this system that’s supposed to be, it’s a fundamental part of the operating system that sometimes just doesn’t work and you just literally got to kick it around.

Kevin M. White:
So honestly, I just want software update to work reliably. And it is kind of disappointing that something that Apple expects us to do. You go to Apple, there’s only two versions of macOS, the beta and then the next beta. They want you on the latest and greatest. But the mechanism that they have to update really only is reliable locally. And even that’s like 95% of the time, right? Because sometimes you get into a network situation and it can’t find whatever registration server.

Kevin M. White:
And so, a reliability aspect would be nice. In the short term though, I’m going to keep adding code to Super to work around this stuff as best I can so that we can make it more reliable. Because honestly this is what we do and that’s why we’re here as Mac admins, if Apple’s not going to make the tool that we want, we are going to make that tool, right?

Emily Kausalik-Whittle:
Here’s a script. I’m looking at the section that it reminds me a lot of the concept of gentle parenting. It’s like, I hear that you just don’t want to check for the software updates right now. And I understand that you’re just feeling frustrated at the thought, however, we really need to do this right now. So, if you could please just try. It’s like me trying to get my toddler to brush your teeth every morning. I get that you don’t want to get ready right now. And I understand it can be frustrating, but we still have to do it. So how do we find a way for you to feel okay about brushing your teeth this morning?

Tom Bridge:
Yep.

Emily Kausalik-Whittle:
Mac, how do we make you feel okay that there’s an update available for you to install?

Tom Bridge:
I 100% empathize with treating Macs like toddlers sometimes because it feels very much like I’m trying to convince the thing to do what it’s supposed to do out of the box because that’s the responsible thing to do. But sometimes you got to coach. I mean that’s why there are so many Mac admins out there. We’re great coaches. We’re coaching the machines to do what they need to do. And Kevin, thank you for Super. This gives us a lot more. This is extra coaching material here. I feel like this gives us an extra set of hammers to apply to situations that would normally not have a good resolution except for remote access.

Kevin M. White:
Yeah. I mean ultimately it stems from, again, my frustration and my customer’s frustration that there’s not good tools for this. I mean, credit to Microsoft, they have a much more … Now you have to do it all the time on their platforms so that sucks. But they have a much more sophisticated update mechanism that has a lot more controls in it for stuff like no, really do this tonight or do it when the user logs out. It’ll bother you when you’re like, it looks like you’re going to sleep, want to upgrade? What a freaking concept, right? These kind of things just aren’t in Mac OS and I get it. It’s a consumer operating system and Apple wants to keep it simple, but …

Tom Bridge:
Yeah. It was funny when we started looking at patch management at jump cloud, what we realized was we understood the landscape pretty well for macOS. And then we went and go looked at the landscape for iOS or excuse me for Windows. And we were just boggled at all of the different options and controls. We ended up using Windows for business, Windows updates for business for this, which is really straightforward and offers a lot of really great features that I wish apple would add. We go to software update with the tools that we have, not the tools that we might wish we had at some future.

Kevin M. White:
Exactly. Exactly.

Tom Bridge:
If you’ll forgive the terrible paraphrase of Donald Rumsfeld, but yeah. Here at the Mac Admins Podcast, we want to say a special thank you to all of our Patreon backers. The following people are to be recognized for their incredible generosity. Stu Bacha, thank you. Adam Selby, thank you. Nate Walk, thank you. Michael Sai, thank you. Rick Goody, thank you. Mike Boylan, you know it, thank you. Melvin Vives, thank you. Bill Steits, thank you. Anush Dorville, thank you. Jeffrey Compton and Don Marsh, Stu McDonald, Hamlin Crusin, Adam Burg, Thank you. AJ Petrek, thank you. James Tracy, Tim Perfit of Two Canoes, thank you. Nate Sinal, Will O’Neill, Seb Nash, the folks at command control power, Stephen Weinstein, Che Swarthout, Daniel McLoughlin, Justin Holt, Bill Smith and Welden Dod. Thank you all so much.

Tom Bridge:
And remember that you can back us if you just head out to patreon.com/macadmpodcast. Thanks everybody.

Tom Bridge:
So, if folks can find this project at github.com/macjutsu/super, we’ll have all of the links for the stuff that we’ve talked about today. The Super channel on Mac Admin Slack, IBM notifier, erase install and of course the incredible Wiki that you have spent a lot of time on, Kevin. This is amazing. Thank you for making great documentation to go with your great software.

Kevin M. White:
I have a little bit more experience in documentation than I do in scripting. So at least that part of it. Although probably a third of my script is documentation so there is that because initially I didn’t have the Waki. So yeah. And also for those of you who read the documentation, look at the script. I have a very verbose scripting style. That’s also why I wrote it in Bash and not another language because Bash is the lowest common denominator. And I’m not using that in a derogatory term. That’s the language I use. Yeah. The script has a lot more in it than even the Wiki has. And especially, if you want to customize the language right now, you got to go into the script and muck around but there’s a ton of documentation on that.

Kevin M. White:
And yeah, please join us on the super slack channel, because that’s not just for support, but it’s like, oh, you wanted to do that. I feel like Apple sometimes, you want to do what with it? Oh, okay. And that’s how you find out about new features. It’s like, oh I guess I really should try to do that feature. I never even thought of that. Yeah.

Tom Bridge:
Awesome. Well, thanks so much for joining us today. Kevin Wolf said, get everybody on the links there. It’s been a great pleasure. Thanks of course to awesome sponsors, Kandji, Black Glove, Moley and Meter and of course all of the amazing Patreon backers that we have out there. So thanks everybody and we’ll see you next time.

Emily Kausalik-Whittle:
Bye.

Tom Bridge:
The Mac Admins Podcast is a production of Mac Admins Podcast, LLC. Our producer is Tom Bridge. Our sound editor and mixing engineer is James Smith. Our theme music was produced by Adam Kudiga, the first time he opened garage band. Sponsorship for the Mac Admins Podcast is provided by the macadmins.org slack, where you can join thousands of Mac Admins in a free slack instance, visit Macadmins.org, and also by Technolutionary LLC, technically, we can help. For more information about this podcast and other broadcasts like it, please visit podcast.macadmins.org. Since we’ve converted this podcast to APFS, the funny metadata joke is at the end.