Episode 288: Mike from Bombich Software

Meter is the easiest way for businesses to get internet, networking, and WiFi. Our full-stack approach combines hardware, software, and operations so that any company can seamlessly run on a reliable and modern network.

• Streamlined installation: We take on the complexities to make designing and deployments easy, fast, and stress-free. We manage the entire installation process, and provide ongoing maintenance and support.
• Network hardware, security & management: We design and build our own controllers, switches, and wireless access points. After the network is deployed, review your speed, usage, and security in one unified dashboard. No need to hire vendors in every location or have IT teams fiddle with manual configurations — everything is automated with our software.
• Simple pricing: Pay one monthly rate with no up-front costs for installation, configuration, or hardware.

---

James Smith:
This week’s episode of the Mac Admins Podcast is brought to you by Kandji. Automation in IT is a hot topic and for good reason. Automating repetitive tasks frees you to focus your skills on more strategic projects that move the needle for your organization. Kandji, the Apple device management and security platform features over 150 pre-built automations to multiply your effectiveness and impact daily. To see how to take the repetition out of your to-do list, visit kandji.io. That’s K-A-N-D-J-I.io.

Marcus Ransom:
Hello and welcome to the Mac Admins Podcast. I’m your host today, Marcus Ransom as Tom is sunning himself in lovely Barcelona, an objective by the sea, but I am joined by Charles Edge. How are you today, Charles?

Charles Edge:
I’m good. I’m good. It’s a beautiful day, no complaints. How about you?

Marcus Ransom:
It’s wonderful today. Spring vacation is just finished here in Australia, so everyone’s back at school for work and for classes. So I’m in the working from home equivalent of having shifted from the open plan office with the foosball table right next to me to the corner office with a lockable door or something like that. So it’s eerily quiet around here, which is nice.

Charles Edge:
Except the cat.

Marcus Ransom:
But to change things for me-

Charles Edge:
He’s staring you down.

Marcus Ransom:
Except the cat. Well, I can’t see him at the moment.

Charles Edge:
That’s that worst.

Marcus Ransom:
So that’s always a worry. Yeah. So if I suddenly disappear off the recording, everybody knows that Alfie got hungry or whatever. But to keep us entertained today, because you and I talking about weather and pets isn’t really, I’m assuming what people are tuning in for. So we’ve got a wonderful guest today. So one of the tools we don’t talk about that much on the podcast, but we’ve all used is Carbon Copy Cloner. So, before Carbon Copy Cloner we wrote scripts to clone drives. We had to figure out rsync, we had to build manual imaging workflows with firewire drives. And in general, we just had this barrier of entry to do what should have been simple tasks that we really didn’t need. So, after CCC there was a simple gooey with some fairly complicated logic underneath. Then it went far beyond the basic cloning technology to do more and do it faster.
So we should have had Mike Bombich, the author of CCC on far earlier, but we’re stoked to finally nab an interview with him. The notes say, “Insert hero worship applause here.” So, unless James can come up with something much better. So welcome to the podcast, Mike. Thanks for joining us.

Mike Bombich:
Thanks a lot. I’m glad to be here.

Marcus Ransom:
And I do think we’ve all used your software at great lengths and sometimes in some of our worst moments. And so it’s totally an honor. Before we talk about the software, when guests join us for the first time, we love to do a little origin story. So do you mind telling us what you do there and how you came to be doing that?

Mike Bombich:
Yeah, sure. It started about 21 years ago, which I mean, it seems insane that it’s two decades. I was actually working at a university and macOS 10 had just come out and it was the fear that all of the Mac religious had was that this new OS was going to come along and there was going to be no way to deploy it. And then all the ITs at the universities would just be like, “Nah, forget it. It’s too hard.” And I was convinced that it had to be possible. So I just sat down and started playing with it. And before I knew it, I figured out you could actually make copies of this system and push it out. And I actually thought about the concept of the origin story a little bit, and that really was the necessity is the mother of invention, origin of CCC. But in the last decade it’s really gone far beyond that. I worked for Apple for a while. I built a utility called that restore that I think the Mac admin mass is probably not what it’s about.

Charles Edge:
That we all use about. Yep.

Mike Bombich:
Yeah.

Marcus Ransom:
Exactly, but loved.

Mike Bombich:
Really, my passion has been saving people’s data and I can pretty much remember the exact day and just burn that into my mind. I was working at the university doing IT support and I had to work with a lot of Windows systems, not by choice, but just because there were a lot of them out there. And this lady’s hard drive had died, no, it hadn’t died. She was just having some trouble with windows. And I got her system back in the office and for whatever reason I accidentally erased the wrong disc. I erased her production disc, it had her thesis on it and she had no backup. You could just imagine the moment when you erase that disc and you just feel the chill dropping from the top of your head all the way down and you’re just ready to pass out. And that’s pretty much the moment that I realized that protecting data is really, really important. And since then I’ve pretty much dedicated my life to it.

Charles Edge:
And you got a biology masters at Bowling Green, is that correct?

Mike Bombich:
Yeah, that’s right. So I actually did my undergrad accounts at College in Michigan and then my graduate work in Aquatic Ecology in Bowling Green. And I studied zebra mussels on Lake Erie, zebra mussels and round gobies. It was cool. And that’s funny, that’s another one of my passions. I love being in the outdoors. I love to fish. Every summer it’s hard to keep working because there’s so many fish out there. But at the end of my graduate research I realize that I have far more passion about developing things. And actually one of the things that I did in my research was developing some software that would take all of these line drawings out of a picture of and it was the length of all of the zebra mussels. And I wrote software to pull all the lines out, determine the length and dump it into a spreadsheet. And at the end of that I was like, “Man, this is so awesome.” And that was really it. So I finished my graduate work and I’m like, “All right, I need to create stuff. Discovery is not my passion, it’s creation.”

Charles Edge:
And I guess, how would you describe Carbon Copy Cloner?

Mike Bombich:
It’s an advanced file copying and backup utility, and it really is both. I mean, there’s some debate about what is a clone and we definitely will at the very basic level, will make a copy of a volume, but we can also copy a folder. It can also update a backup. But then way beyond that we have advanced backup functionality where we can do incremental backups, we can create snapshots so that we have versions of your data and we have advanced verification functionality. So one of the things that we all back up our data because we assume that at some point our hardware’s going to fail, but that actual act that also applies to the backup disc. So we have functionality where we can re-verify files that were copied by the backup task and we can do an ad hoc verification of all of your files that were copied by the task.
And then we also have a backup health check where we can just reread all of the data on the source and the destination. And sometimes a file hasn’t been modified or the modification date hasn’t changed and the size hasn’t changed, but the actual data that’s there is different for one reason or another. And we can catch that scenario and fix that. So the other thing I say advanced file copying utility, one scenario we see a lot is people want to migrate their data to a larger disc. And this is not necessarily this a startup disc. Just say you’re a photographer and you want to move to a new disc. And I hear this all the time, it’s like I’m going to move to this new disc and then erase the old disc. And it used to just terrify me that people would do that because it’s like, all right, so what if the new disc just is a lemon?
So we actually added some functionality in CCC 6. It’s the re-verify files that were copied so that if you’re doing that, we copy all of the files, we flush the buffer, and then we go back and reread all of the data that was written and verify that the destination actually can deliver that data back. And then there’s lots of bells and whistles, lots of different ways to schedule things, lots of different manners to schedule it. We can have it a task run when a disc reappears, just basically a lot of flexibility. We do everything that time machine can do, but then we have a lot more flexibility both on top of that.

Marcus Ransom:
Everything time machine can do except you’re actually able to restore the data you’ve copied into the Time Machine or it may have actually backed up in the last month. Time Machine’s a wonderful tool as part of your backup arsenal.

Mike Bombich:
Yeah, and I’ll never poke fun at Time Machine, but I will always just expand on the features that I offer in addition to it. And then you can just make whatever inferences are necessary.

Marcus Ransom:
Spoken like a true former Apple SE. Multiple copies of your data are multiple approaches to back up. So you can then choose the most appropriate one to restore from. It’s the approach I like to take.

Mike Bombich:
That was actually something I learned in biology is that nature looks from multiple paths of redundancy and we have two eyes and that’s because if one fails, we still have the other and we could still lead a successful life. And it goes the same in hardware. You want hardware redundancy, but it also applies to software. It’s good to have software redundancy.

Marcus Ransom:
And also a large set of data to look at as well. I was at the museum with my youngest daughter yesterday and she’s fascinated by bugs at the moment and looking at the multiple butterflies that they would collect when they’re trying to find out, because if you just collect one butterfly that may be the corrupt copy of the file. Whereas if you’ve got multiple copies of it and that one’s actually different, maybe that’s that the butterfly with nine wings isn’t necessarily your thesis. Maybe it’s a different thesis that you could write. So we know you’ve rewritten a lot of the early stuff, but I guess if not all of it before Carbon Copy Cloner, I feel like maybe we all just used to use rsync scripts and ASR or some other options to do block or file based clones of drive. So you built the perfect gooey for that. So what was the inspiration to create the gooey the way you did?

Mike Bombich:
Boy, 20 years ago, it’s almost, it’s cringy to go back and look at that early UI. It was really a necessity’s the mother of invention approach. We need to get the data from here to here and this is what I want to copy. And it was just a very basic rudimentary UI. But over the years we’ve definitely taken an enormous amount of user feedback. And one of the biggest challenges in UI I think is trying to meet everybody’s needs and you really can’t meet everybody’s needs. So we go for a progressive disclosure approach. We try to keep it really simple. In the main UI, there’s like three choices, source, destination, schedule and then start. But beyond that, there’s a lot of people that really need a lot more advanced functionality. So we layer it.
If you need a little bit more functionality, you can click on the destination selector and there’s some options specific to the destination. If you need more than that, there’s an advanced settings and I’ve always wanted a little circuit board background to that because it’s like you’re really getting into the internals and getting to the exciting PC stuff. And for anyone who’s curious, there’s even an expert settings. If you hold on the option key and click on the help menu, there’s expert settings. And actually for our next version-

Charles Edge:
Guess what we’re looking at after this?

Mike Bombich:
Wait till you see the beta that I’m going to post like Tuesday. I’ve made some new changes to the expert settings. So you can actually see all of the expert settings that we have. Anyway, so the approach is really just try and figure out what works for people and take their feedback. And over 20 years we’ve gotten a lot of feedback and you never make everybody happy. But on the whole, I feel like we’ve kind of finally hit the nail on the head for where we need to be.

Charles Edge:
I mean, for what it’s worth, I felt like you hit it on the head and what ’04, ’05, I think a lot’s changed. And so things have to evolve. If I remember back then we were using cron jobs, so when you hit the schedule button, you were writing us a cron job into the system, if I’m not mistaken. And then later probably moving over to launchd. But I do feel like a lot of guests on the podcast write tools, but most of us Write Tools for systems admins who are still doing what you were talking about back at the old alma mater. And some I think even dream about escaping the day job to make their software a full-time gig. But I guess what was that transition like for you to go into being an independent software vendor?

Mike Bombich:
So the first room was going to work for Apple and going from being kind of an admin person doing tech support at Bowling Green, jumping off to Apple, that was, I mean, it felt like the chance of a lifetime. And then I got to Apple and realized ironically didn’t have an opportunity to be as freely creative as I could be on the outside. And I’m not going to get into all of the politics of what happened at Apple, but at the end of it, I was basically told, you’re not a software engineer here, you’re a systems engineer and you don’t develop software, you’re selling technical solutions. And I knew my passion is development, so I was like, “Okay, maybe it’s time to move on and leave. And it was a really difficult challenge. I mean, you’re basically jumping out of a perfectly good airplane and hopefully everything works out.
And it felt pretty good as we were going along because I had a decent amount of revenue coming in. It wasn’t as much as I was making from Apple, but it definitely felt like I could make it bigger and I just took a leap of faith and everything worked out really, really well. Early on, I did have to make a really important decision. I remember when Apple introduced Lion, possibly one of the worst OSs out there. I think everybody looks fondly back at Snow Leopard. That was a great OS, but then Lion came along-

Marcus Ransom:
Was it just because what came next made it seem so good?

Charles Edge:
No, it was good. It was solid.

Marcus Ransom:
It was.

Charles Edge:
The DS config AD stuff all finally worked perfectly.

Mike Bombich:
Yes.

Charles Edge:
Yeah.

Mike Bombich:
So Lion came along and that was also when I finally started supporting backing up to network volumes. And it was probably the worst possible time to add that functionality because something huge changed in network file systems right around then. But I remember I was getting 100 support tickets. It felt like 100 a day, maybe it was 100 a week, it was a lot. And I was by myself and it was still donation wear. And at that point I’m like, “I’m literally doing support 50 hours a week and I have no time for development, so I have to hire someone.” And that was when I made the transition. It was like, “All right, I’m going to start hiring people. I’m going to hire someone who knows how to hire people,” and I’m going to start charging for it because there’s basically no way to make it the solution that I want it to be to make it as successful and useful as I want it to be without adding staff and without asking for some money.
And it worked out. I mean, I was just blown away at how many people were just so positive. It’s like, “Yes, you really needed to do this and we’ll support you.” And it was successful. And it wasn’t long after that Tim Perfitt called me up and said, “Hey, so how do you leave Apple?” And I was so happy to just give him my perspective on it and so happy that he made the leap too. And he made a successful company outside of Apple. And I’m telling you, if you’ve got the passion for it, go for it because it is totally worth it.

Tom Bridge:
This week’s episode of the Mac Admins podcast is brought to you by Black Glove. Black Glove is about to be your new favorite IT partner. They provide ongoing expert support and rapid deployment services for your current new or refreshed Apple fleets. But what they’re really providing is complete peace of mind that your technology is safe, secure, and operating at its full potential. So no more quick and expensive calls to the Geek Squad or Apple support. Black gloves, strategies and fixes are from the hands and minds of former Apple engineers. So not only is the expertise of this team unmatched, but their services are affordable and easy to get started to. Fortune 500 companies and small budding businesses alike are working with Black Glove to ensure their Apple Technology is doing exactly what they need it to, whether it’s helping manage your remote team’s devices, transitioning your device management system, onboarding new employees or casing tagging and tracking your devices, Black glove can handle it all.
They’re also just really great people pull to work with, in fact, mention this podcast when you reach out to them and the Black glove team will sponsor the next generation of Mac Admins through our Mac Admins foundation. You can learn more and get started @blackglove.com. That’s B-L-A-C-K-G-L-O-VE.com. And while you’re at it, ask them why they’re called Black Glove. It’s a clever nod to how white-glove services just don’t cut it for IT.

Marcus Ransom:
What do you think it is about your approach to software that’s made it so successful? So I know as a Mac Admin CCC, much like Tim’s tools, they’re one of those tools where as soon as you mention it to anybody else technically in the industry, there’s just that look that they give you. Whether it’s like, “Yeah, that stuff works.” It’s not the software like, well, we chose to buy this one, so it’s inflicted upon us and we’re just going to make do with it and do the best we can. Whereas there’s other software and Charles is laughing and we’ve all had that sort of software that the organization decides to go with and maybe they’re trying to solve a really hard problem that there isn’t really a solution for and they’re doing their best.
But CCC was one of those bits of software, as you said, where when it was donation wear, it was like, “Well, this is awesome. I want to donate because this is saving me an awful lot of time.” And if my boss is going to ask why we actually need to pay for a license for this, I’m going to quite comfortably stand up there and say, “This is why. This is what we need.” What do you think of… Do you know what it is that your approach to software turns out that way?

Mike Bombich:
Yes.

Marcus Ransom:
Do you try to do things differently?

Mike Bombich:
Yes, and I think probably the main thing that I do differently is I take about a third of our support calls. Every single day I’m right there on the help desk, I’m coaching my primary support guy, but I’m still taking about a third of them and I don’t want to ever lose that connection to the end user. I don’t want my company to be enormous. And it’s funny, Jamf is a great comparison. I remember way back in the day when the guy doing Jamf was just a guy like me. And then, I don’t know, it was a couple years ago, whenever they went public, I’m like, “Holy crap, that’s a lot of money.”
But I’ve never wanted my business to be that big. I’ve never wanted to take on venture capital or do anything like that. I want to be able to wrap my arms around the business and really kind of not own and do everything, but to know everything that’s going on and to still be in touch with the customer because at the end of the day, people have to want to use my product for our company to be successful. And I think that we definitely get a lot of feedback around that. People are sometimes surprised that they get a response from the head guy, but they just really appreciate that it’s that well supported and that they have that level of access to the head guy.

Marcus Ransom:
What sort of things have you learned in those situations that have made you realize that you needed to change something or there was an improvement or something you hadn’t thought of as a developer by having those interactions, direct interactions with customers?

Mike Bombich:
I can actually think of one specific example. A guy sent me a video of how he was using it and he showed me a way to use it that I didn’t even know you could do it that way. Obviously, we’ve all got our in-laws and our parents and people who don’t use software as much as other people. Everybody has a different perspective coming to it. But when I saw that, I was like, “You can do that. That’s okay.” I never imagined someone using it that way. And it’s like, “All right, I need to know how people are using it, how people want to use it. And I say no a lot, probably more than I should. And then on the other side of that, you look at the expert setting and it’s like, “Yes, I say yes a lot.” Maybe there’s a setting that really kind of need to be in there, but 0.2% of people would actually use it. But I think being that close to the end user definitely gives us opportunities to take the product where it needs to go versus where me living in my bubble would take it.

Charles Edge:
I think that’s interesting as well, because sometimes you’re like, “Well, your example is, wow, I didn’t even know you could do that in a good way.” There’s also, “Wow, I didn’t know you could do that. I need to put a sanity check in place to keep you from viewing that.” And part of that’s just knowing what you want your software to be and having a vision long-term for how it evolves, I guess.

Mike Bombich:
Yes, because there’s the potential of feature bloat, and I have a very good idea of what I want. I’m the user number one. I’ve always been user number one. And actually just last week I thought of something fairly large that I’m going to start working on because I want this. And I know there’s a lot of people probably even listening to this podcast that are really going to like this functionality that’s going to be available a long time out, but I want it, so I’m going to make it. And there’s just no way it’s not going to happen because I’m going to make it. But then at some point I’ll start bringing in the user feedback and I’ll see how other people want to use it, and then I’ll make little tweaks here and there to make it work for more people. But a lot of software starts that way. I remember writing an application for my wife back in grad school, she was complaining about something and I was like, “Ooh, I could make a little application for that.”

Charles Edge:
I’ve done that too. And then you’re stuck owning it. It breaks with a new version and you’re like, “Oh man, why did I do that?”

Mike Bombich:
Yeah, that’s always the risk.

Charles Edge:
Well, it seems like working with Apple, especially on such low-level APIs, speaking of breaking changes being introduced, it can be a lot of chasing after their… To put it really sweetly, I think rapid innovation cycles and how much of the dev time can be for cool new stuff, like this new thing that you’re thinking about and how much is to pay that Apple tax so everything keeps working by keeping up with where the cheese is getting moved.

Mike Bombich:
Yeah. So it’s gotten better and it’s gotten better only because we’ve kind of set aside this whole concept of needing a bootable backup. Prior to that, prior to Catalina, I would say that four to six months of my development time was keeping up with that. I know, it’s eye popping, it was literally that much. It was all summer since the WWDC in June and then into October, November, just chasing all of the little issues that we had to deal with to get the system to be bootable and layer on top of that, the fact that it varies depending on hard drive, external enclosure, interface, so many different factors. And then Apple changes the OS and they change the boot semantics. And it was really hard to find time to do fun new stuff. But nowadays, there’s definitely I spent… I hate to say it, I spent about two to three weeks just changing how we implement a login item.
And that’s because they’ve deprecated all of the old way of creating login items and sometimes they deprecate things before things are fully baked for the replacement. So you end up spending time on that. But in the last couple years, just making changes for the OS changes, it hasn’t been terrible. I would say probably six to eight weeks is what I’m spending now just for OS related changes. And it really is just incredible having more time to do development on the fun. It’s made it fun again. And our new file copier, for example, I’ve had our other developer, Peter, working on that for about six to seven years. And then in the last few years, I really got to dig into it and spend weeks just tweaking performance and refactoring it so it does things a little differently. And like I said, it’s really brought the fun back, having more time for it. So yeah, Apple always keeps us on our toes. They love deprecating stuff, rapid pace of innovation is what you called it.
And I can’t blame them for wanting to deprecate things that make the operating system less secure. It looks like they’re going to take away some key chain functionality, and I’m not looking forward to dancing around that, but that’s the Apple platform and we live with it and we like it. It’s better than Windows. I actually use Windows for the first-time last weekend, and I was like, “Wow, that just doesn’t work.” You click the button, there’s supposed to be a window and the window just flashes away. It’s like, “What? This is a brand-new system. How does this basic stuff not work?” Okay, I love my Apple. Okay, I’m not a fanboy, but I love macOS.

Marcus Ransom:
So do we. But yeah, if we didn’t, we’ve chosen a really bizarre career and path in life. But I think hearing that approach from you and we see for me as well, I wonder if that’s also why your software is so successful and we enjoy using it is your approach is trying to understand why Apple are doing things. And when it’s clear something’s not working to stop doing it and find a different way of doing it was we see so many other tools, we’ll just keep trying to divide by zero or clinging on to something that’s been deprecated for many, many, many years, like login hooks and just knowing that one day it’s going to go away and things are going to break horribly. So you see you mentioned it a little bit about some of the things that changed. So Apple silicon and the read only system volume. How did you respond to those sorts of changes? What sort of impact did they have on how your tool was working and the features you had?

Mike Bombich:
Well, yeah, so actually it was Catalina 10.15.5 where we felt the first shot across the bow. And I mean, the writing’s kind of always been on the wall. We felt it. But in 10.15.5, first of all in Catalina, they completely changed the boot logistics. That’s when they introduced the system data volume pair. And that made, oh my gosh, the backend logic for dealing with that is just so ugly. It’s so horrible. I am so looking forward to dropping support for that. So 10.15.5 was the first shot across the bow. That’s when Apple introduced a bug that basically broke the firm link. We were no longer able to create those firm links. And at that point we were like, “Maybe it’s time to really start thinking about a different way of doing this.” And so I filed some bug reports then, and apparently the APFS lead engineer was in some hot water because APFS was making news and he didn’t like that.
And I didn’t like it either. I mean, we were totally dead in the water. But then the science system volume came along and because of some work we had done in Catalina, we were ready for it. We were ready to use ASR and make a full clone of everything, but at the same time, it was really buggy. It didn’t feel like I could build a commercial solution on top of that. So I actually had a conference call with Apple and there was somebody from the APFS team, somebody from developer technical support and somebody from marketing, which I thought was weird. I’m like, “Whatever, if he wants to come along.” So I thought the point of the call was going to be, let’s work together and work out these bugs in Apple Software Restore. And we get on the call and they’re like, “So what would it look like if you couldn’t make copies of the system anymore?”
And I was just completely back in my heels and I was not prepared for that conversation. I was like, “Well, I mean, that’s kind of the solution that I’ve been providing for people for the last 20 years. But hey, I mean, we could probably work with that.” So again, this is Apple’s platform and these are their choices and little old me is probably not going to have a huge impact on the direction of Apple. And I pitched my case for how people were using bootable backups and basically as a rescue device and so that they could immediately get back up and working on a new Mac or on the same Mac if their internal disc failed. And basically, they were willing to accept the loss of that, which I thought was pretty disappointing, but that’s Apple’s choices. So after that, we’re like, “All right, this is what we need to do. This is where the ship is going and we’re going to go with it and we’re going to skate to where the puck’s going to be. We’re going to build a solution for that platform.” And I’ve made peace with it.

Charles Edge:
I was also working at a place that had a binary at the time that sip and some of these things were just coming out. And I remember getting the call from Apple marketing as well to talk about some of those exact same things and being like, “Wow, this is a marketing call, but awesome. So I’m glad I’m not the only one.”

Mike Bombich:
Yeah. No, Apple marketing is a very powerful division of Apple.

Charles Edge:
Good. It’s well marketed software and hardware.

Mike Bombich:
It is, it is. Yep.

Charles Edge:
I do think that brings up an interesting thing because the recovery volume was also being introduced right around that same time. And it was a few years after file vault went from just being a home folder encrypted volume to being a full disc encrypted. But how did those options change things for you?

Mike Bombich:
Yeah. So that was in Lion. Yeah, the epic Lion release. Supporting a Recovery HD really wasn’t that difficult. It was basically, we had a little pre-flight task and a little post-flight task. And partitioning was definitely more challenging with HFS versus adding a recovery volume for APFS. That was a dream when APFS came along and it’s like add volume, poof, it’s there and this is like fast and there were no partition changes and I don’t have to deal with this little 600 and something megabyte ghost partition in the middle of things. The funny thing about supporting the Recovery HD though is that in retrospect, you mostly didn’t need it. We needed an archive of it and if you needed to restore everything, then we need to create the Recovery HD. But it was a little feather in my cap to support it.
And there were definitely cases where we needed it, but I don’t know. That was actually kind of fun. If there’s a little silver lining in the whole Lion cloud, it would be that it was a new development project and it actually kind of expanded how we approach. It’s like, all right, so we’ve got this file copying task in the middle, but there’s really a lot of other stuff going on and maybe we should give some progress indication of those other parts. So we definitely started moving into that direction.

Charles Edge:
Progress indicators are much harder than they seem like they should be.

Mike Bombich:
They really are. And people, everybody loves a time remaining indicator and the number of factors they have to go into this. I remember thinking a long time ago, it’s like, “No, I’m never going to do that. I’m never going to be able to do it reliably. It’s always going to annoy the heck out of me, so I’m just not going to do it.” And people really want it. And it’s like, “Oh, all right, I’m going to take a stab at it.” I took three separate stabs at it and I remember it was over the Christmas break about a year and a half ago, and I’m like, “All right, I’m going to give this two weeks”. And I just tore it all down. We had charts and graphs and all sorts of stuff and it’s like, “Man, this is as good as we can make it.” And it’s still not great, but it’s pretty good. I’ll keep it.

Marcus Ransom:
I don’t know if Joel Renick coined this term, but I know he uses it a lot. He’s calling it the placebo bar rather than the progress bar, because it’s just trying to work out if something’s happening or if the whole process is stalled.

Mike Bombich:
Yep. Yeah. Sometimes that’s all you need.

Charles Edge:
The number of files-

Mike Bombich:
Just a little indication that things are still working.

Charles Edge:
The number of files, but then the very last file is 300 gigs.

Mike Bombich:
Yeah, yeah. Exactly, that I mean… Yeah, exactly.

Marcus Ransom:
Well, the mechanism to determine the file size timed out at 300 gig. So it’s at least 300 gig.

Tom Bridge:
Deploying, managing and protecting Apple devices at work shouldn’t be difficult to require several solutions. Mosyle is the only Apple unified platform for business. By combining enhanced device management, endpoint security, internet privacy and security, single sign-on and enhanced and apps management into a single Apple only platform. Businesses can now easily and automatically deploy, manage and protect their Apple devices with one solution and add an affordable price with a solution for every business size and the best support in the market. Request your free account today and see firsthand why Mosyle is more than an Apple MDM. Mosyle is everything you need to work with Apple. To learn more, visit business.mosyle.com. That’s business.M-O-S-Y-L-E.com.

Charles Edge:
So eventually you replaced some of the traditional kind of CC or rsync types of options and even improved, I think on clone times as I was witnessing things, I felt like your times were better than mine if I was doing a straight rsync. And you mentioned in the very beginning of the episode kind of tweaking settings to performance tune and I don’t know if that’s block size changes or stuff like that, but I do feel like the guy behind Samba created rsync all the way back in 1996. So improving on that, it’s kind of a tall order, it’s a pretty mature piece of code. How’d you manage to improve on decades of awesome UNIX code?

Mike Bombich:
Yeah, so we’ve used rsync right up until version six and rsync was designed and if I’m not mistaken, initially to sync data between servers and potentially across a LAN connection. So that’s really what it was optimized towards. And they have such a tight protocol for how data is transferred and that’s really where it gets a lot of its efficiency. You’re basically just filling a pipe with data and the receiving end knows what order it’s going to get it in and knows what to expect. And if there’s even just a one bite difference in what you send down that pipe, everything’s corrupted. And we ran into that a lot and that’s one of the reasons we initially decided that we needed to transition away. If we wanted to add support for, like macOS has, I’m going to forget the numbers, it has different sized ackles than other Unix OSs.
So to properly support them, we needed to change the size of that variable and that was completely corrupting the other side. So we work through that, but basically at the end of the day we realized that it’s just not as extensible and adding support for things like clone files and APFS, adding support for… It had support for sparse files, but adding support for concurrency. For example, I don’t even want to think about what that would look like in rsync. Some people have tried, but we just decided, you know what, let’s start from scratch and let’s build something right on top of macOS that’s designed specifically for what we need to do now and also for what we need to do in about five to 10 years. And I’ve got all sorts of functionality mapped out into the future and we know that we can get there because of the changes that we made to our new file copier.
But one of the biggest things for performance is concurrency. Just looking at exFAT support, for example, I was doing some performance testing just last week and I had, I think a 20-gigabyte data set and it takes four minutes to copy it to an APFS volume and it was taking about eight minutes to copy it to an exFAT volume. And after picking that apart, I realized we can make some improvements by… I’m not going to give away all my secrets, but we found a couple ways to eek out a little bit better performance. But then what really surprised me, it’s like, “All right, how long does it take Finder?” Finder took 18 minutes and I got mine down to four minutes. And when I realized what the difference was, it’s because I’ve got four pipes writing data at the same time and I’m literally about four times faster. So that’s really why we rewrote it. And like I said, I’ve got a long road ahead of me and I know that I can provide some of the features that I want to offer in the future because we’ve got a very extensible platform.

Charles Edge:
I feel like this one I should know the answer to, but I don’t. You mentioned different file types. How do you handle the files that haven’t sync to iCloud? Because sometimes you have like, “Oh, I just recorded this thing and then I try to drag it into my audio or video editing app. But it doesn’t appear because it’s in the midst of trying to sync. There’s like a sync a weight type of operation I think happening. Have you found any oddities with that kind of stuff?

Mike Bombich:
I’ve not seen what you described, but the one thing that we see with iCloud and basically all cloud solutions, Google Drive, Dropbox, OneDrive, and I’m sure there’s many others, is that they’ll create online placeholder files. They have a huge mission to get all of our data off of our computers and into the cloud because Cha-ching, Cha-ching, that’s where they’re making their money. And the problem is that when they offload the data off your computer, they’ll create an online only placeholder file. And if you look at it in the Finder, it looks like it’s the full-size file, and it’s not, it’s a sparse file. It’s got zero data on disk and it’s just got extended attributes that describe the file. And because that data’s not physically on your Mac, we can’t back it up. At least we can’t yet.
We’ve actually been looking at ways that we could make a request for that. But it’s tricky because when I request to download the data, this file provider framework wants to store the data on the source, and I’ve always hung my hat on. I never changed the source, but we may make an exception for this specific scenario if we can keep it in check. I can’t download 300 gigabytes of data to a disc that’s got 40 gigs of free space. But if we can chunk it up and reliably evacuate that data after we’ve copied it, then we could add support for that, the reliably part-

Charles Edge:
Wow, that’s interesting. Yeah, I feel like these types of features showed up in Windows 7, 8, something like that, and they’ve been getting better in the Mac with iCloud and some of the other provider frameworks. But yeah, I remember reading that API documentation and thinking, yeah, I’m not touching this one, this isn’t for me.

Mike Bombich:
Yeah, it’s a big subject with cloud only data. And even with the concept of providing a cloud backup, there’s definitely value to having an offsite backup, but I’ve always felt like there’s so much value in having control of where your data lives. I want to own my hardware. I don’t want to have to pay a subscription to have access to my data. And that’s why I’ve always felt that the fastest and most reliable backup is the one that you have physical control over. And it could still be offsite. Some people will. Good grief, one guy said he sticks it in its trunk. I don’t think that’s the best place for it, but I used to send one to my wife’s work on Fridays and she’d keep it in a locked drawer and it can be that simple. But the cloud solutions…

Marcus Ransom:
Coming from the land of really crappy internet upload speeds and download speeds. I know that’s always a challenge here is it’s nice to know that your 300-gig data backup is up there in the cloud, but if that’s going to be like six weeks to get it back, it’s great if the house burns down or if there’s an earthquake or a major disaster or something like that, but it’s not so great when you’ve got a deadline. And that’s where for me, always having something that’s right next to you is so much easier and more reliable getting the data back off. It’s the same with restoring phones where there’s a level of convenience to restoring from an iCloud backup. There’s also a level of certainty from restoring from an iTunes backup because it happens kind of straight away and you’ve got that control over it, know where it is, but having both is even better.

Tom Bridge:
Here at the Mac Admins Podcast, we want to say a special thank you to all of our Patreon backers. The following people are to be recognized for their incredible generosity. Stubacca, thank you. Adam Selby, thank you. Nate Walk, thank you. Michael Sai, thank you. Rick Goody, thank you. Mike Boylan, you know it, thank you. Melvin Vives, thank you. Bill Stites, thank you. Anoush d’Orville, thank you. Jeffrey Compton, M. Marsh, Stu McDonald, Hamlin Crusen, Adam Berg, thank you. AJ Petrepka, thank you. James Tracy, Tim Perfitt of Twocanoes, thank you. Nate Sinal, Will O’Neill, Seb Nash. The folks at Command and Control Power, Steven Weinstein, Chad Swarthout, Daniel McLaughlin, Justin Holt, Bill Smith, and Weldon Dodd. Thank you all so much and remember that you can back us if you just head on out to patreon.com/macadmpodcast. Thanks everybody.

Marcus Ransom:
So we’ve got a question from our listeners. So from Tim Sutton, friend of the podcast, thinking about some generally very well behaved apps like Carbon Copy Cloner that have sections of their UI there to make sure you have full disc access enabled. For example, are there any weeds Mike would like to go into around APFS and snapshots?

Charles Edge:
And to be specific, what Tim was talking about was when you are asking for the privacy, the PPPC payloads, so when you’re trying to say I need full disk access from the software, Tim was definitely giving props for how well you handled those privacy options, I guess, in the code.

Mike Bombich:
Yeah, I mean, every once in a while we’ll get a disgruntled customer. I can’t believe you’re asking for full disk access. And it’s like, “Hmm, I literally can’t do anything at all, unfortunately because I can’t access external drives, I can’t access your private data,” macOS requires that I get that full disk access from you. And it’s funny, I’ve seen a few other applications that have kind of mimicked what we did and we were just looking at one recent, it’s like, “That’s my UI right there.” And I was like, “Oh, that’s okay. I’ll take that as a compliment. That’s pretty cool.” And I added the fish. The fish is fun and because I love fish. But in terms of snapshots, any particular weeds that the color wanted to get into, I could go on for days about snapshots.

Charles Edge:
I would imagine that you could, but I think he was mostly after the privacy control screens where a lot of vendors are maybe not the best stewards of those. And he felt like you were, So that was-

Mike Bombich:
Yeah. Well-

Charles Edge:
… as much a compliment as a question probably.

Mike Bombich:
Yeah. So I guess on the question side, the Apple gives us literally nothing for the privacy controls. I cannot tell if you’ve granted my application full disk access. I can’t tell if you’ve specifically disallowed full disk access to my application and I can’t ask you to just give it to me. I can’t just say, “Hey, would you give me full disk access? Yes or no?” I can’t do that. They don’t want it to be that easy. And I get why they don’t want it to be that easy. But the fact that we have absolutely no interface at all into the privacy controls is really frustrating for us and the end user. It means that I basically can coach you through it, but I can’t really do it for you. I can’t get insight into why it’s not working, and that can be really frustrating. So we definitely do the best we can to coach people through it.
And on that, in Ventura, it’s actually gotten a little bit simpler. They finally got rid of the API that they deprecated themselves years ago where you have to click on the padlock icon. So now it’s literally one step, when CCC prompts you, please give me full disk access. You literally just start dragging the fish icon. And then in the background behind the fish, the security settings or the privacy settings pops up underneath the fish and boom, you’re right on the drop zone. So they’ve, they’ve made it a little bit easier, but every year I ask for some API to get direct insight into it and every year they’re like, “Hmm, no.”

Charles Edge:
Yeah, it would be nice not to be trying to load a framework or, because there are operations, like one that I was working with a couple weeks ago was the Bluetooth, and it’s like, “Well, I’m not even going to try to do anything if I don’t have access to it.” But yet, if I don’t know whether or not I have access to it, now I’m doing some weird stuff under the scenes trying to figure out if I have access to it.

Mike Bombich:
Yeah, you’re poking at files and it’s like, “Do I have access to this file?” Yeah.

Charles Edge:
Exactly.

Mike Bombich:
It feels dirty.

Charles Edge:
It does.

Marcus Ransom:
Mike, I think that might have actually been the first positive feedback I’ve heard about the new system settings in Ventura. So that’s awesome.

Mike Bombich:
That’s all I’ve got.

Marcus Ransom:
That was really, really awesome. There’s been a lot of gnashing of teeth and the skies falling in, but seeing you approach it from… This is a better user experience, gives me hope of everybody else finding other things that are significantly better when we see it in its final form.

Mike Bombich:
I am willing to give props when there are props to be given. It is just one teeny tiny little part of it. And they’ve taken out two steps that I had to coach people through and I’m totally thrilled about that. But it took me a really long time to get used to those new system settings.

Charles Edge:
Yeah, I am very much used to them after screenshotting a whole book, but I personally like it better. I mean, mostly it’s just a search thing now, you don’t actually look for something. I feel like I did use to spend… I mean, I had gotten to the point where I just searched anyways, but I did use to spend time sometimes looking for the one to do the thing, and now I just search and I-

Marcus Ransom:
I’m convinced system preferences moved. That’s the only way I can come to terms with my own ability to find things over all the years is that there’s some randomizer built-in as to it’s alphabetical today, it’s-

Mike Bombich:
Well, they did. Every OS, it seemed like the icons were moving around, they were changing the sections and…

Marcus Ransom:
I’m talking every hour for me, maybe that’s just me.

Charles Edge:
Well, Mike, I feel like I can’t think of anyone who deserves as awesome as a result out of their efforts as you can possibly get. And it’s really been an honor. When you think to the future though, how do you think the industry will continue to shift over the next few years?

Mike Bombich:
Oh, boy. I think we’re going to see a different approach to storage from Apple. You’ve seen this whole concept of storage pools and I don’t know, I feel like at some point Apple wants to get to a complete abstraction away from the storage. And I think right alongside of that, we’re going to see that cloud stuff kind of join into this pool. And like I said, I will go wherever Apple’s platform takes us. Like it or not, it does feel uncertain in terms of that though and really the uncertainty for me is my internet connection because I literally have like, what is it, 12 kilobytes per second upload or 12, I don’t know, it’s 12 megabytes, but it’s not much, it’s not nearly as much as it is down.
So the idea of pushing data out to the cloud seems like it’s just always going to be slow. And I hope they don’t abstract it so much that you literally don’t know where this particular file’s going at any one time. But I definitely see a lot of market interest in going to the cloud and it seems like it’s virtually impossible to stop the market interest.

Charles Edge:
Fair enough. And Marcus, we have a bonus question, right?

Marcus Ransom:
Yeah. I just came up with this one now, so we’ll see how this goes. So one of the things we like to have on the podcast is bonus question where… This one I thought of is over the years, what is your most hated storage interface? What storage interface has given you the most grief?

Mike Bombich:
SMB.

Marcus Ransom:
Is it one that still exists or is it one that you celebrated seeing the end of?

Mike Bombich:
Oh, I wish we could celebrate the end of it. SMB, I would have to say is it has to be the worst. And it’s not all Apple’s fault. You see a lot of NAS’s out there and it’s like, seriously, we’ve got this Windows naming convention. I literally cannot have a file named AUX on this NAS because I don’t know, in 1995 they had Calm and AUX and whatever they are. So there’s that and then it feels like we’re just constantly running into some weird issue with SMB every week, every week. And I always tell people, and we actually added this little switch to AFP or switch to SMB thing within CCC’s source and destination selector, because if you’re having problems with SMB, eh, let’s just try AFP. When was it? Apple deprecated it, I think, if I’m not mistaken, it was like Lion, was it that far back?

Charles Edge:
Well, Apple talk, yeah. But then AFP, AFP lasted for a few more years, I think.

Mike Bombich:
But it’s been deprecated for a long, long time. And finally we’re starting to see Western Digital has completely dropped AFP support from their My Cloud devices. So it’s no longer an option. And we’re forced to use SMB and it’s like, “Ah, all right, well let’s see what else we can do with this.”

Marcus Ransom:
So Charles, how about you?

Charles Edge:
Yeah, so I wasn’t sure if you meant physical interface or…

Marcus Ransom:
Whatever is going to lead to a hilarious anecdote.

Charles Edge:
Yeah, I’m going to say EMC and no offense to the people at EMC, but some of those big sands are just hundreds and hundreds and hundreds of features. And if you don’t check the right box and just the right place or enable that with a config file in their old Dart OS, then the whole thing just doesn’t work properly and you’ve paid potentially millions of dollars for storage that doesn’t work well. So that was always the hardest one for me was EMC storage notes. But again, having said that, they were the first that I worked with to do straight fiber to the disc. I mean, they were the fastest by far. And then I would say on the physical side, some of those old scuzzy interfaces that bent every time you plug something in, those were the worst or had a three-foot cable and you couldn’t twist it, you just couldn’t get everything plugged in. Those were off.

Marcus Ransom:
It was like three-foot diameter on the cable as well always.

Charles Edge:
Yeah. Yeah.

Marcus Ransom:
Just like a core of concrete reinforcing bar in the cable or something to…

Charles Edge:
Yeah, the blue and white had one of those interfaces on the back, I think the G3. How about you, Marcus? You got to have something, you made up the question.

Marcus Ransom:
Yeah. So for me, as much as you know, I absolutely agree with you Mike, AFP kind of worked and now we’re forced for SMB, but I think that’s seen the resurgence for me of AFP being a bad thing, purely from the point of view, there are organizations, so one sort of real use cases design studios where everyone’s working on the same InDesign files, so we want to have all of our assets stored on a server that everything can access on the server when maybe the software from Adobe doesn’t support working off a server and it doesn’t work on SMB. So rather than going, well, that probably means we shouldn’t do it. We’ll then get software that emulates AFP on the SMB share. Now it’s a great idea, but when-

Charles Edge:
I know what you’re talking about.

Marcus Ransom:
Yeah. When it works, it’s wonderful. And unicorns come dancing out of the file system and then when it stops working and you’re in that little bit in the center of the Venn diagram when none of the circles overlap because nobody wants to support it and there’s been some data loss and Apple are going, “We deprecated AFP, Adobe are going, we don’t support working off storage.”

Charles Edge:
Well, I do find that sometimes they deprecate, but don’t remove. So if you hit from the desktop command and K and type AFP colon wack, wack name, it’s going to attempt to open an AFP pipe to that node over 548. Whether you should just do that or not.

Marcus Ransom:
If so, just that it did not work, then we wouldn’t have this problem.

Charles Edge:
Yeah.

Marcus Ransom:
Stop enabling people to create overly complex solutions that when they fail, they will fail hard.

Mike Bombich:
Yeah. There really does have to be a hard cutoff at some point where people are forced to bleed so that they’ll finally complain loud enough and then whoever’s fault it is Apple or the NAS, they’ll finally fix it to work with SMB and then everybody’s happy.

Charles Edge:
And then we can all use NFS again and it’ll be like the ’70s all over again. Bell bottoms, it’ll be awesome.

Mike Bombich:
Oh my, not NFS.

Marcus Ransom:
So Mike, thanks so much for joining us on the podcast. So if people want to find you on the internet, where’s the best place to go? Looking apart from calling your support line where you answer the phone every day.

Mike Bombich:
Bombich.com, and if you’re using CCC, I’m right there in the help menu, ask a question about CCC and then it’s either me or Rob or Nancy and we get you straightened the way.

Charles Edge:
Sweet. That’s as easy as it comes. We’ll put a link to that in the show notes for sure.

Marcus Ransom:
Okay. Well, thanks very much to our sponsors and thanks to our awesome Patreon subscribers and we will see you next time.

Charles Edge:
See you next time, and thanks for joining us again.

Mike Bombich:
Thank you.

Tom Bridge:
The Mac Admins Podcast is a production of Mac Admins Podcast LLC. Our producer is Tom Bridge. Our sound editor and mixing engineer is James Smith. Our theme music was produced by Adam Kudiga, the first time he opened GarageBand. Sponsorship for the Mac Admins Podcast is provided by the macadmins.org Slack, where you can join thousands of Mac Admins in a free Slack instance. Visit macadmins.org. And also by Technolutionary LLC, technically, we can help. For more information about this podcast and other broadcasts like it, please visit podcast.macadmins.org. Since we’ve converted this podcast to APFS, the funny metadata joke is at the end.