Episode 333: Henry & Eric on GitOps

Please note that this transcript was generated automatically

Speaker 2 (00:01:21):
Hello and welcome to the MCAD Men’s podcast. I’m your host, Tom Bridge. And Charles, that was a lovely banana stand. You briefly showed on the zoom call there. How is your weekend progressing?

Speaker 3 (00:01:31):
Oh, all the printers are working today, so excellent. That happens rarely. There’s always at least one broken down every now and then. They’re all broken at the same time, but today they’re all working.

Speaker 2 (00:01:45):
I understand that every three D printer spends at least part of its life broken.

Speaker 3 (00:01:48):
Yeah, you might be a redneck joke in there,

Speaker 2 (00:01:52):
But

Speaker 3 (00:01:54):
Because I have one that hasn’t worked in eight months until I spent hours working on it and getting it fixed, the Thompson tube had collapsed, if that makes any sense. But not the Thompson twin

Speaker 2 (00:02:10):
Twin. There are people for whom as long as the Thompson twins are okay, we will be fine. But telling me that the Thompson tube bit broken is I’m just going to smile and nod and say, yep, that sounds really unfortunate. I hope that gets fixed. And Henry, welcome back to the podcast. It’s always great to have Henry Steinberg Gohan join us from ral. And you were last on a couple of years ago and this is your third time joining us. How have you been? How are things in your world?

Speaker 4 (00:02:43):
Hello, thanks. I’m good. So today was a whatever, another hot summer day, so I had a lovely weekend. I’m not very good with Fahrenheit, but I think it’s 78 to 80. That’s the temperature here at Kids Went swimming. I had early dinner and now I’m joined you. So it’s a great Sunday. I was

Speaker 2 (00:03:09):
Going to say that’s spectacular.

Speaker 4 (00:03:11):
Yeah, definitely. And yeah, since last time of course a few things happened. We have rebuilt the company behind Centra. So we operate as an independent, well-founded organization still, and we take full focus on the Centra platform and we have doubled our headcount. So that’s a few things

Speaker 2 (00:03:36):
That’s spectacular. And you brought a friend this time and welcome to the Mac Admins podcast, Eric Fa.

Speaker 5 (00:03:44):
Yeah, hello, nice to be here. Thanks for the invite.

Speaker 2 (00:03:48):
Yeah, it’s a great pleasure. I was going to say, I think we’ve crossed paths at a number of different conferences over the years, and it’s such a joy to have you with us today to talk a little bit more about GI Ops and a lot of really great topics for MAC admins. And Charles, you wrote a really great synopsis for this. I’m going to let you introduce this topic. I think this is going to be fascinating conversation.

Speaker 3 (00:04:12):
Well, the synopsis was pretty short, but basically we’re going to look at what Git ops and infrastructure is code mean and how the modern code management tools help shape an emerging modern device management approach. We talked about infrastructure as a code in a couple of previous episodes and last week we even talked about GitHub actions. But do you guys mind unpacking what you see as GI ops? The definition for GI ups?

Speaker 5 (00:04:50):
I can take it this one if you want, Henry, or

Speaker 4 (00:04:54):
Yeah, go.

Speaker 5 (00:04:56):
I took some time this afternoon to look around because it’s this kind of concept where you think you have it, but it’s always interesting to see what the industry is thinking about it. But I think I agree with what most of the people say. So first it’s a way to describe infrastructure as good. So you need a language, you need files, you need something to describe what you want to achieve. And then there is the process of making a merge request or a pull request to manage the changes. So this allows peer review, this allows actually change management. This is really at the heart of the system and this is really the interesting part. And of course there is a last part. There is a C I C D pipeline to actually implement the changes and make sure that the infrastructure or what you are trying to configure using this system is not drifting away and to make sure that you are always staying on track. So this is a combination of these three systems.

Speaker 3 (00:06:15):
Yeah, I would say what you’re describing, because there’s a whole bunch of stuff inside a concept, and I guess in the code world we might call that a framework and in the business world I guess. But I’d say that if we’re going to describe OPS as a framework, I guess, how flexible do you find that it needs to be for organizations of different types? Because that definition really does a good job of covering the software development DevOps pipeline type of thing. And it feels like what we’re doing is we’re describing the way that we manage devices. If we’re going to apply GI ops to M D M or to device management in general, we manage code. So I guess how flexible do you find it in that regard?

Speaker 5 (00:07:20):
So you’re totally right, it’s a framework because I’ve read somewhere people describe it as an operational framework. So it’s totally ly a good way of describing it and the pieces can be changed and even the processes can be adapted. So in terms of flexibility, if you are talking about the scale of a company and how complex the processes are, it can start really small. I mean, as soon as you can describe something in code, you could have, even if you are a single person working on this kind of stuff in your company, you can start really small, meaning you need a Git repository, you don’t really need MER regrets because you’re on your own. But it’s probably better to document the change, the changes and group the changes. But code reviews are a bit lame if you’re on your own. And yes, it could be a couple of scripts to apply your changes.

Speaker 5 (00:08:33):
So it can start as small as that. This is the way we structured the TO in Vancouver when we talked about that, it was one of the first steps. But of course, as complexity can evolve, as soon as there are code reviews, you can protect your branches and make sure that you cannot push directly into Maine. And the last steps we had in the talk in Vancouver for example, was a much more complex pipeline where we even had a staging and a production environment in the same repository and merging to testing or merging to prod would deploy the code in two different environments all based in the same GI repository. So if the question was how flexible in terms the size of the company and the complexity that the company is ready to manage, it’s quite flexible in terms of the assets that you want to manage.

Speaker 5 (00:09:42):
It’s not that flexible. You depend on what is available out there. If your product has a good I, if it has a Terraform provider for example or not, if you, you have to write one. Of course, in terms of infrastructure, there’s much more to be managed as in terms of software because stuff we do for example with Central is kind of an evolution taking what is done usually for infrastructure and applying it to configuration. But yeah, so being a framework, GitHub’s being a framework, it’s extremely flexible, but you depend on what is possible in terms of the assets you want to manage.

Speaker 3 (00:10:29):
Yeah, and I mean I can see that as being a pretty simple code review. It’s like, oh, this is an XML file or this is a J ss o n file. This is what changed in those two files. Do I want to push it or not? So I think for a larger atomic operation where there’s 10 or 15 files being touched in order to set a configuration change, it might be a little bit more, but that’s still not that many places to be looking. Although in I guess declarative, there can be a whole bunch of stuff here and there. So that’s a thing. I guess speaking of tools, so you mentioned GitHub, you also mentioned your C I C D pipeline. So how do you see tools like maybe Kubernetes or other C I C D tools factoring into this paradigm? So

Speaker 5 (00:11:26):
Kubernetes, my little experience with it, of course, I mean you describe what you want to achieve and you let Kubernetes achieve it state. So

Speaker 3 (00:11:40):
Somewhat declaratively,

Speaker 5 (00:11:42):
It’s exactly what it is, a bunch of mechanisms in the background. My understanding, I don’t, but my understanding is that it’s a bunch of mechanism in the background trying to implement the changes required so that what you are describing is actually what is running and you describe everything in YAML and you’ve got a templating language included in that. Or you can use another templating language and you push it to Kubernetes and Kubernetes itself is going to reconcile it. There are some limits of course, because there’s Kubernetes on Amazon and there’s communes and Google and some things that are not as easy as that because something still have to happen before other things, and this is where it gets really interesting. But yeah, so Kubernetes obviously from the ground up the way was organized, obviously a perfect candidate for GitHubs, the C I C D pipelines. Yeah, they got integrated in every code management tool. GitHub started without the actions and GitLab started without the pipelines. And there’s obviously the integration of the pipeline next to the code is a logical evolution of these tools because, and even I think even GitLab’s changed their marketing and they are saying they invented, and I forgot the name, but even saying that they are more an operational tool than just source management of course, and there are nice things about identity, but I’m anticipating your next question, but there are nice things around identity and these tools.

Speaker 3 (00:13:46):
Yeah, no, I do find sometimes it’s hard to know when the juices work, the squeeze. If I start writing a web app, at what point do I containerize aspects for deployment? Do I start that way or do I jump into that? When do I bring in something like Jenkins or Artifactory or some of the other tools that we use to hold and make it easier to deploy things? Sometimes it feels like if I start too early with certain aspects, then I’m just paying down tech debt later on what I shouldn’t have done so early. So sometimes deploying the right tool at the right time, it can be hard to know when that right time is. But other times, having said that, I’m also finding that I moved to writing my test cases before I write code, which is not how I used to do things. And so now I’m writing code to fit my test cases, if that makes sense. So sometimes you’re like, but maybe it’s better to start that way from day one in the Artifactory example, maybe I’d throw my packages in there just so I got ’em as soon as I’m doing it, rather than going back six months down the road and being like, oh, let me find where I got all this stuff and throw it in there if I know I’m eventually going to do it.

Speaker 5 (00:15:28):
Yeah, I totally get it. This is where the platforms as a service, I think this is where they come in because they give you less flexibility, but this deployment and the path from code to deployment is really short and that was like it’s still a very good system. I guess now we have cloud run or the Lambda functions or of systems like that, depending on the scale of what you are building is probably a better way of hosting and deploying stuff because there is a pipeline that is not that complex having to maintain a Kubernetes cluster, even if it’s on a w s, I guess at Google it’s a bit better, but a w s, it’s still complicated and especially even if it’s managed. And so if someone in your company there’s a bunch of Kubernetes clusters and you can have access to them and someone is managing them for you, why not? You could start directly using Kubernetes deployments. You get the rollovers and as soon as you are producing nice containers that are stateless, everything is fine. Depends on a lot of, there’s a massive stack of infrastructure that is required sometimes and that can be scary, especially if your day job is to manage macros and not to

Speaker 3 (00:17:16):
Manage

Speaker 5 (00:17:16):
Coup stack.

Speaker 3 (00:17:18):
Right?

Speaker 2 (00:17:18):
Yeah, I was going to say that’s always the point at which I always start to get a little bit nervous. When am I managing max versus when am I managing server clusters or containers that are getting built up and torn down all the time? Adjusting between those two spaces, it’s a pretty big adjustment, especially for individuals who are used to, Hey, I’ve been managing this M D M configuration, I’ve been managing these configuration profiles and by extension these devices, is it really that scary to be managing Kubernetes instead of just a set of profiles and deploying those out there? I think it’s a mental hurdle more than it is a textural hurdle that people need to go through.

Speaker 3 (00:18:01):
I don’t know. I mean if you’re trying to host your own cobes environment, that’s when I would start to disagree a little bit. But if you’re doing it on Amazon, I think Eric made a really astute point. Yes, those tools limit your options because it’s like these massive Swiss Army knives that have a hundred things and you’re like, I just need the bottle opener. Well, that’s a bad example. That’s the easiest one. I need the nail file and you have to open 40 blades before you find the nail file as opposed to doing it the Amazon way. You’ve got eight blades in front of you and so you’re trained to fit it into, sometimes it’s nice to be told how to work as opposed to be told you can do anything now go and you’re like, but what’s step one? I don’t know. I think Eric made an awesome point with there are less options, but you have a predefined workflow and also a support structure in a way ish with a W S and G C F. It’s hard to know exactly what the support structure is sometimes, but they’re there ish. So

Speaker 2 (00:19:21):
For varying values of there for

Speaker 3 (00:19:23):
Sure

Speaker 5 (00:19:27):
Tend to make a little excursion in one. I have one question and I never got mean. Some of the Mac admins, they have to manage macros and they were for companies who are software companies, they operate products at a planetary scale. They have people and they have operations, they know how to manage Kubernetes and they know how to manage everything. And sometimes this means I think they are not getting the help they could from their own company to help them run their product.

Speaker 3 (00:20:06):
Sometimes that’s a tribal knowledge thing. I see where you’re going with that and other times it’s kind of a budget thing like, oh, that person reports into another group. So we’re not really supposed to bug ’em, but as buddies I can ask, but if I don’t actually know them then as a buddy I can’t really ask Go ahead.

Speaker 5 (00:20:31):
Yeah, because they are like people, their job is to actually make the tools that the developers are using. So what the developers are more productive. It could be like people, they could help the admins be more productive as well. Of course not the same pot of money. And this is why the admin are always seen as being a cost.

Speaker 3 (00:20:55):
And I think it’s fine to reach out and get that help until, and this seems to always happen until someone misses a deadline and is like, oh, I was helping Eric with that project, and then their boss is like, but that’s not my problem. Boss’s problem. Yeah,

Speaker 5 (00:21:16):
Sorry, frosty to was maybe a silly question, but sometimes it’s really frustrating.

Speaker 3 (00:21:19):
No, it’s a great question. Yeah, I mean I’ve definitely run into that and sometimes it can be something as esoteric as, oh, I’m trying to figure out this weird P k I issue and there’s a P K I guru over there and I know that I can ask that human a couple questions a month without getting any blowback from their boss until you ask that one question that takes two days to resolve and then it all rolls downhill from there. So yeah, I think it’s just the, not to overuse containerization, but it’s the containerization within management infrastructures, I guess human management, not computer management. But now we’re getting off track. Sorry. No, no, it’s great. It’s a great question because you are absolutely right and so many companies that don’t call themselves software companies are, most companies are technology companies these days, even if you’re running brick and mortar stores, that the technology is the weapon that allows you to scale or be consistent or do all the business functions. But Tom, you asked something about M D M, do you want to take the next question?

Speaker 2 (00:22:44):
Sure. And I think that it’s really, really interesting for us to think about all of the different ways this is relevant to the mackman and one of the really important parts of any Mackin men’s life these days is declarative management and the mobile device management tools that they use to do those jobs. So as we think about the world in which we live, what kind of role can GI ops play in mobile device management or declarative device management on top of that?

Speaker 4 (00:23:15):
I think it can start very, very lightweight. So whatever, you have some custom profiles and put them into a version control, and then of course you can look into what M D M I’m using and does it have an A P I? Then eventually you find a way to bring this better together. However, some of the MDMs are whatever, quite some years old, they may not have a super modern A P i and there it can get tricky. So I know there are some projects scripts mainly that try to just ease this up for and they’re maintained in the community. And so the idea and the direction is always the same. So how can I have my version of an artifact and how can I make an update and then have bells and suspenders of deploying that in a way it should be then deployed to the devices. But yeah, so I think Eric can just join in and get one level deeper in some of the difficulties that we faced.

Speaker 5 (00:24:41):
Yeah, we identified when developing central, we identified change management as something that we wanted to address. And for the same principle, let’s go back to the principles why people are doing GitHubs or DevOps is for reliability to avoid mistakes because there are less changes that mistakes are being made and if mistakes are being made, they are actually, we can see the differences, we can revert them and there is a kind of an automatic documentation of what was going on. So these mechanisms are really interesting and because with an M D M, you can really cripple the productivity of a world company if you are making mistakes done. There

Speaker 3 (00:25:38):
There been that,

Speaker 5 (00:25:40):
And so yeah, we thought it was nice to have this, a mechanism to allow this kind of workflow. So yeah, it can start as with an A P I, an A P I to upload a confi profile that could be enough to start having the first steps and to start implementing a GitHub workflow for the M D M and system. Make a little parallel with whether we said about Kubernetes is so Kubernetes being allowing you to describe your state and Kubernetes, making sure that the state is actually what is observed, what is actually running. I see. And this is a way we develop central. I see central a bit like it’s not Kubernetes, but it applies the same principle, meaning you describe the configuration you want to be deployed on your machines and you don’t actually send the comments yourself. You don’t make sure that the comments are successful or you don’t retry.

Speaker 5 (00:27:01):
You don’t have to do that because we want central to do that for you. So that allows you to describe the state you want and central should be the mechanism to apply this state and to make sure that the right conflict profile is distributed to the right machine to make and to verify that it has actually been distributed. And having an M D M transforms that is taking care of applying the state and is not only a switchboard for a P I to comments or stuff like that. I think this is what can enable much better workflows and that it can make it easier for the operators to think in terms of this is what I want to achieve and the M D M is going to take care of that because the goal is not managing an M P M. I think that the goal is to have good healthy macros clients.

Speaker 3 (00:28:06):
Do you mind explaining what Amazon Terraform is?

Speaker 5 (00:28:11):
Terraform? Yeah. So when we were looking into implementing that, I wrote the white paper about what it should do. And the first thought, because it’s always exciting, was to implement this in central to say, oh yeah, we are going to have changes and we are going to apply these changes and there’s going to be a history of all the changes we have applied and we even used it as interview questions for software engineers. How would you go and build something like that because it can address, there’s so many subjects. And of course while using that during interviews and getting some feedback and everything, I realized that the scale of this project is incredible. So I was using a lot of Terraform at the time for deploying the infrastructure for deploying Central itself. And I thought like, oh, but it’s great because it does take care of the state because, so to go back to your original question, problem is if you have an a P I to create update, delete and view resources, a resource can be like a config profile for example.

Speaker 5 (00:29:40):
Then if you want to remember in the GitHubs thing, there’s the pipeline and the pipeline is applying your changes and making sure that your changes are exactly the changes are present. That means that every time you have to read, see the resources present. If it’s not present, you have to create if it’s present, but it’s different, you have to update it. And if it’s present and should not be present, you have to delete it and you have to keep track of the stuff that you have to delete it or I mean there’s a state and Terraform is amazing at that because it does exactly that. It manages the state, it remembers the resources that have already been created, remembers the last time they were created, how they looked like, and it gives you two great step for the pipeline. It gives you a plan so that you can see if I’m going to apply this new configuration and you can see the changes and it applies them and it applies them using providers that meaning you have plugins for each kind of infrastructure are going or service you’re going to configure. So there’s a plugin for a w s, there’s a plugin for CloudFlare if you want to change a D N S, and we have written a plugin like a provider for central. So you can manage central resources, but it’s the same language. So they call it H T L, which is not a good language. You’ve got variables, you’ve got a little bit of control flow, a little bit of conditional stuff, and you describe your resources, you give them to Terraform. Terraform is going to authenticate and do the loop.

Speaker 5 (00:31:27):
It is going to build a tree of resources as well. So it’s going to kind of manage dependencies, building their dependencies that the resources that all of the resources depend upon first, which is great. And then it’s going to, yeah, look, is the resource there? If it’s not there, it’s going to create it, it’s going to update it, it’s going to make exactly the configuration converge. So this is the best thing you want for C I C D, the C I C D part of GitHub’s workflow.

Speaker 3 (00:32:04):
And when it’s done right, I find now I’ve got this declarative configuration file. I can say I want this S three bucket and I want simple email service and ap, AP and SS set up. I’ve got all the pieces. And like you mentioned, there’s some control flow, which is not terribly complex, but I don’t think it should need to be terribly complex because we’re not doing that many conditional statements. It’s like, oh, if then not this crazy nested where I need to define types and all that. But then I say in my config file, this is how it should be, and I upload a new config file or more poignantly to the conversation, I commit a config file and then a second human eyeballs it and make sure I don’t screw something up because we’re all human and we do that. And then once someone hits that button, it goes up and all the mechanisms, the providers that we’ve configured get set into place and that big thing that I said happened just happens. That’s declarative management. In a nutshell. I think

Speaker 1 (00:33:29):
This week’s episode of the Mac Monds podcast is also brought to you by Collide. Our sponsor, collide has some big news. If you are an Okta user, they can get your entire fleet to a hundred percent compliance. How if a device isn’t compliant, the user can’t log into your cloud apps until they’ve fixed the problem. It’s that simple. Collide patches one of the major holes in zero trust architecture device compliance without collide. It struggles to solve basic problems like keeping everyone’s OSS and browser up to date. Unsecured devices are logging into your company’s apps because there’s nothing to stop them. Collide is the only device trust solution that enforces compliance as part of authentication, and it’s built to work seamlessly with Okta. The moment collides agent detects a problem, it alerts the user and gives them instructions to fix it. If they don’t fix the problem within a set time, they’re blocked. Collides method means fewer support tickets, less frustration, and most importantly, a hundred percent fleet compliance. Visit collide.com/mac admins podcast to learn more or book a demo. That’s K O L I d.com/mac admins podcast. Thanks to collide for sponsoring this episode of the Mac Admins podcast.

Speaker 3 (00:34:55):
You have a Terraform module that allows admins to manage M D M profiles. Henry, that was the first thing you mentioned, like this is the easiest entry into this GI ops flow of device management, but your Terraform module allows admins to manage M D M profiles, apps, and enrollments with, I’m guessing maybe something like micro M D M under the hood, but I’ll let you unpack what’s under the hood there if you don’t mind. So

Speaker 4 (00:35:26):
I can take this in. I think Eric will join in again. So we started with St. Holland 2015, and of course back then it was the idea to bring osquery and Sunar next to a popular management system. What was gem? In 2017, we experimented with M D M, and so since then it’s in central, but it was experimental. And lately we have changed that to become prod ready. And Eric, you can just give it a little more under the hood.

Speaker 5 (00:36:04):
So when we started implementing the M T M, we’re still a consulting company and it was exploratory. It was like, how does it actually work and let’s go and implement it and see how it actually works. It was to get get really deep, move deep, I don’t know, try to understand how it works and the challenges because we, people are always surprised that some MDMs or they say, oh, my M D M is not reliable and stuff like that. But yeah, we wanted to know why. And you quickly discover that the protocol itself is not the easiest protocol to build something that is reliable. So no, actually we don’t use micro N D M, but we have our own implementation. We have added a, it allows us to implement stuff quickly. Sometimes we have added declarative device management. I think one week after the announcement, that was fun.

Speaker 4 (00:37:24):
That was same week. Same week,

Speaker 5 (00:37:27):
Yeah, that was, yeah, because we could do it. We have this in our code base, we have it in our code base, and this is exactly why we wanted it in our code base and we wanted, this is with one reason. The second reason, it’s possible to coordinate a lot of tools that are going to help you do the device decorative device management, but there is a level of integration that has to be achieved between the identity provider, the comments that you are sending, remembering who is the user that authenticated at D E P so that you can use it later on for, I don’t know, variable substitutions and stuff like that for the config profiles. And yeah, it’s easier with a monolith and I know that it’s not going to be a popular opinion around maybe, but yeah, sometimes a monolith is not the worst architecture.

Speaker 2 (00:38:41):
Yeah,

Speaker 3 (00:38:43):
Totally.

Speaker 5 (00:38:45):
So while it’s easier for us anyway, I’m not saying that it’s not possible to do it otherwise. I mean people are proving that works, but it works well for us like that and it allows us to quickly implement these workflows that are complicated. And so for the declarative part, I mean the Terraform module is only, you only have to implement crude operations. A p i crude a have to pull out a crude a P I on you or resources, and then you build a go library that can use this a P i, and then you add the resources to the provider. But it’s mostly crude stuff and translating between your json, the way you serialize your resources using JSON under representation of your object in Terraform. So it’s a translation layer and the library to interact with your crude operations. It’s mostly that. It’s not that complicated.

Speaker 3 (00:39:53):
Mapping between two methods of serializing objects is always fun until you get to the point where you’ve identified all the places where the dragons are, where you didn’t know what was being mapped where and how, oh, that was a sub array or whatever, and then you just bang out a script that does it all for you. And now that’s not a thing anymore that I guess that’s kind of integration work in a nutshell. But yeah. So in regards to the M D M piece, so Xtra doesn’t have to be the M D M, you can still use another one like a JAM for any others.

Speaker 5 (00:40:46):
Yes. That

Speaker 3 (00:40:48):
Was in unison, definitive, yes. I love it. And I guess, do you want to tell us a little more about packing OSS query and maybe Google Santa on top and what the thought process was like and I guess how much work that was,

Speaker 5 (00:41:07):
Which process to put it in a Terraform way or to implement it itself?

Speaker 3 (00:41:15):
Let’s start with the implement itself and then move from there to Terraform, because that’s the logical way to do that in my head.

Speaker 5 (00:41:24):
Yeah, so Osquery was the first thing we did. I mean, the interfaces are pretty clean, the protocol is pretty clean. That was a quick job. I mean talking to the agent, having the agent talk to the web application was a quick job. It’s more complicated to try to, the architecture of central itself evolved over I would say two or three generations to get it really right. And this is how to make all these pieces fit into the same event system, pipeline system and having central as a kind of framework itself to develop these kind of applications and have these applications work with each others, sharing the same event system, event metadata and allowing us inventory as well and allowing us to have these applications and the information that the agents are contributing actually work together in a unified inventory and with events that are attached to the same machine and stuff like that. So this part is the core central. The implementation of talking and building dynamically the configuration and give it to the client was pretty easy.

Speaker 5 (00:43:15):
The Santa protocol to synchronize the rules is different because for example, this is really interesting to see how defining a protocol can make the life of the people implementing it really like a nightmare. Because for example, trying to save on resources, I think they didn’t want to have to pass, I don’t know, 200, 400, 500 words every time there is a synchronization. So the agent Santa, so Santa is getting to give some context sometimes getting binary alo or block rules based on the signature on the team id, the bundle ID or this kind of information. And instead of just giving the client all the rules, like giving the S query client over queries, no, it is only asking for the differences. That means you have to send the operations at this role, remove this role, and that means that you have to manage the state of each agent in your app so that you can remember which when was the last time the client showed up. And while we are the rules he got, and we even had a bug like that, that was funny because we send the rules, but somehow the network failed. So we marked the rules as sent, whereas the client never received the rules. So that’s why now we are only marking, we are making another loop to make sure that if a client adds our next set of rules

Speaker 3 (00:44:48):
For the,

Speaker 5 (00:44:50):
That’s why. And the last round of query is always returning zero rules because it’s just a confirmation for us, it’s like a commit.

Speaker 3 (00:44:58):
Now you’ve now you’ve created a race condition,

Speaker 5 (00:45:04):
So depending on the agents, but the protocol is not the developing workflows that can help people and don’t abstract the agent. We want people who know que to be able to come to Central and after, I don’t know, 15 minutes to start using Osquery with San because they know Osquery the same thing with Santa. But Santa for example is one thing to distribute rules, but how do you build rules? And a lot of people here are missing output. They want the rules to build, they want people to build the rules themselves, right?

Speaker 3 (00:45:47):
Sometimes the way that we help admins is actually to point out useful things. And I think with OSS query, one of the useful things are query packs. So did you find that you were adding any query packs that you found useful for the Mac when you were doing it? Or are you just, oh, here’s the stock deploy. It’s do as you will.

Speaker 5 (00:46:13):
What I can see, I mean Henry can compliment later, but what I can see is we want to always query as the control loop for D M D M or for any kind of configuration that is set, maybe using, or even if you want to distribute an app, you make sure that it’s actually the app is running. For example, if you want to distribute center, you distribute center that you have to make sure that Santa is running, that system extension is up and that it’s the right version. So these are what we call compliance checks. These are the queries that we keep sending to our customers, for example, to make sure we have a bunch of them. And it’s really to make sure that the system is actually doing what it is configured to do to be the control loop of the configuration. So talking about the stuff that is always scheduled, and I think you have more to talk about security, for example.

Speaker 4 (00:47:24):
Yeah, I think in general a query pack can give you a good point, but then from there pretty quickly it goes very specific. So what is the context? Why are you using that? What you looking after and what are you trying to achieve? And so in this regards, what query has powerful features? So you can have automatic table constructions, but then okay, what you are looking into, and then you have to use the capabilities of always query to find the thing you are after. It could be, I want to look into the app usage that monkey is collecting. So this is in a database, you can try and filter out the information, but then you suddenly have tons of information. And again, so what they’re trying to achieve here and a copy pasted query pack as asset is a good starting point. But the next level of using query bring back the information. This is, as Eric said, the concept you find also in other products where it’s more like a compliance check, is the value as expected? And then if it changes for every reason you have an event and act on this event. And in this regards, a clari pack only is the wrapper for distributing that.

Speaker 2 (00:49:01):
Love it. On a previous episode you talked more about identity and Eric, you talked a little bit about identity here today. How does identity and identity management factor into the architecture that you’re building here?

Speaker 5 (00:49:17):
So about one of the functionality that is really great and that is built in GitLab and GitHub and GitHub for example, is so in the building, in the pipeline, in the C I C D pipeline a stage, the job that is running can actually get a signed JSON web token. And this JSON web token is actually a start open I connect token and information in, it could be like for example, the repository, the branch, even the commit the name of a job and everything is signed. And you can configure, for example, your A w s account to have an open ID provider that is actually GitHub and means that your jobs, they can exchange their for short-term credentials that they can use to configure your infrastructure. And this way you are not pulling secrets in your GitHub configuration. You are just configuring the trust between, you are just configuring a W S to accept your job and to trust the jobs signed by GitHub so that temporary credentials can be obtained.

Speaker 5 (00:50:56):
And this is, for example, I would say like an evolution of a nice use of open ID connect. And you can do the same with ub. So with Google Cloud you can do the same between Terraform Cloud on a w s, we use it a lot. And this is one example of identity in the C I C D system, being able to remove secrets, remove a P i secrets, we would love to implement that for Central. We still haven’t gotten to that. But yeah, we would love to be able to give the clients temporary credentials to change, to do changes in central based on shared open ID identity. This is one example, but I mean identity is everywhere. I mean it’s at the authentication of D E P I told you that it’s one of you. You want to be able to reference claims, sign claims in your payload or in your skip payload to put some attributes in your department organization. You want maybe also to be able to reflect changes to tag machines based on the groups of a principle user. You may have to implement an S C I M server, a new M D M solution

Speaker 3 (00:52:32):
And assertions. And for what it’s worth, all of those vendors that you mentioned are also identity providers. So you can use your GitHub identity with your A W s or your kognito identity with GitHub or your I A M identity from Google with both of them. I think there’s a lot of, this is another one of those places where there’s just so many ways to do it. And every now and then personally, I’m like, just tell me how to do it. Please somebody make this decision for me. It’s too much.

Speaker 5 (00:53:14):
And and the best part, because you are talking are trying to implement something between two systems, you not only have to read one documentation, you have to read two documentations.

Speaker 3 (00:53:23):
Oh yes,

Speaker 2 (00:53:24):
If you’re lucky, it’s just two.

Speaker 3 (00:53:26):
And I just don’t trust the walkthroughs that you find out on the interwebs anymore. Having written plenty of them in my career, there’s just so many things that like, oh, that one little checkbox that they didn’t even mention, that changes everything. And that’s one of the hard things from a development standpoint, going over to button mashing around an A W s, one of the reasons that Terraform I think is so attractive to a lot of us, there’s too many buttons and you’re searching in three levels deep to find something and clicking around to find something when you’re like, it’s so much easier just dropping it into a J SS o n or YA L if you’re in the Google side or whatever.

Speaker 5 (00:54:17):
Yeah, sometimes it’s hard to help customers because there are some questions I cannot answer them anymore because I don’t do that anymore. I just do Terraform apply and I have to look at my Terraform files and look way down in the attributes to find out, oh yeah, oh, did you tick this box? Because there’s so many things that I don’t do that anymore. I just have them apply.

Speaker 3 (00:54:48):
Yeah, the first time I noticed this thing where it’s faster for me to do stuff in command line interfaces, sometimes, not always. Sometimes it’s ridiculously lengthier to try to do this, but than it is in a gui, was trying to manage Cisco ANets and back in the Cisco wireless days versus trying to manage sonic points from SonicWall. And on the SonicWall side I’d be clicking, clicking, clicking, clicking, clicking, where’s this under this other weird menu, I can’t find it. And then on the Cisco side you’re like, okay, let me just upload a config file to the As s a and bam, everything’s done. So yeah, the more things change, the more they stay the same. Although the way you serialize data and the Cisco file was always like, okay, this is just rife with problems anyways.

Speaker 2 (00:55:46):
Absolutely. And in addition to OSS query and to Santa and things along those lines, monkey is also supported by Xtra as well. What makes monkey’s implementation a little bit special compared to OSS query or Santa in this regard?

Speaker 4 (00:56:03):
The monkey integration is on two levels. So one is of course a reporting part pre and post-flight can ship information as it’s possible with cell and monkey report. So we get a lot of precise information on that level. And then we’re also using, and we call this monolith, it’s a layer to have your existing workflow of monkey where we take the manifests and the catalogs and can have a dynamic way of presenting it to the endpoint. And in this way, we’re just having a very flexible way of providing an endpoint, the manifest it should receive. And yeah, there’s some automation capabilities with tax and we have a mechanism like sub manifest where it can have groups, it can whatever browsers or things that are required for business unit. And then you can assemble this and it’s quite flexible, but it doesn’t bend the way how monkey is working. So it’s a very thin layer on top. I hope I described this well. Maybe Eric, you can just join again and just correct where I described whatever, not precise enough.

Speaker 5 (00:57:42):
Yeah, so you can work with your existing monkey repository and Central would read the information about all the packages and then we would build a manifest with some manifest dynamically to present this repository to the client based on tags. But it could also based be based on sharding. And this is an interesting thing. We developed that with a customer. They had a fleet of 15,000 max, and it was really, really interesting to see how they could really slightly increase the distribution of a new version of a software. And of course you can do that with rebuilding your manifest and everything, but it was really, really easy to do that using just the sharding at tribute on the version and could see with the reporting, we had a beautiful burnout graphs of the new version coming and how fast the new version was being distributed. So yeah, basically we are in between the monkey agent and the monkey repository and we do dynamic manifest and catalogs that can be configured using Terraform

Speaker 4 (00:59:14):
Again.

Speaker 5 (00:59:16):
Love it. Here at the Mac admins

Speaker 2 (00:59:20):
Podcast, want to say a special thank you to all of our Patreon backers? The following people are to be recognized for their incredible generosity. Ubca, thank you Adam Selby. Thank you. Nate Walk. Thank you. Michael S thank you Rick Goody. Thank you Mike Boylan. You know it. Thank you. Melvin Vez. Thank you. Bill Stites. Thank you. Anush Ville. Thank you. Jeffrey Compton, m Marsh, Stu McDonald, Hamlin Cruin, Adam Berg. Thank you. AJ Reka. Thank you. James Traci, Tim Perfi of two canoes. Thank you, Nate Sinal, will O’Neill, Seb Nash, the folks at Command Control Power, Steven Weinstein, chat, Swarthout, Daniel McLaughlin, Justin Holt, bill Smith and Weldon Dodd. Thank you all so much and remember that you can back us if you just head out to patreon.com/m ADM podcast. Thanks everybody. Charles, we’ve got a bonus question this week, and I thought you phrased this brilliantly, so I’m going to let you ask

Speaker 3 (01:00:22):
Away here. So for the listeners as we go, one of us will type up a bonus question. So this one is getting at the heart of declarative quote, because I find that we talk about declarative management as though it’s something crazy. Instead of just applying a new programming paradigm that’s actually not that new to existing stacks, but in the real world, we can take a declarative approach. We can’t take a declarative approach to everything. So for example, there’s a warehouse. We have widgets in that warehouse that we want to go to three other warehouses. We can’t teleport widgets. And I know that we could map it to tell a human to go load a thing on a truck and then move the thing, but that doesn’t make this question very fun. So putting that to the side, what in your life would you love to make declarative that isn’t today? I see Eric laughing and that makes me happy.

Speaker 2 (01:01:31):
Yes,

Speaker 5 (01:01:32):
That’s a very good question. I’m trying to think of a good example.

Speaker 3 (01:01:39):
How about I kick this one off getting the kids to school. I wish I could just have a chase on file that I drag over into GitHub that makes the kids fed, closed,

Speaker 2 (01:01:57):
Packed up correctly with

Speaker 3 (01:01:58):
Their lunch in their hands, makes their lunch, and then has them at school, and then I can go about my day. That would be quite lovely. So there’s mine.

Speaker 2 (01:02:12):
I think that’s solid. That’s a great one.

Speaker 5 (01:02:14):
I think I have one. I would love the tech rider of my band to be a declarative management for the venue where I’m playing.

Speaker 2 (01:02:24):
Oh man. I was going to say then you don’t even have to worry about the green M and msms thing or the Brown M and msms thing. I feel like that’s exactly right. Anybody who’s ever heard this story understands this viscerally, which is to say every venue is different and every venue has different inventory. Every venue has different capabilities and functionalities, and so that’s why you write a tech writer. And so you might say, I need this amount of space, I need this much stuff. Some of it’s a competence check that you just put in the tech writer, and that’s where the whole Brown M and MSM thing came from, which is to say, Hey, we’re doing this show with a whole bunch of pyrotechnics. If they leave the brown M and MSM in the bowl, maybe they didn’t read the section on the fire extinguishers or those kinds of things that are associated with the fires that we’re about to light off. So I love that. I think that’s a solid choice.

Speaker 3 (01:03:25):
Sometimes it’s not. Just to make sure that they read it though. Like Charlie Daniels played at my fraternity house when I was social chairman, so I was responsible for paying him and making sure that all the writers were there and a few bottles, 1.75 liter bottles to be specific of whiskey was it was checked, this is here, and then it was consumed during the show. Anyways,

Speaker 4 (01:04:04):
I have a very simple one. I’m just looking right hand here. Whereas analog synthesizer, eurorack module, and of course dangling cables and I love it however, so sometimes I take a photo, but once you rip out the cables, it’s gone. Yeah. So having that little kind of magic going on the declarative, storing it away and restoring it, but keep it as a way as it’s like analog and not a computer or some have a computer inside.

Speaker 3 (01:04:41):
Yeah, I have to say, I think we all moved from the Sure SMM Sevens to the,

Speaker 2 (01:04:52):
Was it MV seven?

Speaker 3 (01:04:54):
MV seven, yeah. And getting rid of the X L R cables, the breakout boxes, the cloud lifters from my desk to record this podcast was so it made me so happy

Speaker 2 (01:05:08):
And

Speaker 3 (01:05:09):
That no more. So I guess we did somewhat take a declarative approach. Now I just plug in A U S B C cable and I’m recording. Obviously I don’t get the awesomeness that is analog synthesizers.

Speaker 2 (01:05:25):
I think those are both really solid spots. I’m going to go straight up product manager for a minute, and I’m just going to say my roadmap, I was going to say, if I could just merely by the proper documentation of j ss o n files arrange for certain things to have been built by specific times. Oh man, that would just be all the

Speaker 3 (01:05:49):
Dependencies. Are there nothing, we don’t have to refactor this piece before we do that piece. It just

Speaker 2 (01:05:59):
Blows. Oh, the dream.

Speaker 3 (01:06:01):
If that were possible, there would be no product managers.

Speaker 2 (01:06:04):
Well, and here’s the thing, if a Jira plan could just be something that got built into life, I think that software development would be a very, very different space because the artifice is in the planning, not in the execution. And we all know that that is the opposite of true,

Speaker 3 (01:06:24):
Truly. Yeah. And it was written about in the sixties with the Mythical Man month. I mean, there’s nothing new.

Speaker 2 (01:06:37):
There are just lessons we have to learn over and over again.

Speaker 3 (01:06:40):
But he even says that in his book. This lesson has to be learned repeatedly

Speaker 2 (01:06:47):
And every Learn it until you learn it. That’s exactly right. But until next time, we all have things to learn. So I will encourage folks to go out and take a look more at Centra. We’re going to have a lot of really great resources here in the show notes for this episode. And folks can go and read more about the Terraform provider that we talked a little bit about this week, as well as all of the open source documentation for Ttra, as well as a couple of really, really great presentations from this year and from prior years associated with Mac DevOps. And so that is fantastic. And of course I will get to see Henry and Eric both next month in Sweden, and I’m really looking forward to it. So lots of really great stuff for us all to think about and talk about.

Speaker 3 (01:07:32):
And we’ll be recording in Sweden.

Speaker 2 (01:07:34):
We will be. I was going to say, we will have a live episode there to end the conference and it will be Charles, Emily, and myself. So that’s going to be a really spectacular time. Thank you all for listening to another wonderful hour of the Maced Men’s podcast. It’s been a pleasure to have you both, Henry and Eric, thank you so much for joining us. Thank you for, it was great, met you guys. Of course. Thanks so much to our wonderful sponsors for this week. That’s our friends at Kaji and Collide. And thanks everybody. We’ll see you next time.

Speaker 3 (01:08:09):
And I get to say, see you next time. Last, because Marcus isn’t here waiting for me to say it so he can say it. So see you next

Speaker 2 (01:08:15):
Time. I love

Speaker 6 (01:08:17):
It.

Speaker 2 (01:08:28):
The MCAD Men’s podcast is a production of Mcad Admin’s podcast, L L C. Our producer is Tom Bridge. Our sound editor and mixing engineer is James Smith. Our theme music was produced by Adam Coga the first time he opened. GarageBand sponsorship for the Mac Admins podcast is provided by the mac admins.org Slack, where you can join thousands of MCAD admins in a free Slack instance. Visit mcad admins.org and also by illusionary L L C. Technically we can help. For more information about this podcast and other broadcasts like it, please visit podcast dot mac admins.org. Since we’ve converted this podcast to A P F S, the funny metadata joke is at the end, I.