Build vs Buy: When Puppet's customer education outgrows the home-grown virtual lab

In this episode of Instruqt Creators Connect, Sean Carolan sits down with two members of the Puppet education team: Alex Williamson, Principal Training Solutions Engineer, and Greg Larkin, Senior Principal Field Solutions Developer, to discuss why they migrated from their home-grown labs to Instruqt and how it increases the productivity of their education team. Alex and Greg explain the motivation behind choosing Instruqt and how they have used the platform to build an engaging learning experience.Puppet by Perforce delivers great software that promises peace of mind and time. Scale infrastructure automation the right way so your team can innovate rapidly. With Puppet, you can find anomalies at scale, take pre-defined action to fix them, report the correction, and predict future events.

TL;DR

• Puppet uses hands-on labs for customer training workshops and on-demand training
• Homegrown labs lead to mediocre lab features, high costs, and large technical debt.
• In building home-grown labs, they  spend more time on maintaining their  labs rather than creating compelling educational content
• Puppet recommends that you should look for a lab solution  that has flexible browser-based  labs and amazing support to help you on the journey to building engaging hands-on labs
• Puppet saves hundreds of thousands of dollars on cloud costs with lab automation.
• Puppet does more with less FTEs using Instruqt.
• Puppet offers on-demand Kubernetes, temporary cloud accounts for learners, and automatic clean-up of environments

The journey of building home-grown labs

Greg Larkin, Senior Principal Field Solutions Developer, started as a professional services engineer at Puppet eight years ago. He spent half his time teaching in-person and the other half going to customers, implementing Puppet, and helping them get ramped up. Greg walks us through Puppet's transition from stitching together home-grown hands-on lab solutions to purchasing a full feature-ready lab solution.

In-person, networked classrooms (Hubs, switches, cables, and DHCP)

Physical setup

When I started at Puppet, one of the first things they gave me was a physical professional service engineer tool kit. This is a collection of hubs, cables, and all kinds of presenting tools. Back then, what we would do was we would go to a company or a micro center type training facility with our laptop, which had the Puppet software installment of the server software.

The process of setting up virtual machines

All of our students, before the class, were given instructions to download a roughly two-gigabyte virtual machine. They had to set up the virtualization, deploy the agent software, and prepare everything before class. Then we would get to class and network ourselves together. This was incredibly painful. Not many people did the preparation, which makes sense. They have their jobs. They don't have time to be doing this. We get into the class, and sometimes we burn half to three-quarters of the first day of a three-day course with a ton of material just trying to get our machines connected and talking to each other. Super frustrating.

Building home-grown virtual labs called Hydra

After 2017, we started migrating some of this to a virtual setup on AWS. We were still doing in-person training, but they were beginning to use the virtual platform. Then the pandemic. So we went to all virtual. We were experimenting with different ways of launching classrooms on AWS. Through that time, we started building more and more custom tooling. We built this internally coded tool named Hydra.

The switch to buying a scalable virtual hands-on lab solution

With Instruqt, we have roughly 18 to 20 tracks available for free right now on our education site. You can just go in. We've opened the door, so anyone can experience Puppet without talking with a sales guy. You can launch any of them, and they're set up sequentially, mapping the exact sequence you see in our instructor-led training.This lowers the barrier for people to try things and not feel like "I'm just going to get tons of sales calls out of the blue", and you don't have somebody trying to convince me to do something. I want to do it on my time, and we want them to have that feeling and ability."

The problems with a home-grown lab environment

The task of building your own home-grown lab always starts small and innocent. Until your team wants more people using your duck-taped lab solution and some fancy features to build on top of your side project. When the solutions engineers and developers at Puppet took on building Hydra, their home-grown labs, they realized that their DIY labs were only great in theory. The reality was that their home-grown lab solution was creating more problems than helping their teams.

Messy code and maintenance issues

Hydra was a collection of TerraForm and Python and YAML and all manner of other types of formats and languages that eventually grew to about 17,000 lines of this scaffolding code. This code was simply there to spin off the classroom. These lines are not the amount of content we had or the actual lab instructions. This is just scaffolding that we were trying to get things set up. We would constantly have maintenance issues and other types of problems popping up.

Large technical debt

It got hard to keep track of the environments. Because the code had outlived some of its original designers, keeping it alive was expensive from a cognition standpoint — understanding what the code did and coming in as a new engineer trying to take care of it. This is a story you hear all the time. The code outlives its original designers, and someone comes in to take care of it. As a result, this has cognitive as well as monetary costs. That’s when we decided to look at it from a new perspective and started asking ourselves:

• How do we fix this?
• How do we make this run a little bit better?
• Can we extend it to do some of these other labs?

Setting aside extra time for building features

We knew we wanted to build for other products eventually. Fancy additional feature sets that we wanted to demo. But how to do it in the original tool escaped us when we tried to dive into the source code.

Expectation vs. reality: building home-grown labs

There's a mismatch between what the world sees as work going into home-grown environments and the total work hours that actually go into this project.You build one great lab environment, and your colleagues right away ask, "Oh, that was a great course. Can you just whip out another one?"But here is the reality of the situation.

Product content is most important — but your mind is on maintenance

The reality is that it usually takes one whole dedicated week of eight hours to build something compelling. A product where we're showing you a whole other set of technologies and concepts we'll teach. Don't forget. This work is entirely different from what's hosting it. Over its lifetime, a handful of developers spent too many hours just maintaining it and not adding value to our courseware development. We started to focus on how we could minimize the maintenance time. We don't need to be spending time maintaining this thing that is very brittle and has a lot of issues. We need to be working on content and training people, getting them excited about using Puppet.

Building and maintaining environments takes too much time

The biggest risk is the time we need to build new features and maintain a stable lab environment. Our job is to educate our customers, not to be system administrators.The most significant benefit of using a vendor is that it's much more comforting to have a separate organization to maintain the environment. This way, our focus is on building content to add value.I want someone I can call, email, or Slack and say, "Hey, like we're having this issue. Are you noticing it?" Somebody else might escalate things on your end when I run into a problem and provide us with a fix. There is a comfort in reaching out and talking to somebody and knowing it's being handled. The fix will happen quicker than us running through our home-grown code, trying to figure it out ourselves.

Escape the instability and chaos of home-grown labs

For Greg, the tipping point was when he started getting pager alerts at two in the morning because a classroom crashed in the UK. That was a red flag that started popping up, and it began to escalate. The moment of evaluation and reflectionWe also had management changes above us come in and evaluate our systems and look for ways to improve and streamline.At this moment, management empowered us to say, "Hey, like this isn't working well for us. We're spending a lot of time maintaining our home-grown lab." That's when management gave us the green light to go out and start evaluating platforms. We all recognized that it was time for a change. The original developers were no longer with us. We're in there debugging every day and not getting a lot of benefit from the labs. So we set aside some time and said, let's just rethink this and open our minds to using something else.

Puppet's requirements for a virtual lab solution

Greg and his team evaluated several products. They had high-level requirements for a virtual lab tool to improve our students' experience. Here's a list of their top requirements.

Flexible Sandbox

We've got Puppet Enterprise server software, Puppet Agent software, Git repository servers, and workstations. All these things comprise a development environment for learning Puppet. We needed the ability to have any number of machines spin up instantly. We also wanted the ability to define the labs any way we saw fit with different machine types and specifications. Instruqt allows you to do that. We've looked at other vendors that did not have that flexibility, which was a huge selling point for us with Instruqt.

Zero installation

We no longer want our learners to download two or three-gigabyte VMs. We also don't want to spend time trying to get people's laptops set up with virtualization. This was a terrible thing to have to deal with before. We were using either VMware Fusion or VirtualBox. A lot of corporate laptops were not very happy with those tools.There was always a scramble like, where can I find a laptop that doesn't lock down. So by having zero installation, it's given our labs' massive wind. We saw a considerable drop in the barrier to getting students working and successfully using Puppet tooling.

Browser-based labs

Browser-based labs are fantastic because we don't also have to worry about can somebody get through SSH to get access to AWS. We've seen that happen in the classrooms before.

A vendor that will be a partner and support us

Our experience with Instruqt has just been amazing. I feel like we've been on a journey over the past nine months and established a tight partnership. We've just enjoyed talking to you and working with all the engineers. We feel both organizations' openness has allowed us to propel forward much more quickly.

Full-feature-ready solution

We want to focus on our coding and other content development types. This way, we use our time on things that add value to our customer experience. We're no longer interested in building scaffolding. That's what Instruqt is an expert at accomplishing. We've entirely handed that over to them, and we trust them to do it.

Building advanced labs for multiple technologies and integrations

Puppet has a few very advanced and complex training labs, one called Puppet Enterprise 501. You'll find a GitLab instance in these labs, some hardware running Docker and Docker images, pre-deployed, etc. The entire day is focused on hitting the ground running and getting a continuous working delivery for Puppet enterprise installation set up. Alex, Principal Training Solutions Engineer, explains what technical features Puppet loves using in their training.

Pre-configured on-demand Kubernetes infrastructures for learners

It's incredible how we've reduced all that complexity into just such a quick start-up time. Manually setting up Kubernetes infrastructures could take up to 30 minutes, especially if you don't have Kubernetes already deployed. Now we can offer pre-configured on-demand Kubernetes infrastructures that spin up in minutes.

Windows workstation, AWS, CD for PE, Puppet Enterprise, and GitLab servers all in one place

In the past, this was something we made internally and had been being asked for a long time to show people using CD for BE because we want people to understand what it's doing right and not worry so much about having to set it up. Now we have a GitLab instance, and all these tokens are set up in one place. People can easily get started, and we can show them all the cool stuff our tooling can do quickly.

How long did it take the Puppet team to ramp up and see real value from your training?

Puppet's first initial launch was a year ago. The team decided to start small and expand. Greg began by looking at some opportunities for how he could build a couple of tracks, intro tracks, and things that are relatively simple and very easy for people to understand.

Building a lab for a simple tool

So we decided to start with another Puppet tool called Bolt. This is a standalone orchestration tool. We wanted to use that as a first application to demonstrate with Instruqt.It has a low threshold for installation. It runs a YML install if you're on a Linux machine and has this standalone application. So there's no Kubernetes cluster, Puppet Enterprise Software, or anything like that. It's straightforward.

With maintenance issues gone, we had time to start thinking about content

Everything was out of the box. We built a couple of tracks to teach people fundamental usage of Bolt, started working on that, and proved the concepts worked for us. As we did that, we had to make certain decisions.Do we want a single track for each day of classes with many challenges? Do we want to have individual tracks with single challenges?We chose the latter to have an extensive offering of tracks in our education. We just started with our day one training, which focused on getting Puppet Enterprise up and running. We had to go through many processes and set the parameters around development. After having that all in place, we just dove into the work.

Business Impact

Puppet's customer education department is reaping the benefits of switching from home-grown labs to Instruqt. The team now has increased productivity while having the numbers and facts to show the impact of their customer training on engagement in managers and stakeholder meetings.

Higher productivity, more content creators, more product content

Instruqt has been so empowering for us in the technical education department. Now, instructional designers, editors, and anyone in our larger organization are responsible for enablement and design work for our courses.Before this, we would have to go in and update various homegrowns and slide presentation tools. That works, but it wasn't as User-Friendly as some of the things we're doing now with Instruqt.People can create content using Instruqt's GUI interface on the website or the command-line interface.In the GUI interface, you've got an entirely visual editor that you can go in and make changes. Markdown is relatively easy to learn. It has a few different syntax decorators and things for bolding in italics, code formatting, etc. After a bit of practice, I think we're all comfortable with adding instructions in the UI.

Saving at least one FTE from maintaining labs

As a whole, I would say that we're saving easily a full-time employee's worth of time by not maintaining the home-grown solution anymore.

Fast ramp-up time for developers

Being the least tenured person on the team, I could still show up and, within days, start contributing to building labs in Instruqt. This is a typical story where the ramp-up time for a developer or an engineer to consume something custom is a cost. The time it takes to get a new developer productive. With Instruqt, I was productive within a day. I learned how to make changes and write a set of scripts. I all clicked because I didn't have to worry about things like,

• Where are these machines, and are they secure?
• How do people get to them?
• How do I get to them?

All the complications and worries were just gone. I'm not worried about SSH keys, RDP connections, or anything.

Saving on $96,000/year on cloud costs

In the past, it was common for the whole infrastructure phantom to be left running, and you don't even know about it. The lab is just sitting out there, and you haven't cleaned it.If you're smart, you've built some type of destroying process. In our case, it would often fail midway, and now you've got a bunch of instances, load balancers, and DNS records just hanging out in eight of the US consuming resources and costs. The only way you'll know about the costs is when the Finance Department sends you a message, "Wait a minute, why are we now spending $8,000 a month on this?" With an invoice attached. You've hit that limit where they notice, and that's how you figure this out. We don't have to worry about that anymore with Instruqt. We really like that.

Teams and departments are buying in on Instruqt

Using Instruqt for our training has completely changed our team's performance, and others are now jumping on Instruqt. Because this is now the fastest way in the company for people to see precisely how Puppets works, we've been asked by other internal stakeholders, "Hey, can I get access to that? I want to, I want to be this like look at these screens, and I don't want to take 2 hours to set it up myself".That's been a tremendous value.

Advice for New Content Creators on Instruqt

Start small with Instruqt's web UI

It's really easy to use just the web UI to get started with a basic track and see how easy it is to get a virtual machine that can do whatever you want in 30 seconds. From there, just keep adding, and you can take it a small step at a time and build these things over time with minimal effort.Just take something with a very simple installation. Try to cobble it together on your own, maybe do some commands on your workstation and then start moving things up into the Instruqt environment. Then you watch it and start adding some instructions and a check script. Take on each feature one by one. With time you'll be taking advantage of almost everything the platform has to offer, for example, tagging graphical API, the webhook interface, setting up scripts, checks, scripts, etc.The timeline for us hasn't been very long. I feel we're pretty far along with what we can do now. It's been a great learning journey for us.

Like what you read? Get inspired with Instruqt’s test drive

Let us show you how Instruqt enables you to develop premium hands-on product education quickly and increases the productivity of your technical education teams. 👉 Take Instruqt for a Test Drive

Show Notes

Alex Williamson LinkedIn Profile
Greg Larkin LinkedIn Profile
‍About Puppet
‍Try out some of Puppet Labs
‍What is Markdown