Jeremy Tangren is a Technical Program Manager specializing in infrastructure and vendor management. Throughout his 15+ years in IT program/project management, he has managed local- and global-scale multi-million dollar projects at companies like Facebook, Splunk, and Cisco. Jeremy is based in San Francisco, CA.

Mike: This is the Real World DevOps podcast and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools to the organizers of amazing conferences or the authors of great books to fantastic public speakers I want to introduce you to the most interesting people I can find.

This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where Influx comes in. Personally, I'm a huge fan of their products and I often recommend them to my own clients. You're probably familiar with their time series database influxDB, but you may not be as familiar with their other tools. Telegraf for metrics collection from Systems, Chronograf for visualization and Kapacitor for real-time streaming. All of these are available as open source and as a hosted SaaS solution. You can check all of it out at influxdata.com. My thanks to InfluxData for helping make this podcast possible.

Hi, folks, I'm Mike Julian, your hosts for the Real World DevOps podcast. My guest this week is Jeremy Tangren. He's an expert on everyone's favorite topic to completely ignore, vendor management. He's been a technical program manager for companies such as Cisco and Facebook, Splunk, and a whole bunch of other really interesting places. Most interesting to me is, Jeremy, you and I have known each other for God what? Nineteen years now?

Jeremy: Yeah, 19 going on 20 it's been a while.

Mike: It's insane. Yeah, which is weird. We met each other on a Final Fantasy Forum when we were like 12.

Jeremy: Thank you Final Fantasy Seven.

Mike: Yes. Thank you. So you've been working a lot with vendors throughout most of your career and I know you kind of fell into it accidentally.

Jeremy: Yeah, totally accidentally. It started when I was just a general IT guy and I was by myself. I needed more hands and more capabilities and I had to start bringing in vendors to get work done. It was a necessity.

Mike: Now, I imagine that most people listening to this thinking like, "Oh, my vendors, my vendors are awful," and I mean vendors are kind of annoying to work with aren't they?

Jeremy: They can be challenging. I definitely get the perception that most folks view vendors as a necessary evil and they would avoid interacting with them at all costs if they could.

Mike: That's been my experience as well. But I mean, there have definitely been a lot of vendors where I'm like, "I don't really want to work with them, but they get the job done that I need done and I don't want to do it myself." But at the same time, it doesn't really need to be that way.

Jeremy: No, not at all. You don't have to have an adversarial relationship with your vendors. In fact, you should be looking for more of a partnership than anything else with a vendor.

Mike: Tell me more about that. What do you mean by this partnership?

Jeremy: Well, take for example, when going back to my original IT guy explanation, I was by myself, but I had to have help. So when I brought in a vendor to install fiber or whatever it was, I had to work very closely with them to tell them where the work needed to be done, what my expectations were, all of the major details of the project, and then I had to again work closely with them during deployment and then finally on close of the project. At no point during that process was there an opportunity for me to step away from them and just let them run and expect things to go hunky-dory, and that's what most people tend to expect with vendors is that, "I pay them, I cut them a check, they get the job done. I don't think about it anymore," but there's just so much more than just cutting a check that goes into working with a vendor.

Mike: Okay. Let's dig into that. What else is there? Well, I mean when I have a problem I'm thinking, "I'm going to go find someone who is very good at this problem, some company, and I'm going to tell them what I need, cut a check and walk away." It's like hiring an electrician or a plumber. They're just going to get the job done. I don't have to think about it. So is there more to it than that? What else goes into it?

Jeremy: There's always more to it than that. When you're bringing in a vendor, they're specialists at whatever they do, the electrician, the plumber, but they still have to be told what it is that needs to be done. "Do you need to wire this entire house with three-phase power?" "Do you need to only fix this toilet?" They need to be given clear instructions and expectations or else they're just going to do what they think is right and that might not align with what your expectations are, so you have to work closely with them to get the right results.

Mike: I imagine that gets pretty complicated when we're talking about a software and IT. I can just imagine hiring a data comm person or data comm company to come in and wire a building for Ethernet and how complicated that would be.

Jeremy: Absolutely, you can't just bring in that cabling vendor and say, "Go." They don't know where all of the gotchas are in that building. They need a blueprint of the building. They need to be working with the facilities team. There're lots of cross-functional stuff that goes on just to wire a building with cables. So you have to have that partnership with the vendor and not just expect them to cowboy it up and take care of everything magically or again, you'll end up with results that you weren't expecting or didn't desire.

Mike: So on that note, the idea of hiring a vendor, just letting them loose. It seems like the thing that everyone does, and on that same note, people just don't think about their vendors, especially ones they're using long-term. I know for a long time I didn't until you told me how wrong that was, so why are we doing that? Why are we just kind of ignoring our vendors?

Jeremy: Because it's easy to do. I think that's what it really boils down to is, especially from an engineering standpoint, it's very easy to focus on what you're working on and what you need to get done versus the dependencies and interactions with these other teams, and vendors included. So it's very easy for an engineer or even a manager to forget that they even are working with a vendor. And when a vendor comes knocking, they go, "Hey, we're doing still doing x, y, z. Is it to your satisfaction?" And manager or engineer goes, "Oh, no, that's not right actually. You didn't do it the way I wanted," because they weren't partnering with and working with the vendor to guide them to a successful conclusion.

Mike: Something you said there made me think about this one particular thing that is often overlooked in vendor relationships, which is the role of the client. So as when me, the company, is hiring a vendor, I expect certain things from them, but to make a good relationship, they actually expect certain things from me too beyond just payment. But it can be as simple as that. Right?

Jeremy: Yeah, it can be.

Mike: So I have contractors I work with from my company and I follow the Patrick McKenzie rule of net 30 minutes on payment, and invariably they're like, "Oh, my God, you're an amazing client. The best client I've ever had," is simply because I paid them early.

Jeremy: Yeah, that's actually, that's one of the biggest successes that I've found with vendors is when you establish these really solid relationships with them, even if you haven't paid them yet, they'll do work for you sometimes. You have to have that incredibly tight relationship and partnership either on a personal level or on a business level before the vendor will go above and beyond for you and do things like dedicate their entire engineering team to your project before you've cut a single check. So that's a really good underline of the relationship between the two entities.

Mike: I know that you've got vendors you've been working with for 10 plus years now.

Jeremy: Yeah.

Mike: For a lot of us, it's absolutely wild that we would have vendors that we continue to work with for that long.

Jeremy: Some would call it favoritism. I've definitely seen that in some businesses. There was a company I worked for a number of years ago that required that we acquire three quotes for every piece of equipment or every vendor that we worked with. And I understand the motivation behind that, but very often that's just overhead that you don't need. And it creates an adversarial relationship between the company and the vendor, because as the company, you are now trying to squeeze the vendor for everything they've got in order to compete against these other quotes, and that's not really the right way to do it.

Instead, if you've got a vendor that you trust and have worked with for many years, you know that you can rely on them to get your project done. Especially, given the interactions that you've had in the past, the relationships, partnerships that you've had, they trust you and you trust them. It goes both ways.

Mike: I have heard that you have some fantastic stories about how this actually has played out in practice.

Jeremy: Ah, just one or two.

Mike: So one of my favorite stories is a data center on a boat.

Jeremy: Yeah. So data centers on boats aren't generally a good idea. However, they happen.

Mike: Tell me more.

Jeremy: A number of years ago, a client brought me in to shore up their infrastructure and initially, the request was, "Hey, things are a bit unstable, we just need you to get things rolling again and bring up our uptime." So I came on site and I took a look at their server room after I asked to be escorted to it. So I was escorted through the boat downstairs into the engine room and then into another side room where all of the servers lived and I'm looking around in this small server room and things are functioning. It's a server room. And I asked a question and I said, "Where's the water line?" And they said, "Oh, it's about 10 feet over there," over there being above our heads.

"So what you're telling me is that your entire server room that operates your entire company and brings in all of your revenue is sitting here on a boat under the waterline?" "Yeah, that's about right." "Oh, okay. Is this a concern for anyone?" And the response I was met with is, "It's a little bit of a concern, but we check the hull every year. It should be fine." "The Pacific Ocean, the entire Pacific Ocean is less than 20 feet away from all of your servers. What do you think about moving them?" And, of course, the next question was, "Well, where? Where do these go? They're super important to us. We need all of this hardware, all these services, what are we supposed to do about it?" My first answer was, "Get it off of this boat."

Mike: Doesn't really matter after that, right?

Jeremy: Yeah, anywhere off of the boat is an improvement, and it required interfacing with multiple vendors and consultants to scope out all of the various dependencies and hurdles in moving all of this company's services elsewhere. We ended up migrating them over to Amazon. So now, no longer can the data center sink into the Pacific Ocean and they can have multi-region tendency and resiliency. So it's a good lesson where working with good vendors can save you from a terrible, terrible outcome.

Mike: Yeah, so how did all that work? I imagine you were not physically picking up these servers and running them to the nearest Amazon data center yourself?

Jeremy: Oh, no, no. I mean I do my squats and lifts every once in a while, but servers are a bit much.

Mike: So, you say you relied on a lot of vendors to get this done and a lot of people would be like, "I don't want to rely on a vendor for something that critical," but you did. So why is that? Why did you go that route instead of trying to do this internally? How did you find the vendors to do something so critical for the company? And how did you manage those relationships?

Jeremy: Well, the core answer to your question is that the client didn't have the expertise and competencies necessary to do this kind of a migration. They had the engineers necessary to keep the lights running and keep everything going, but not much beyond that. So we really needed people with expertise in migrating services from standalone data centers to the Amazon Cloud. And we needed someone to maintain these systems after they were migrated because again, this company didn't have the expertise in this area, so somebody had to do it for them. So we ended up hiring one consultant to architect the Amazon migration, another architect that I think actually performed it or another consultant that performed it. And then we hired an external support vendor to manage the entire infrastructure after it was deployed. And they still do that to this day.

Mike: So how did you find these people that were that good and that you could trust with such a migration and what was the relationship building like on that?

Jeremy: As with a lot of good vendor relations, I happen to have some good sources, suggestions from folks that I knew either to point me in the right direction for people that I didn't already know. For example, the vendor who was going to maintain all of the services after it's migrated, I didn't know who could do that right off the top of my head. So I had to reach out to my network and learn who would be reliable. And this is where it goes again, the experiences that people have with vendors vary wildly, and so you kind of have to take every story with a grain of salt when they tell you, "Oh, I had this terrible experience with such and such." So I was looking for good experiences with such and such to maintain their infrastructure and then continued to reach out to my network, looking for those consultants to actually perform the move.

Mike: Do you interview your vendors?

Jeremy: I do actually.

Mike: How do you interview a vendor?

Jeremy: Not entirely dissimilarly from how you would interview an employee. You have to think about the long-term with a vendor, and what the good and the bad with the vendor could be. So you have to plan for things to go well. You have to plan for things to go sideways. You have to plan for things to run for maybe a year that this will take and will be involved with this vendor. So you have to really take these into consideration and talk with somebody in leadership, preferably in the C-level, the C-Suite, and not a salesperson. Every bad experience that I've ever had with a vendor has been because of a salesperson. Every time I've had a good experience, it's been because I engaged the CIO or the CEO or whoever at the top level and said, "This is what's going on. These are incredibly important requirements and we need this done right."

When you speak to the right players who understand what your needs are, they will make sure you're taken care of. Salespeople, not to discredit the entire realm of them but a lot of times salespeople will say things that aren't true or will promise things that can't be done. And I've had this happen to me, so I always follow-up whatever a sales guy says with their C-Suite. So yeah, definitely get an idea of how they function and maybe who their other clients are and maybe talk to some of their clients and see what their experiences have been. And again, with that grain of salt, understanding that maybe things weren't managed right over there, maybe they were and the vendor was bad. So there's a whole lot of evaluation that goes into, "We're bringing in a new vendor," and not just simply signing a check.

Mike: You mentioned this idea of planning for things to go wrong with a vendor and that seems kind of counterintuitive. When I'm thinking about working with the new vendor, I really don't want to be planning for things to go south, but you do. Why is that?

Jeremy: Well, I'm a PM for starters, a large part of my job is managing risk. The thing is is even though you may have established this partnership with this vendor and everyone participating in this project or whatever it is, has the very best of intentions. The company has the best of intentions. The vendor has the best of intentions and everyone is trying to stay on the same page, but at the end of the day, the vendor is not your employee. They don't have the same business goals, they don't have the same business direction and somebody could change their direction in a way that affects the project unknowingly and causes things to go sideways on you.

And this is really true for any project, vendor or not. You should always plan for a project to go sideways or just explode in your face and how you'll handle that failure. A good vendor if you're working with them and there are large risks ahead, you should be planning with them on how to address those risks or at least have mitigation plans should something go wrong. And again, a good vendor, will be happy to work on those plans with you that they will support that and say, "Yes, we're going to do what it takes to make this successful." A bad vendor will just wave you away and say, "No, we've done this a thousand times, it's fine," and I've had exactly that happen and had a bad experience.

Mike: Going back to partnering with vendors, there's been a clip floating around for a while now about if you're spending several million dollars a year with AWS, there are no longer a vendor they're a partner and you should treat them as such. The relationship is different. It is different than just the person I pay to mow my yard. So you were talking with me before we started recording that while that is true, the size doesn't actually matter.

Jeremy: That's absolutely correct. AWS, they're giant. I mean absolutely tremendous and yeah, you definitely want to consider them a partner if they are where all of your infrastructure lives, for example, it's super, incredibly important to your business. They're not simply a vendor at that point, but even smaller vendors, way, way smaller vendors can have that impact to your business and that partnership with your business. A key example, I was working with a company a couple of years back. They had engaged a very major vendor to do a deployment for them and that deployment was-

Mike: So we're clear, what do you mean by deployment? Are we talking like software? Hardware?

Jeremy: This was a hardware deployment.

Mike: All right.

Jeremy: A physical deployment into a number of offices and they vastly overspent on this deployment because the relationship between the client and the vendor at that time were, "I'm going to cut you a check and you're going to do this deployment," and the results were basically a completely failed deployment. Many, many locations that were deployed to the systems did not work at all. And the locations that it did work, the experience was so far below par that it was almost unusable.

Mike: My first inclination when you tell me that is that the vendor's to blame, but I feel like you're about to tell me that wasn't true.

Jeremy: Ha. You know exactly what I'm thinking. Yes. You could very easily say, "Yeah, it was the vendor's fault." They overspent on every single thing because they wanted the money, they wanted the cut of the profit. Or you can think about what the situation was that led up to this and that situation was that the specific team that was responsible for these services engaged the vendor, gave them a very high-level scope, very basic and basically, cut them a blank check and said "go" -- expecting them to take care of the entire project end to end with no project management. No check-ins just go do.

Mike: That sounds like they essentially outsourced both authority and responsibility, but were expecting that everything would be perfect when it was done.

Jeremy: You're absolutely correct. Sadly, the result of that was me coming in with my team, after the fact, and yanking out millions and millions of dollars’ worth of hardware that was completely useless and replacing it with all new stuff through a new vendor. And this vendor was much, much smaller. We're talking about the difference between say Accenture and your five-person consulting company. So we're now dealing with this five-person or so company and I'm working with them directly, very tightly. Establishing what it is that we expect. The level of the user experience. I dictated a large portion of the scope for these deployments and so we went forward with the project. They got their check, the equipment came in, we did the deployments and users came in that following Monday and were ecstatic that all their systems worked.

Mike: Nice. It sounds really what went into that was there was a lot of up-front planning of working with that vendor to set expectations and who is responsible for what. It sounds like a lot of actually ongoing management.

Jeremy: Yes, you're absolutely right. It deviates from the idea of, "I'm cutting a check and walking away." So far it's 180 degrees different from that. It is now, "Someone else is cutting you a check somewhere down the line, but you and I, you and I vendor are working hand in hand to get this project done. I don't care who's cutting the check. We're in this together. As far as I'm concerned, we are on the same team during the duration of this project."

Mike: And that's really what you mean by partnership with a vendor.

Jeremy: Yes, absolutely.

Mike: In the State of DevOps Report they actually, Dr. Forsgren actually talks about this, that when you are outsourcing functional things, it's actually very tightly correlated with a low performing company and what you've done is though you have outsourced the work, you haven't actually outsourced responsibility and you're still actually treating the company you've outsourced the job to, as part of the team, though they legally speaking are not functionally they are.

Jeremy: Correct. In this case with that vendor, they called into my daily stand-ups every day during the course of the project.

Mike: That's wild to think like, I'm going to have my vendors on my stand-ups.

Jeremy: Yeah. No one else would ever do that. But it was critical to the success of the project because they needed to know what was going on just as much as the rest of us.

Mike: So on that note, there's actually, having been in really large companies myself, there's this concept of you have internal customers too and everything you're talking to me about here seems to say that there are situations where you could actually be the vendor and also have vendors of your own.

Jeremy: You're absolutely right. I've been in a couple of different positions where you have entire service teams or infrastructure teams within companies that are vendors to the rest of the company. Maybe there's a cross charged system or contracts or whatever there may be, but ultimately these teams are vendors to the rest of the company providing them services.

Mike: How do the relationships change on that?

Jeremy: The relationships, honestly I feel should be very similar. If you're having good relationships with your external vendors, then you should be able to have a good relationship with your internal vendor. If it's the other way around, then you need to change how you're approaching one or the other set of vendors.

Mike: Yeah, so on that note, I have one last question for you have of let's say that I now realize that my relationships with the vendors I support or the vendors I work with are not very good. What can I do about that? How can I turn that around and start to improve their relationships?

Jeremy: First I would try and define what good looks like for you because you need an understanding of what good and bad look like for your vendor relationship and your circumstance.

Mike: Do you have some suggestions for what that might be?

Jeremy: Let's say, for example, you have a SaaS service and you're planning to scale. You're expecting to grow in the next six, 12 months and you reach out to your vendor and maybe they don't respond to your email for a couple of days. Okay. Email them again. Okay, you're not hearing back from them. Okay, so this now sounds like kind of a not super clean vendor relationship. How do you fix that?

Mike: I've definitely had those.

Jeremy: Yes. I'm sure a lot of folks have those, and really one of the things that you have to have is empathy. You have to understand that whoever this individual is that you're reaching out to, they are servicing other clients as well. Almost 100% of the time they have more than just you on their plate and they're trying to make everyone happy. That they're trying to do a good job. So don't take your first of couple emails that have not been responded to and get angry at your vendor because things happen. The next best thing that you should do is call them. Pick up the phone.

Every time I've ever picked up the phone and spoken to my vendor, be it one of the engineers who's working on the project, the project manager for it, the manager over those folks, or even one of the C-Suite. It doesn't matter. If I get someone on the phone and explain to them what is going on and how I'm trying to work with them, they will respond. They always answer the phone. They always call back. It gives them that impetus of someone needs me, whereas email doesn't have that urgency.

So that would be the first place that I start is if I have vendors who are a little less responsive, be proactive, engage them and get them to be responsive. If your account manager is still unresponsive, not responding to however many emails, not responding to calls, maybe you need to escalate it above them because again, you're working with a vendor, not an individual. So just as you would inside your own company when you're not getting engagement from an individual, you escalate up the ladder. And what you'll find is as you go further up the ladder is people who care more about the vendor-client relationship.

It's very similar to within any other company. As you move up the ladder, the people who work their care much, much more about the vision and the strategy. So engage those higher level folks if you need to and they will give you the attention that you need.

Mike: That's all fantastic advice. Well, thank you so much for coming on the show. This has been wonderful.

Jeremy: Well, thank you for having me, Mike. This has been a great chat.

Mike: Where can people find out more about you and your work?

Jeremy: Have my LinkedIn, jeremytangren on LinkedIn, and I'm open for hire for folks who need me.

Mike: Awesome. Yeah. On that note, Jeremy's a fantastic program manager. As of today, he's looking for a new company to join and kick some assets, so if you're looking for him, there you go. And to everyone else listening, thank you for listening to the Real World DevOps podcast. If you want to stay up to date on the latest episodes, you can find us at realworlddevops.com and on iTunes, Google Play, or wherever it is you get your podcasts. I'll see you in the next episode.