Full Transcript of Episode 02: The Problem of Bad Stats and a Closer Look at Pastor Attrition
Lizette: Podcasting from Nashville, Tennessee, this is Keep Asking the weekly podcast that helps you dive in a bit deeper and wider into the research providing insight into today’s church and culture.
Welcome back guys. Good to be here. Scott, how are things going? You have a Junior in high school?
Scott: I do.
Lizette: How’s that? Is the Junior year going well?
Scott: It’s going really well. We’ve got marching band going on right now and we’ll probably be doing a couple more college visits, which is a little crazy as a parent to be thinking about, but exciting.
Lizette: Alright. Now marching band competitions, are those more enjoyable than the freezing cold overnight Boy Scout camps that you did, used to have to do?
Scott: Well, sometimes the band competitions can be freezing cold, for 12 hours. At least it’s daytime and you’re not trying to sleep. So far, we’ve been fortunate, good weather and so one down and two to go. Yeah, fun stuff.
Lizette: So Casey, Mary Beth, she’s feeling okay? You guys, when, when did you say?
Casey: January 25th.
Lizette: January, and this is . . .
Casey: Elijah’s due.
Lizette: Elijah. Elijah! Now that’s interesting. Is that long for Eli?
Casey: Well, actually Eli Manning’s full name is Elisha but um, but yeah.
Casey: It was a part of the inspiration for sure, but we also just like the name, the biblical character.
Scott: Everybody gets those two confused anyway, so you’re fine.
Lizette: We’re not going to jump in today but one of these podcasts we’re going to address the fact that you’re a pro sports snob and you look down on college sports. We can’t really jump into that today. I do want to open it up for your rant of the day. One of the things that we’ve talked about is when there’s bad data out there and that confuses people and throws people off track on really important issues. What’s your beef with bad stats, guys?
Rant of the Day: Convenience Sampling (02:10)
Casey: Yeah. I mean, we, we’ve talked about it here before, the, your friends aren’t representative of a wider group, so sometimes when we see stats out there we want to know hey, who did you ask? How did you ask them? Where’d those come from? And a lot of times the answer to that is: they asked the people who were convenient to ask. And we call that convenience sampling. Sometimes that’s through their social media account or sometimes it’s through, their audience, maybe it’s through a newsletter.
Lizette: Well hold on. If your friends matter to you and it’s their opinions you want, isn’t that a good group to ask?
Casey: Yeah, it is as long as that’s who, afterward, you say that this is about. There’s still some issues there with voluntary response sampling. And if you ask people, the people who are most likely to say something back are the people who have a really strong opinion on it. If you kind of put it out there for anyone to respond to, sometimes what you’re getting back is the people who feel strongest about that—not a representative group—even among your friends. Particularly if you’re going to put it out there as somehow representing a wider audience, that’s when you can get into real trouble.
Scott: The first thing that comes to mind when I hear convenience is like a convenience store. And if you’re hungry, sure. You can run in and grab something to eat. It may not, actually most of the time, it is not that tasty. I mean, it, it’s not a real good meal.
Lizette: I disagree. I love taquitos from gas stations and corn dogs from gas stations.
Scott: Or the hamburger that’s in the shape of a hot dog. I don’t even know what they call those things.
Lizette: Yes. No, those are gross. But taquitos and corn dogs from gas stations are exquisite.
Scott: …yeah. I was driving down the interstate the other day and on the restaurant sign, those blue restaurant signs, it was, I think it was a taco place. It started out with the name Citgo. And I was like you just spoiled the whole thing. You probably could’ve gotten me to come in and try your tacos. When it starts off with gas station as the setting, that just doesn’t sound right.
Because a convenience sample is not the setting you actually want to describe. It’s the people you can get easily. And so if you’re at an event and you ask the people at that event to answer some questions, that can be helpful for the conversation you have in that room. As soon as you go to another setting, which would be a conversation where you’re talking about churches across the nation or you’re talking about adults or men in America, those conversations are completely different. It’s a different setting. And so you shouldn’t be using that convenience sample to talk about this bigger group of people. And the people you want to describe, we talk about them as the universe. The universe you want to describe and the sample you go to get needs to be describing that universe. That means the people need to be a part of that universe and need to be randomly picked, not picked because they happen to be the first people you see or the people you have in your address book.
Lizette: “Universe” throws me because it felt like you were about to talk about science for a moment and science and math are hard for me.
One of my pet peeves is when I see folks – and this happens in ministry. I’m not dogging on people in ministry except people who do this – but they interview, they survey the people on their newsletter list and then they say it represents all pastors or say it represents all church members or all females. The universe there, when you’re saying it’s that whole population of pastors and it, their sample is messing it up because there’s bias and the people they’re surveying already like them. They probably already like their product. They probably share some similar views or something like that?
Scott: Probably the first time you read a newsletter or read a magazine, that question crosses your mind. Like who else is actually reading this magazine? And so you might want to know that. But, you never ask that again. And so why would you look at research that is only from that magazine or only from the subscribers to a newsletter? It’s not the universe. It’s not the picture that you want to be taking a snapshot of. It’s an anthill instead of a beautiful mountain range is kind of the comparison. It’s not the right picture
Lizette: And that, it’s not the right picture for the reader but it could be helpful to the people in that ministry or the people in that magazine because they can learn because their universe that, is that the best word to use? The universe is we want to hear from, a random group of our listeners or our subscribers or our customers. That’s when it works.
Casey: Yeah. I think basically anytime you’re doing research on any level, you just want to have a well-defined population. You want to know who are we trying to get at? And if your sample, like Scott said, isn’t randomly coming from that population, whether it be your audience or whether it be Americans or church goers or whoever, your results are basically meaningless. And I’m a statistician and I can do a lot of interesting analytics on things. But I’m not a magician so I can’t make the report .
Lizette: But you can juggle.
Casey: Right. That’s true. But I can’t make your poorly collected data mean something. There’s nothing a good statistician can do with poorly collected data except an autopsy of what went wrong.
Lizette: One of the areas I’ve heard you all talk about around the office is bad stats or bad reported stats that have been out there about pastor attrition: why pastors leave or how many pastors are leaving and that prompted one of our fairly recent studies. Is that right?
Scott: It is. We’ve seen a lot of statistics. Honestly, a lot of times it’s somebody quoting somebody else who quoted somebody else who quoted somebody else. And it’s hard to get back to the original statistics. We actually had Dr. Richard Dawkins and the North American Mission Board approach LifeWay Research saying, we’re having trouble getting to the real statistics that keep being quoted. And from what we found, they don’t appear to be very reliable. And so as we dug into it, the research that most people were quoting said that many pastors:
- were leaving the ministry every year,
- were struggling with pornography,
- were struggling in their marriage,
- and were ready to quit.
We found that those were incredibly inflated because [the survey] took place at a conference and it happened to be a bunch of pastors who were needing help.
And so we were thrilled that conference was taking place and that they were helping people. As those stats were quoted and requoted, they were projected to be all pastors in America. Our project said let’s ask all pastors in America, no not literally every one of them, but a sample of them, a random sample of them, not biased by who’s at a conference or who’s on a list. let’s randomly select them.
We did narrow it down to not be all pastors. We said let’s get those who are in Evangelical denominations or traditionally Black Protestant denominations and so among them we took a random sample. What we found is that about one percent of pastors leave the ministry each year prior to retirement. That is regrettable. That is [a number] we would love to see even drop further. It’s a much smaller number than has been purported by those who have been, frankly, guessing based on some convenience samples.
Lizette: If we can put a little parentheses in here, if everyone could visualize the parentheses I’m making with my hands here in the air.
You’ve used the phrase “random sample.” The word random is a little unsettling because it seems like that disqualifies the sample.
Scott: That’s really true. Random sounds bad and convenience kind of sounds kind of good, doesn’t it?
Lizette: Yeah. ‘Cause I’m a big fan of convenience.
Casey: Yeah. I think to try and frame it a little more: when you think about random, that means that everybody has a chance to be in the study. It’s not just the people around you. If you’re a pastor listening to this, you could’ve gotten a phone call study because you were a part of a large list of Evangelical or Black Protestant denominations. If everyone has an equal chance of being involved in the sample, that gives us a more reliable and representative sample at the end of it to be able to analyze and make conclusions from.
Lizette: Sometimes I read books that simplify statistics for simple people because I have to talk to people who know statistics at a complicated level.
Scott: I saw a book just yesterday that explains statistics using comics.
Lizette: I have that, I have that! Did you see it at my desk? I have a comic book! It’s at my desk! I will, I will put a picture in the show notes. I have, so you look stunned. I can’t believe after eight and a half years of working together that would surprise you.
Scott: I was going to be mocking of the person holding the book.
Lizette: Go ahead!
Scott: I’ll stop. That’s alright.
Lizette: Feel, feel free. Go ahead. Mock away. Anyway, I’ll give you a second to recover, but I own that book. Casey, I’ve already told you I own that book. Can you back me up there?
Casey: Uh huh, she does.
Lizette: In one thing I’m studying, it compared random [vs convenience] sample to [going] out on the streets of New York. If you just started catching the height of people walking by, a hundred people walking by, you would probably get a nice random variation of tall and short as long as there wasn’t anything else significantly influencing that. You’d get short, tall, and you’d land probably pretty close to the average. The difference was if you went out and you started to count people and it was basketball players, pro basketball players exiting a game, from both teams. If Kristaps Porzingis, did I say that right?
Casey: Yeah. Sounds good.
Lizette: And his fellow players were doing that, then what you’ve got is a convenience sample. And it would not be trustworthy to say this is the average height of people in America.
Casey: That’s correct. And the example I had jotted down here basically sampling from a burnout conference to gauge pastor burnout is like going to Comic Con to find out if Americans like super heroes. You know, it’s a . . .
Lizette: That’s fantastic! That is so much better than me working in Kristaps Porzingis.
Casey: I totally appreciated that.
Lizette: Alright. Since we’re talking about it, what’s some of the clarity that we got on the pastor protection study?
Scott: There were several warning signs that the research identified that can really signal when a pastor might be in trouble. We began with qualitative research where experts kind of guided us toward the topics we needed to cover in the research. And these experts were people who work with pastors who are leaving the ministry, who are thinking about leaving the ministry. Some of those broad categories were:
- conflict in the church,
- issues in their own family,
- moral lapses and
- lack of preparation or a poor fit in their church.
And some of the warning signs that surfaced were that over half of pastors indicate that they’re concerned about the financial security of their family. And honestly, that doesn’t land in the family category. It lands in the conflict category. And that’s because if you’re always worried about your own finances, you’re going to be making decisions a little more defensively in that church setting. And so, when a little bit of conflict comes up, you’re going to be looking to get out of that conflict because you don’t want to risk your job and your family’s welfare because you really need that job. That’s one warning sign that that a lot of pastors are sitting in the middle of just because the pastoral role is not the best paid job in America. Newsflash. And that can actually be a detrimental thing when it comes to interpersonal relations.
We also see that consistently protecting time with your family is a very important thing. Now thankfully nine out of ten pastors say that they’re doing well in that area. Even 10 percent of pastors struggling with that and when we actually surveyed it – in the final phase of the research, pastors who had left the ministry – we see that that was an area where they failed and that was an area where they struggled.
And so it can be a warning sign right there if you’re not protecting that time with your family.
When it comes to burnout, we see that one out of five pastors say that my church had unrealistic expectations of me. And so, obviously there’s some preventative things that would help on that. When the people in the church and the pastor are not connecting on “So what am I here for? What’s my job? And what is the definition of success?” Very quickly it’s demoralizing. You’re always trying to meet those expectations, you’re physically and mentally going to be burning out.
Lizette: If we can back up just a little bit, we always refer to this as pastor protection. Was that designed from the very beginning—the looking for ways to protect pastors—was that the question or the problem to be solved that initiated the project?
Scott: Yes. Dr. Richard Dawkins, who approached us, has a background in working in more occupational settings saying: when we see that there are hazards on the job and we see employees getting hurt, let’s work backwards and see how we can prevent that. And so, he really brought that same mindset to this project to say, not only let’s see how many pastors are leaving, but let’s work backwards and say, find out why they’re leaving and how can we prevent that?
Lizette: You know, we’re not in charge of pastors and we can’t make churches do anything to fix this. Tell me, from a little bit of a personal note, what was rewarding for you working on this project or how do you feel like our research can possibly be helping? Is that a little too warm and fuzzy or touchy-feely?
Casey: No, I’m good with that.
Casey: One of the things we were able to do is correct some false narratives out there about where most pastors are. We see that a lot of pastors are not in that place where there’s a danger that, one thing goes wrong and they’re about to leave ministry. And correcting that narrative is encouraging because even for myself and for a lot of people out there, you can be not mistrusting the heart of your pastor, you know. I think we’re able to say here’s, pastors have real concerns and challenges and things that they’re facing. But, they also love ministry and love their churches and also love their spouses. I think that’s one of the things that really came out where we saw a really big contrast between some of the work of some of the bad studies out there and what we found. We saw that the life of a pastor’s spouse tends to be far more fulfilling than numbers from bad studies suggested. Some of those numbers, Scott mentioned over nine out of ten protect time with their family. Ninety-eight percent say their spouse is very satisfied in their marriage.
Lizette: Well, and we’ll, what we can do is we’ll put in the show notes some.
Lizette: . . . some charts. I realize it’s riveting to have us relay stats to y’all through the podcast. But, but I wanted, these are important. And so we’ll have some graphics and charts. And I’m actually advocating that we have a podcast later on that’s how to read a chart because some people, myself included who need cartoons, that can be helpful. It sounds like there’s some pretty compelling support out there for pastors–a lot of pastors–doing well. At the same time, it is acknowledging the reality that there are those who struggled and left.
Casey: And I think that even, that equips us to be able to, to minister to the needs of pastors now that we know what the actual needs are. You know, I think, as Scott mentioned the financial security aspect of that that’s an issue for, that a majority of pastors are facing then, that’s something we can sort of add more support and, come alongside of them. Some of the alarmist narratives would tell you that the main thing you might need is to have, I don’t know, daily family counseling for pastors, ‘cause their marriage is about to fall apart or, different things like that. And so yeah, being aware, having assessed the actual difficulties that many pastors are facing helps us to protect them.
Lizette: Who are some of the groups that can benefit by taking a look, a little bit deeper look? One group I thought of—seminary professors—simply by keeping these top of mind regardless of their particular subject matter.
Scott: We’re definitely reminded that pastoral ministry is a people business. It’s about people. And I think a lot of times in a seminary setting, you’re needing to learn a lot of facts and how those facts work together. And any time those courses can put those in the context of people’s lives, that’s going to help pastors apply it in their ministry later on because it’s all about people.
Lizette: We want to encourage everyone to look at some of the more of the detailed stats on this in the show notes. What are a few tips that we can give the listeners about when they’re just seeing research in general out there? How can they know if they’re getting close to some bad stats? If they’re getting close to someone not using it well? Are there any kind of quick tips or clues?
Casey: (1) I think one is can you answer: who did they ask? If you can’t find a primary source for “this is where the study came from,” I would take that with a grain of salt.
(2) If it sounds alarmist, again I think that’s one of those things.
(3) If it doesn’t fit any of, the ideas or preconceived notions that you have then I’d want to look for something to back that statistic up. before I say ‘oh this came from a legitimate research organization or it came from a well conducted study.’
I think the big takeaway of what we do is that every aspect of the research process matters and how you conduct the sampling is one of those aspects.
Scott: I think as you go look at that research, understand the context of it because there can be some alarming things in the course of that study. Yet, that may not be the whole picture. In this case, we definitely find several statistics where pastors are saying, this is a hard job. I think the rest of us in the congregation probably say ‘Our job isn’t exactly easy either. There are challenging things about my job every week as well.’ I think as we see the whole context of it, we see that pastors are really bearing down and they’re working hard and they’re trying to be proactive in many of these areas. Some of the best preventative things are some of those kind of things where the pastor or the congregation can be proactive.
Three of those would be
- Does the congregation have a document that clearly defines the expectations of the pastor? Some of those unrealistic expectations kind of get thrown out the window when everybody’s on the same page, literally.
- Another easy one is the congregation providing the pastor’s family with genuine encouragement. That’s not something just to do every October during Pastor Appreciation Month. It’s to do year-round and it’s not just the pastor. It’s their family that needs to be included in that encouragement.
- And a third tip would be equipping your pastor financially and with the time to take courses on interpersonal skills. It may sound like a slur the first time you do it. Maybe they have a problem with that. At the same time that’s not part of the normal coursework in a seminary setting. And so to encourage them to take advantage of courses that are available would definitely be helpful.
Lizette: Well we certainly would love to have feedback from listeners. Any particular questions about this project and the research or questions about working with bad stats, we want to answer as many questions as possible through the podcast in later episodes.
Thanks again for listening to Keep Asking. Remember you can Tweet us your questions, comments, feedback @LifewayResearch and @smcconn, @statsguycasey, and @LizetteBeard. We look forward to talking to you next time. Keep asking, learn more, do better.