Podcast: Play in new window | Download
Full Transcript of Episode 06: Keep Asking (Good) Questions and Tricks to Writing Surveys On Theology
Lizette: [00:00] Podcasting from Nashville, Tennessee this is “Keep Asking,” the weekly podcast that helps you dive in a bit deeper and wider into the research providing insight into today’s church and culture. Hey guys. Great to be back. Scott, have you been doing any cycling lately?
Scott: [00:29] I have not. I’ve been at band competitions on Saturdays. That’s my normal day to be out riding.
Lizette: [00:37] How about you, Casey? What’s your fitness…?
Casey: [00:40] I have not been doing any cycling lately.
Scott: [00:42] How about recycling?
Lizette: [00:45] That fits.
[00:47] How are you guys doing with “Pokémon Go”? What levels are you on?Casey: [00:52] Is there a zero?
Lizette: [00:54] There is a zero.
Casey: [00:55] Then I’m on level zero.
Lizette: [00:57] How about you, Scott?
Casey: [00:57] I have no even stared. I’ve driven my son several places to do a little Pokémon but…
Lizette: [01:03] You drive him so he can go by Pokéstops? Or you get him to a location and then you make him walk around?
Scott: [01:08] He’s out of the little Pokéballs, and so even if he encounters something you’ve got to go to one of those places to actually get more of them.
Lizette: [01:16] Would you actually drive some of us on our team around to do that? Because downtown Nashville has a lot of Pokéstops, but you can’t drive yourself and do that.
Scott: [01:25] The intention is for you to be getting exercise.
Lizette: [01:27] It is, actually.
Scott: [01:28] It just happens to be I didn’t want my son crossing a four-lane highway to go to the Pokéstop that’s closest to our house.
Casey: [01:38] Is it a grand master at this point, once he gets up his highest?
Lizette: [01:42] My Pokémon level is embarrassingly high, which I’m not willing to put…
Casey: [01:48] Say it.
Lizette: [01:48] No. I don’t know that I’m willing…OK, 31 [at the time of this recording], which is, if you know anything about it, it’s…
Casey: [01:52] I don’t.
Lizette: [01:53] It’s embarrassing.
Scott: [01:56] You’ve taken over the gym at your church.
Lizette: [01:58] Yes, I’ve taken over several, and I do have a few Pokémon that are so powerful it can squash Pokémon almost immediately. I do have one, if you guys are…have you seen the one that’s like the three eggs on top of a pineapple? Have you seen…?
Scott: [02:13] No, I can’t say I have.
Lizette: [02:16] There’s egg in the name. In recognition of our former executive director of Lifeway Research I’ve named it Egg Stetzer.
Scott: [02:26] Very nice.
Lizette: [02:27] You get to rename them. That’s been my kind of fun thing. Pokémon Go has been getting me out walking. It keeps my mind going so I’m not thinking of other stuff. It’s been fun. I’ve been able to hatch a lot of eggs, which I was told by my friend Amy Whitfield when we were walking.
[02:46] She said, “Please don’t yell that out loud while we’re walking.” Anyway, getting us back on topic, we encourage everyone to tweet your questions about research, about some of our topics. Our colleagues have been great to jump in with a few. [03:07] Barnabas Piper (@BarnabasPiper) who works here at Lifeway and does a podcast with Todd Atkins, “Five Leadership Questions.” Great podcast to check out. He asked a question that ties in with our last podcast. “How can people learn how to interpret data, since stats can easily be manipulated to say all sorts of things?” [03:28] I think that ties in with the whole purpose we want to have the podcast. In the last podcast, it’s paying attention to stats, but then here, how do you use it? Any quick tips tying in with the last one or what we’re about to talk to today about just one thing?Casey: [03:53] Today we’re going to talk about question wording. We’re going to talk about it more on the before we field a survey half of it, but I think it’s also important afterwards. A lot of times people are summarizing statistics.
[04:06] Maybe you’re getting a summary of a summary of a statistic by the time it gets to you because maybe it’s been in news media or whatever. Getting back to that original question wording, does it paint a different picture when you read that question wording as opposed to what that summary said at the end of it?Scott: [04:23] Most statistics are just percentages. They’re not that complicated in and of themselves, but as Casey just mentioned, making sure that you understand the exact wording of that question is going to be your best ammunition because a lot of times if people are comparing two studies, the questions were actually asked differently.
[04:42] The numbers won’t match between those two studies. Whichever question is closer to the point that needs to be made, or the topic of discussion is going to be the better question to inform that. [04:56] To make sure you understand what that percentage is that agree, that disagree, that say they do it, that say they don’t is going to be the most helpful thing as you jump into that conversation.Lizette: [05:08] Though one thing I thought of, as you were talking, is also pay attention to who is sharing the statistics. If you listen to that person a lot, are they willing to share statistics that disagree with them as much as the ones that agree with them?
[05:26] Or does the person, are they selling something or trying to get you to do something? What’s in it for them, I think is just a good filter to make you pause and do that. Casey, you’re raising your hand?Casey: [05:37] Yes. Another thing I just thought of, I’ve shared this with you all before but when the NFL had a labor strike they brought up…you’d hear the owners saying a lot, “The mean NFL salary is something in the range of $2 million or whatever.”
[05:55] Players would be saying, “The average player only makes $700,000 a year.” They were citing the median as opposed to the mean there. The statistics are both true. In one sense, they’re grabbing different aspects of that reality. [06:11] Because the median might be more indicative of what’s going on for the typical player, whereas the mean has got that long tail of if one player on the team is making $20 million plus, that affects the average drastically. [06:28] It affects it in a way that the owners don’t care about because they don’t care about what’s happening for each individual player. They’re saying, “How much is all this costing me?”Lizette: [06:35] I have a question. Pretend I’m eight years old and you’re explaining mean and median. Remember, I’m eight years old. Pretend.
Casey: [06:45] The mean is typically what people refer to when they say the average. If there are three numbers and they’re one, two, and three and you add them up you’d say, “OK, the average of those numbers, one plus two plus three is six. Divided by three, so the average is two.”
Lizette: [07:01] I respect you for being able to do math on the fly like that.
Casey: [07:04] Uh‑huh. I should have picked a better example, honestly, because the median of those numbers is actually two, as well, but that’s because it’s the middle number. It didn’t matter if I’d picked one, two, and three or one, two, and 50. The median still would have been two because that’s the middle number of the data set.
Lizette: [07:18] Excellent. Keep going.
Casey: [07:20] All I’m saying is it might not necessarily be the case that one of those numbers is being painted to provide a dishonest impression, although one of them might be, but they might just be pointing to different realities, as Scott was saying.
[07:37] Just being aware of what reality are they pointing to and does this line up exactly with what I’m interested in.Lizette: [07:44] A special thanks to Barnabas Piper. We want have some listener feedback, listener questions. We’ll probably start adding as a regular segment. Again, you want to check out his podcast, Five Leadership Questions, and also “Happy Rant” podcast.
[08:01] Though that one can be a little irreverent so that’s not an official endorsement from Lifeway Research. Now, shifting and focusing a little bit more on questions. Scott, everybody can write a survey, right?Scott: [08:17] Anybody can write a survey, yes.
Lizette: [08:19] Because we get a lot of folks who…we see a lot of surveys out there. If you have a question, and we’re encouraging people keep asking. If we say keep asking, that implies keep asking questions. What’s the tension there that I’m setting you up for?
Scott: [08:41] Sometimes people will ask us to review a questionnaire that they’ve written. The hardest part about that is I can’t tell you if it’s a good questionnaire unless I know what you’re actually trying to find out.
Lizette: [08:52] We’d like to be clear. Please do not send us your questionnaires to review. It’s not that we don’t love you all. We don’t have time. That’s a client…let’s just be realistic. Please do not start emailing us your questionnaires to review. Is that too harsh?
Scott: [09:09] No, that’s right because we don’t like to jump in in the middle. We want to start that relationship with the ministry from the very beginning.
Lizette: [09:16] The very beginning, yes.
Scott: [09:17] What do they want to be measuring? To work backwards is incredibly hard. You can write a question. We learned to do that in English class in second grade or something, how to write a question. How to write a question that gives you actionable information for the decision you have to make, that’s what a good questionnaire is. There’s a lot to that.
[09:42] You have to understand what’s involved in that decision, but you also have to understand how a respondent’s going to read it and what’s going to be going through their mind and how can they put the answer in their mind onto our questionnaire. [09:58] That comes with experience. Knowing which questions have worked through the years, which type of questions don’t, and how to get around some of those “gotchas.”Lizette: [10:07] One of my friends, @DanaRunyan, who is a curriculum coach in Missouri. She’s always talking about when they’re writing assessments or working with teachers on how to write questions and the importance of asking the right question to find out what you really need to know.
[10:31] I’ve learned from talking to her and what they’ve done that that is a important thing that’s been going on in education, a heightened emphasis for the past 30 years or so. They look to research to say…if they’re writing questions, how do they write quality questions. Or if they go to a source for it. [10:52] I also see that in the medical field. I see it more. I see it in business. But I see a lag or a skepticism in the ministry, in the church. I know we’ve talked about it. Where do you see that coming from? Why is it hard to adapt? Why is there that gap?Scott: [11:15] A lot of times, in ministry, we can have the not invented here mentality of somebody in another place who writes a survey, how can they really understand what’s happening in my church and in my situation? You may feel like you’ve got to write it just to capture what’s going on right here, right now.
[11:37] There’s an element to where there can be some contextual things that need to be incorporated into a survey or into a question but, at the same time, as you mentioned, these are things that get better over time. The researchers writing the questions are better at writing those questions as they have experience, both in a topic area as well as a style of question.Lizette: [12:01] We ask a lot more working on projects. We’re asking the folks that we’re working with, “What decision are you trying to make?” as we look at the overall purpose of a project. Then, as we start to break down questions.
[12:16] A lot of times we find they haven’t drilled down to that yet. Or it takes a while to get there. Or they want to make 20 decisions. What’s the importance of narrowing that down and getting it to the decision-making point, a point where they can make a decision on that question?Scott: [12:35] Usually research is not free, or even inexpensive. If you’re going to go to the trouble of spending the time on it and the out of pocket expense to do a research project, you want to make sure it’s something you’re going to use. It’s not going to be a notebook on a shelf. It’s not going to be a file that never gets read.
[12:56] You want something that you’re going to actually be able to make decisions with. It’s going to change your thinking. If you haven’t thought about and articulated, is probably the more important piece, the things that are keeping you up at night. [13:11] If you’re not articulating the strategic things that you’re going to have to grapple with in the weeks ahead, then as you begin to think about what goes into that survey you’re going to have topics that are nice to know instead of topics that will actually influence some action.Lizette: [13:31] One thing I would strongly recommend, and I have recommended when talking with churches, that even if they are going to write a survey and they know what they’re trying to answer. They know what questions they have. I encourage them to keep it very simple and keep the scope very narrow.
[13:49] Maybe two or three questions and then send it out and then take action on what you learn immediately because we hear from folks a lot that if a group, a church, a non‑profit organization puts all their folks through the process of going through a survey and then there’s no action, no decisions, that’s very frustrating and it’s more difficult to get people engaged. [14:17] It can be done at a very simple level. If you can answer the simplest question of, “We know exactly what kind of decisions we can make on this.” Casey, as the guy who’s looking at the data, what’s it like for you when you see the results, but then you look at the question and you see some of the problems that you can see and are frustrating for you?Casey: [14:43] A lot of times one of the things we talk about is, is it a double-barreled question. What we mean by that is are you asking more than one thing and then acting like it’s one thing, potentially.
[14:54] Do you agree with A, B, and C. Yes or no? If somebody says no, does that mean they disagree with A, B, and C? Do they disagree with A and B but not C? There’s no way to parse out that information. [15:06] Breaking it down into simple questions allows you to know why did they agree or disagree. That can be one aspect that can be frustrating after the fact. If it’s not clear and precise. [15:19] One of the other troubling aspects is if it’s a biased question, if it’s a leading question. Is there an outcome that it seems like the question itself is driving people, a desirable answer to that question?Lizette: [15:35] Jumping in, let’s talk about a simple topic that’s super easy to write questions for. Theology. There’s no trickiness there. This is what prompted us as our theology project. What were some of the outcomes with that project, Scott? What was challenging?
Scott: [15:56] The project came to us from Ligonier Ministries. They’re a group that has already done a lot of thinking about theology and a lot of writing about theology. It was a good group to work with on this topic.
[16:12] They came to us with a good research question, which is, “Can we parse out who is theologically on the right page and where are some spots where Americans, and specifically Christians, are actually heretical in their beliefs?” [16:30] One of the difficulties there is trying to ask questions that are not all expecting a positive answer so that people don’t camp out at the agree end of an agree/disagree scale. When we get into something that might be heretical, trying to word it in a way that it’s still… [16:55] We’re not biasing it by it sounding bad. This can be a challenging thing, just from a question writing standpoint, as well as how they respond to them.Casey: [17:09] Any time we’re doing it more with statements…we’re making a statement. That already leads itself to the possibility that you have an acquiescence bias there. What I mean by that is that people are more prone to agree than disagree when they’re not really sure about the answer.
[17:28] One of the ways that we can tackle that in the aggregate, at least, is by asking questions both on the positive and on the negative side and then any one individual question, they might be leaning a little more towards agreeing than disagreeing, but we’ve got a full picture when we asked…what was it, 43? [17:46] We asked a lot of questions about a variety of theological stances that people have. You can see where are they falling short. You can follow those trends.Lizette: [17:54] If I was working near an elementary school, I would hold up a sign right now that said, “Slow Children Crossing,” and I’m going to ask you to back up to acquiescence bias. Did I do that, say it correctly? Acquiescence bias. Just help me with that.
Casey: [18:09] Sure. How I’d originally try to define it there, and then I’ll break it down a little bit more. When people aren’t sure about a statement, they are more likely to agree than disagree. If I say something like, “Hard work creates success.”
[18:22] You’d in your head say, “That’s kind of a truism. There’s probably times when I can say that’s true and sometimes when it’s not true. That’s a pretty broad general topic. Overall, you’d say, “I guess I agree with that,” because it’s one of those things where you have to pick a side.[18:41] When, instead, you have a theological statement like some of the ones that we were asking, you’d hope that people aren’t in that murky space of, “This sounds nice, I guess. I can’t agree or disagree very well so I guess I’ll agree.”
[18:55] Some of the ones that we observed that with on the Ligonier data where we saw a majority of those, even with evangelical beliefs, agreeing that Jesus is the first and greatest being created by God. The Holy Spirit is a force, not a personal being. [19:14] Some places where you’d say, “I’d hope that they have the theological grounding to be able to say, “No, I know that’s true. I can disagree.” Instead, they’re in that murky space where, “I guess I’ll agree because that sounds like it could be true or not true.”Lizette: [19:31] We’ll have these slides in the show notes so you can see the specific data and links to some of the press releases that we’ve done. Scott, didn’t we see this when we were testing out our church planter assessment?
[19:42] Is that where acquiescence bias ‑‑ I’m pleased with my new phrase ‑‑ but when we would have these outrageously high responses. “Yes, I’m a self‑starter. Yes, I’m this.” We called it the Jason Bourne, John Wayne effect. That we had to work harder at writing the questions.Scott: [20:01] Definitely. In that case, it’s true in both studies, where you’re describing yourself. I think it’s going to be even more likely, then, if you’re describing somebody else or an outside group. When you’re describing your own theological beliefs, or describing your own ministry as a church planter, it’s really hard to say, “Well, no. That’s actually absent from my life.”
[20:26] There’s been studies done across countries. Across countries, some people are more likely to give an agreement answer, or a yes answer, than a no answer. Really, I don’t think there’s any countries that are biased toward giving a no or a disagree answer. [20:45] It’s the extent to which people want to be agreeable. The United States is one of those, where we would prefer to agree more often than we disagree.Lizette: [21:00] Focusing on the area of theology, how do you stay out of hot water? When you’re writing these questions, because there can be some pretty big disagreements, even within evangelical. Orthodox evangelical, Christian folks that we would all say, “Hey, we’re brothers and sisters in Christ.” How do you navigate that?
Scott: [21:22] Like many topics in research, you have to very quickly get up to speed on what the issues are. One of the most important things is to have multiple readers. In this case, as we worked with Ligonier, they had a number of theologians who were reading behind those drafts of questions and flagging, “Well this word, we’re starting to hint at this, which we don’t want to go there.”
[21:47] That’s incredibly helpful. Some of it, too, is that basic practice of multiple eyes and even those same people reading through the questions multiple times. If you have any hesitation, flag it. Note it. Let’s talk about it because when 1,000, or in this case 3,000 respondents are answering, if a portion of them are hesitating what does that mean? [22:16] I could answer it this way and that way. If they’re hesitating in multiple directions, then we’ve got a question where we might interpret the answer wrong.Lizette: [22:27] That makes me think of when we were validating the church planter assessment in Australia. We had experts looking at the questions. There’s one where, here, checking for emotional resilience we asked something alluding to temper. A bit of a temper.
[22:46] The Australian experts, across the board, said that is not an appropriate question because in ministry context, no matter who you are, the idea of having a temper is completely inappropriate. But it was OK to say, “People who know me well can say I have a bit of a short fuse.” [23:06] One was extremely offensive and they said no one would agree to that, but the other would work a little bit better. Anything else on what we learned from how we asked the questions on this project?Scott: [23:20] Even the example you just gave touches on something similar, but we would call it social desirability. Will people perceive it better if I answer this question in a certain way?
Casey: [23:34] With social desirability bias, we’ll ask some questions sometimes that we know that people are going to feel like they’re a better person if they answer the wrong way than another.
[23:51] If I ask, “How often do you brush your teeth?” the people who brush them three or four times a day are going to be really energetic about answering that question. The people who brush them once or twice a day are going to be a little bit more hesitant and might bump up that number a little bit.Lizette: [24:05] Wait a minute. Is brushing your teeth twice a day a bad thing? Is that not enough?
Casey: [24:10] OK, whatever.
Lizette: [24:14] I would answer twice a day and feel pretty comfortable. Sorry.
Casey: [24:17] That’s fine.
Lizette: [24:18] I’m now just wondering if I’m not socially acceptable.
Casey: [24:20] I feel like I grew up…you always heard in elementary school three times a day, after every meal. That was the…I don’t brush my teeth three times a day. I’m willing to say that on air, even.
Lizette: [24:30] Excellent. If we can come clean with a little transparency.
Casey: [24:34] In terms of that social desirability bias, how do we avoid it? One of the ways we can do that is, again, by hopefully couching the question in such a way that they feel like they can answer honestly, that there’s not a judgment happening.
[24:50] Sometimes, for really sensitive questions, we might consider what mode are we asking this question in. Sometimes those questions might be better to get on honest responses online than over the phone because if they’re having to tell a live person who’s talking to them that they don’t brush their teeth, they’re going to be less prone to saying that.Lizette: [25:09] I do brush my teeth. Now I’m feeling a little defensive.
Casey: [25:12] Lizette does not brush her teeth.
Lizette: [25:12] I do.
Casey: [25:13] That’s what we’ve learned from this discussion.
Scott: [25:14] Is that why this microphone smells?
Lizette: [25:16] No! Hey! Let’s not take this in an ugly direction.
Casey: [25:19] Shots fired.
Lizette: [25:20] We’ve really lost focus. I had a really good question, but now I feel like I need to go brush my teeth. I think this ties back with what we looked at with Barnabas’ question at the beginning. How people can interpret stats when stats are being used, tying back into the question.
[25:44] Look at how the question’s being asked. Look at who the sponsor is or who is doing the research. Take the time to see if there is a motivation. Motivation can be a positive thing or a negative thing. [26:00] Just know what’s going on there because if you can understand the theological family that a group is in, whether it be broad or narrow, or the theological argument that someone may want to make, I think that will help. [26:14] Casey, you’re raising your hand again?Casey: [26:16] Yep. Again, I think that sometimes when these criteria that we’ve been talking about, about clarity and precision and whether it’s biased or not, is it an actionable question? Sometimes we can fail them, or people fielding surveys can fail to meet them unintentionally.
[26:37] I think other times, and particularly in some promotional work, you’ll see surveys created to ensure a desirable outcome. The researchers might have, researchers being used loosely, have an outcome in mind as they’re fielding that survey. [26:51] One of the examples I saw, I remember there was a question out there about do you think that we should bomb Agrabah, which was a fictional city from “Aladdin.” They’re asking that with the intent to hopefully embarrass some people by them saying yes or no. [27:09] One of the ways that they got after that was it wasn’t just the question wording of that question itself, but if you give them 30 questions leading up to that that are about Middle East conflict, then that preps them and loads them to a place where then you ask about this fictional city. They’re prepared to feel confident giving an answer about it. [27:32] Sometimes it’s disingenuous in a way that’s, again, with malintent. Other times it’s just somebody hasn’t really thought through all the implications of what they’re asking.Lizette: [27:46] That’s again, looking back to who the source is and then also, what is their record in the research that they’ve released and any kind of bias. Since we covered the importance of good questions in this podcast, join us for our next episode where we’ll cover more about what we’ve learned in this study on theology.
[28:09] We do want to hear your questions and your comments and your feedback. Your nice feedback. Well, alright. We can handle your honest feedback, everything. Thanks for listening to Keep Asking, the Lifeway Research podcast. [28:23] If you’ve liked what you just heard, feel free to like us and rate us on iTunes. We hope you’ll pass along our Web address, LifewayResearch.com, to your friends and colleagues.Be sure to Tweet your questions and comments to us: @LifewayResearch and individually: @smcconn, @statsguycasey, and @lizettebeard. Join us next time for another edition of Keep Asking.
Links for The State of Theology Study
- Ligonier Ministries – sponsor of research highlighted in this post
- The State of Theology: A Poll of Eternal Significance by Stephen Nichols
- The State of Theology: Interactive web site for the project
- Americans Love God and the Bible, Are Fuzzy on the Details by Lifeway Research
Podcasts to check out
- SBC This Week – A weekly news roundup from around the Southern Baptist Convention as well as guest interviews from today’s top church leaders hosted by Amy Whitfield and Jonathan Howe
- 5 Leadership Questions Podcast – Each episode co-hosts Todd Adkins and Barnabas Piper will ask five questions of different guests or about different leadership topics
Other links of interest
- How to Interpret a Statistical Bar Graph” Statistics for Dummies by Deborah J. Rumsey
- The Mean and Median (Calculator Soup Website) To get hands on experience, you can go to this website and it will show you the difference between the mean and median (and mode) for the same set of numbers.
- When and how often should you brush your teeth? Mayo Clinic