Podcast: Play in new window | Download
Full Transcript of Episode 18: Interview With Religion Writer Bob Smietana (Part 2)
Lizette Beard: [00:00] Podcasting from Nashville, Tennessee, this is “Keep Asking”, the weekly podcast that helps you dive in a bit deeper and wider into the research providing insights into today’s church and culture.
Lizette: [00:23] Welcome everyone. I’m Lizette Beard. I’m here with my colleague, Scott McConnell and another colleague, Bob Smietana, here for his second podcast with us, for Keep Asking. We’re kind of excited about that.
Scott McConnell: [00:47] Drum roll. Just like Saturday Night Live makes a big deal about repeat guests, Bob Smietana is our first repeat guest.
Lizette: [00:53] That’s awesome. Congratulations.
Bob Smietana: [00:55] Thank you. I’m so honored.
Lizette: [00:57] You should be.
Bob: [00:58] You love me. You really do.
Lizette: [01:00] Yes. That’s awesome. Today we’re going to talk with Bob. I didn’t prepare any surprise questions. Bob would be great with a surprise question. I feel like we’re going to learn quite a bit about him.
[01:13] Last time, when he was with us, he was talking about his research process and a little bit about what he does, when he’s writing for stories. Actually Bob, to get our new listeners familiar, tell a bit about who you are, your national prestige and your amazing skills.Bob: [01:30] I don’t know if it’s national prestige or national bad reputation.
Lizette: [01:35] Either way, you’re welcome here.
Bob: [01:37] I am a long time religion writer. Since 1999, I’ve made my living writing about religion. I was a freelancer for a long time. I spent six years at “The Tennessean.” I’ve written for USA Today, Christianity Today, The Washington Post, all kinds of folks.
Lizette: [01:55] Excellent. Today we’re going to talk a little bit about when you’re writing a story and giving the narrative to data. Bob works on our communication team, which is right next door to our Lifeway research team. You’ll see, very often, Bob is the one that’s telling the stories about data.
[02:17] We thought this would be good just to know a little bit about that process, maybe how he thinks about it. We thought it would also be insightful for folks who may be blogging or doing their own communication. [02:28] When you’re seeing stats and data, what do you do? How do you handle it when you’re unfamiliar and on the receiving end of getting that? Going back to when you first started, you’ve probably written about data for a long time, when you see it. Was there a big difference when you had to start writing about it all the time?Bob: [02:49] Yeah, there’s a big difference. You’re always looking for a story in the numbers. You see those numbers and you say, “OK, what does this mean?” One part of the story is so‑and‑so and such‑and‑such happened. 5 out of 10 people don’t go to church on Sunday, or 8 out of 10 people. Six out of seven people think the antichrist is coming soon.
Lizette: [03:07] Bob is making these numbers up.
Bob: [03:09] I’m making this stuff up.
Lizette: [03:10] Just as an illustration.
Bob: [03:10] About half of Americans, for example, according to Pew Research, will switch their religious identity at some point. It’s mostly Christians switching from one kind of Christian to another, from Protestant to Catholic, from Methodist to Presbyterian, from Church of Christ to non‑denominational. There’s a lot of us switching. That’s one fact.
[03:32] What does that tell us about the way that people think about religion? You think, “OK, what do the numbers here tell us?” We had a story recently about domestic violence. You look for contradictions in the data. Churches say, “Boy, we really want to be helpful to domestic violence victims.” Do you have any programs? “No.” Are you prepared? “No.” Do you think it really happened? “Maybe not.” [03:59] These data points are telling a story that maybe the churches are not able to live out what they think or some of the things they do, impede what their goals are. You’re always looking for that kind of thing. It’s harder now that I write about it all the time and I know the researchers on a first‑name basis. [04:18] When you just get a report from somebody, especially if you look at the crosstabs, the crosstabs, because of the margins of error, you can see a big difference. There are some different kinds of churches. You go, “Oh, there’s a huge number here. Evangelicals do this. Mainliners do this.” [04:32] They go, “No.” Scott will go, “No. They’re within the margin of error. That’s not very different.” You’re like, “Wait. It’s 15 or 20 points.” Yeah, but the sample size is so small, you can’t say. You have to look at the questions super carefully. What did you ask? There’s all this stuff about this election. 81 percent of Evangelicals voted for the current president. [04:55] What Evangelicals are those? They’re self‑identified, white Evangelicals, who voted in this election. Then we have a smaller set of people. I can’t make a broad statement about evangelicals, without saying this is the group of people we talked to. You see this all the time. You get a survey, I remember on immigration reform. It was something like…Lizette: [05:17] I want to just point out that these numbers, Bob is just pulling them off.
Bob: [05:22] Yes. I’m making this up.
Lizette: [05:23] I want you to feel free to do that.
Bob: [05:26] Yes, I’m making this up. I saw this survey about how few Evangelicals wanted immigration reform. They really wanted it all about security and no immigration reform. You look at their thing. It was likely Republican Evangelical voters. That’s a segment. That’s not all Evangelicals.
[05:45] It’s not all self‑identified Evangelicals. You have Evangelicals by belief. You have Evangelicals by denominational affiliation. You have all kinds of ways. Sometimes you need a shorthand, because you don’t want to always say, “Self‑identified, Evangelical voters in the survey.” [06:04] At least, at some point you say, “This is the number of people that we’re talking about. We’re not talking about a broad group of people.” You can sometimes use a narrow focus to make broad conclusions. [06:19] The other thing you find out is I can get a survey to say, as you know, whatever I want. When you’re reporting on it, what you want to do is say, “What does this data tell me?” I can find it out. If you are a pastor, or a blogger, or an opinion writer, you go, “I have this idea. What data can I find to make me look like I’m smart and right?” [06:42] You can misuse the data to make that point. The data doesn’t tell you what it means. It just tells you information. You have to sort out what it means. You always have to be careful. Am I using this data to promote an idea that might not actually be true? I always fact check our pastor’s sermon. [07:01] A pastor is giving a sermon. They tell an anecdote. I’m like, “I don’t think that really happened.” Or they give a statistic. A couple of years ago, the youth pastor was like, “Half the people in this county are all getting divorced.” I’m like, “No, that’s not true.” [07:15] Marriage is good and marriage is hard work, but don’t tell them half the people are going to get divorced, because they’re not. The numbers just don’t say that. If you give bad statistics, there’s a way to look at that statistic to make it right. That’s misleading.Lizette: [07:32] A good check for me is I need to be able to disagree with some of the statistics I share. If someone is telling me things, if I see it coming from a blog or someone is advocating something, I’ll ask, “What are they trying to sell me?” whether it’s their platform, whether it’s their point of view or it’s their product.
[07:57] If they’re only using data in this, wherever it’s coming from, to sell me something, then that will make me a little…Bob: [08:09] We have an annual meeting of religion writers, called the RNA, Religion News Association. Annual conference, maybe seven years ago, our old friend Ed [inaudible] and former boss did a presentation on how you deal with data. He said the first question you should ask is, when somebody gives you a statistic is, “How do you know that?” You can ask that.
[08:28] It doesn’t have to be a statistic. When someone makes some claim, you go, “How do you know that? How did you come up with that information? Could you know that?” Sometimes you just can’t know that. There’s no way for you to know that information, or if you did, tell me how you found that information and I can evaluate whether your process of getting that information is actually accurate or not.Lizette: [08:51] That’s actually a great question. Even for conversation, when people are just making some outlandish claims, “Now how do you know that?”, if you ask in a kind way.
Scott: [09:00] We get questions. Have you done research on this? Have you done research on that? A fair percentage of the time, they’re asking something that cannot be known or cannot be measured in the way they’re at least framing it. No, we haven’t done that and nobody else has either.
Lizette: [09:17] Yeah.
Bob: [09:18] Sometimes there’s no way to get at that answer in a way that’s helpful. Numbers can tell you certain things, but they can’t tell you everything. When you’re a reporter, you find these numbers and then you go out and talk to a bunch of people. I’ve got a story that’s posted on our site today that is about people who don’t go to church on Sunday, who sleep in.
[09:36] It’s an unscientific sample. I went and did a social media kind of blitz. “Tell me some people who don’t go to church.” We have data, but we also have real, live stories of people. Some people just didn’t go because they weren’t interested. Some people had been very involved. Two of them were former pastors’ wives and then the pastors imploded. Then they were left holding the bag. [10:01] Trying to find a new church was super hard for them. Other people were just like, “I have lost faith in this institution.” Those give bones to why we know a lot of people don’t go to church on Sunday or are involved in worship services on Sunday or during the week. Why is that? We get the stories, which give some life to those data.Lizette: [10:28] Now that you’re having to write stories with maybe multiple data points, instead of taking the hot data point, do you find that constraining or do you find that you’re able to dig a little bit deeper? What are the pluses and minuses of digging into specific points and really working with multiple pieces?
Bob: [10:48] You try and give a couple of different points out of any study, unless there are only one or two questions. You try and give as much info as you can. Sometimes there’s data that you just don’t know what to do with. You get a question and you go, “Eh.”
[11:02] [laughter]Bob: [11:02] It doesn’t fit with this other stuff.
Lizette: [11:03] What’s a “Eh”? Does anything stand out about when you’re like, “Eh”?
Bob: [11:12] We did stuff on Bible reading.
Bob: [11:14] There was something about, “Have you seen Bible reading on social media?” I’m like, “Eh”. OK, someone puts a quote on social media. People put stuff up all the time. I wouldn’t look at it. I don’t know what to do with that. What does that tell me?
Lizette: [11:29] That’s fascinating. I mentioned that, to me, it immediately makes me do a crazy check. I have several streams. There’s Micah Fries, who has Bible verses running all day long in between his Facebook Live. There are other folks. Then there are folks who’ll post those, their angel pictures and then a cussing rant, then their break‑up. You’re like, “Ah.” For me, it’s the self‑sorting in Facebook.
Bob: [12:00] It’s nice to see a Bible verse. Then I’m like, “Is that really what that Bible verse says?” I don’t think of the Bible verse any differently than if I read it in a book, unless I suspect that they’re using it in a misleading way. It might not really say that.
Lizette: [12:17] It’s the journalist in you not to go on a deep dive there. There’s just no curiosity if there are other people who do react, people who maybe don’t read the Bible regularly, then they’re like, “Oh.” Does that bring something to their attention?
Bob: [12:32] I suppose so. I just was like, “It doesn’t really fit with this other stuff.” In any study, often there’s too much information. You can only tell one story. I can’t tell you every part of it, because then I could go off in every direction and lose you.
[12:48] If you’ve only got, say, 700 words, you only can do 700 words, you try and think what are the most important 700 words? What are the two or three most important concepts that I can get to this, knowing that any story is more complicated than that? Any story you read in the news, there’s more information. [13:05] A friend of mine told me, early on, that every story…he was at “The Chicago Tribune.” He was an editor and then a VP. He’s like, “Everything we print is true, but for the five percent that the people really involved know all the nitty‑gritty in, that you could never know and you could never explain all the nuances in 5,000 words. [13:26] There’s always a limit. If you’ve got a limited number of words, you can only say a limited number of things. You try and be as concise and accurate as possible, knowing that there are probably nuances that you’re going to leave out.Lizette: [13:41] Any words of advice for folks who are writers or who they’re good writers but they’re new to incorporating data? In the previous podcast, we talked about make sure you’ve got good data. Make sure you’ve got good resources. Say they’ve already vetted it and they’ve got good data. Any couple of pieces of advice you’d say with how they step into that, in 700 words?
Bob: [14:06] I would be careful to make sure, unless you’re doing an op‑ed piece and you know you’re doing an op‑ed piece for a certain point of view, that you’re not reading into the data something that’s not there. You can have good data but you can misuse it.
[14:21] You should always try and ask the people who did the research. A lot of them will talk to you, to media. Say, “Hey, this is what I think this says. Is that right?” They might say, “No.” They might say, “Yeah, that’s exactly right.”Lizette: [14:34] We’d like you to address those questions to @smcconn at Twitter.
Bob: [14:37] You can do it on Twitter. Pew has people, PRI, library research. We have somebody who will probably answer. Now, if you got 5,000 emails, you wouldn’t answer them. You’re not going to get that many. You’re going to get a few who ask you a question and say, “Oh, am I reading this correctly?”
Lizette: [14:55] Right. Certainly, if you have the phrase written out, we’d much rather have the data out there presented correctly.
Bob: [15:03] You go and you talk to someone who’s an expert in that field. Bible reading, go talk to some Bible reading experts. How do they react to this research? What do they know about this group? If you’re writing about a different group or a different program, find out what you know about other people who know about them and how they react.
[15:23] For those people who stay home from church, say you see a thing on people who are unchurched. Talk to some people who don’t go to church. Ask them why. Get some more information to flesh that out.Lizette: [15:35] I want to set the table for a little bit of a debate here you touched on earlier. Statistical differences, the kind of numbers Bob wants to see to say this is different, whereas what you and Casey always have in mind, indicating those parameters. Discuss.
Scott: [15:57] We want to make sure that Bob or any other writer is saying something that’s real. Yes, this number can be different than that number, but is it indicating that those two groups are different?
[16:12] We’re using different statistical tests, depending on the type of question it is, to test for that. Yeah, sometimes it can fool us. Depending on the sample size of the overall study, it may look like a good‑sized difference, but it’s not enough.Bob: [16:29] Yeah. I want to know if it’s statistically significant, because it looks like it, but I’m not a statistician. Sometimes I want it to be statistically significant and you go, “It’s not.” Or sometimes you say, “There’s just not enough sample size for me to know.”
[16:45] If I have 1,000 people and the sample slice is 20 people, I need to know that, because I’ll go, “I can’t really make a lot of claims. We talked to 20 people.” The margin for error is huge. I just need to know that. I’d rather you know that, but I don’t want you to tell me that it’s not. I want you to tell me, “Bob, that’s genius. We never saw that.”You’re the first person to write that.” I would rather have you say, “No, you’re not a genius.” so I don’t look stupid.Lizette: [17:12] Do you ever find yourself on the hunt for something that’s statistically interesting? I want to see these bigger numbers. Or I want the numbers to be bigger than they are. Do you ever find them saying, “These are statistically significant, but it’s only 15 points apart.” You’re like, “Yeah, nobody’s going to care. That might mean something.”
Bob: [17:31] 15 is pretty good. What you don’t want is data that tells you nothing. You go, “Oh gosh.” Sometimes the data confirms something. We had this data before the election on how different Christians, who hold the same Evangelical beliefs, had different views based on…people of different ethnicities had different views, who had the same religious beliefs.
[17:57] They also had different party affiliations and that played a role. Other parts played a role, but there were differences. That was cool, because you could see something we suspected. If it had all come back the same in a mush, you would have been like, “Oh.” That would have told us something else that was interesting. [18:13] You want the data to be compelling. If you get 49‑51 and there’s a margin of error of three, that doesn’t really tell you a lot. You want something more. 40 percent, 70‑30 is better than 58‑42 still, even though that’s pretty far apart, because 42 feels like it’s close to 50.Scott: [18:42] One of the tough ones we look at are age differences. We might see what age group is different than another, but it’s not a clear pattern. What I want to see is something that stairsteps up from the youngest to oldest, or down from the youngest to the oldest. If that pattern’s not there, does it really matter that 35 to 44 year‑olds are different than the 65 plus, when the ones around them aren’t?
Bob: [19:08] That’s the worst, when the young people are really different than the old, that’s awesome, but then if the middle people are just different than the younger…
Scott: [19:16] They’re all over the place.
Bob: [19:17] or it’s all over the place and you’re like, “I don’t know how to sort that out.”
Lizette: [19:23] That could be a stage of life. It could be a factor of kids, empty nest.
Bob: [19:28] There’s not a clear answer.
Lizette: [19:32] There are always more questions.
Bob: [19:34] There’s not a clear answer. Sometimes it just seems like it’s just an anomaly of having a thousand people. They’re going to have some things that are interesting but not necessarily informative or helpful. It’s just the worst. Sometimes you really do see age differences or size of church differences, but sometimes you don’t. Sometimes you do, but it’s in a way that’s hard to get at.
Lizette: [20:00] Are you ever frustrated, if you’re starting with the narrative story and then you’re out to look for the data that’s out there and the data does not confirm or doesn’t feed the story? Do you ever run into that?
Bob: [20:14] No, because then you have a but, you have the opposite. So and so says that, but that’s not necessarily true. You can say the other part of it. You have enough variety to say. It’s always terrible when you think you know the answer and then it’s wrong. Sometimes you’re like, “That thing I got wrong is really interesting.”
[20:33] You can have an aside that not all people look at it this way. Different kinds of people look at it differently. That’s really interesting. Sometimes people you don’t expect look at it. Sometimes it’s isn’t a sizeable enough group of people and they do something unexpected, which is really awesome. “Oh, those kinds of groups of people do it differently.”Lizette: [20:59] I think this is a loaded question, so I want to be fair. Just to folks in general, who may want to talk about statistics, preach or do that, what’s the benefit of taking the time to write about the data and thinking and preparing before you’re speaking or passing it along in other channels?
Bob: [21:23] It helps to always ask, “How do I know this?”, so I’m not passing on bad information. We have enough bad information. We have lots of it. Once you get bad information out there, there’s no way to get it back. There’s no way to unsay something terrible. Twitter, Facebook has that button, where it’s gone. It’s gone and you’re wrong. It helps not to be wrong.
Lizette: [21:44] It helps not to be wrong. That’s Tweetable.
Bob: [21:48] It helps, because you want to give people helpful information. There might be something in there that you’ve overlooked. You want to write about it. You want to talk to people. You want to have as much information. It’s like when you do a sermon. I’m sure there are some pastors who get up, and just open it up and go, “Here’s what the Lord has said to me at this moment.”
[22:05] Usually, you look through it. You see what the commentaries say. You look at different ways people have interpreted it. You get some information and then you present a more accurate response to that text. You don’t just make it up. [22:21] You can get a number and go, “Five out of ten people don’t do this.” What does that mean? Why don’t they do this? Who’s involved in that group? What people do it and what people don’t? You need more information.Scott: [22:35] It’s so easy to word your statement about data in a way that’s not accurate. It’s not true to the way it was asked or it’s not true to the number itself. We process things in our minds based on our frames of reference.
[22:52] Once we write that, then, when we reread it, we go, “That’s not what that said, did it?” I find myself doing that all the time and I think I know our data. Yet, once you write it, you read it and go, “That’s not quite right.” I’ve got to reword that. Sometimes I have to throw out the whole idea, because I framed it wrong.Lizette: [23:15] You helped me get to where I was thinking. Once you see it written out, you think, “If that’s all anyone sees about our data, is the percentage and that line of commentary, is that fair?” You can acknowledge it doesn’t say everything about the data, but is what it says in that sentence or those two sentences correct?
Bob: [23:36] When you’re writing it out, you get some feedback. Sometimes the words you use…I like the word skeptical. If some group agreed with this, I’d be a little more skeptical. Maybe they’re not more skeptical sometimes. Maybe that’s the wrong word sometimes. You need shorthand to make it punchy.
[23:52] You have to make sure that you’re using the right, what are the right words in that, describing something. Sometimes somebody else is reading it, going, “Oh.” You don’t want to mislead people into getting the wrong conclusion. [24:04] You want it to be punchy so they read it. Then you send it to the statistics guy. I’d rather have Scott say, “That’s just wrong.” before the whole universe sees it, then it goes out and I thought I was right and clever and I was just misleading. [24:22] It’s the worst when you get it wrong. When you get an error in the story or you just didn’t say it right, it’s the worst feeling in the world. It’s like I’m a terrible person. It’s like, “Oh my god, we messed up.”Scott: [24:34] As a researcher, my prayer is that we catch our own mistakes, before they’re out the door.
Lizette: [24:38] That’s good. Before we wrap up, I’m curious how much push‑back do you ever get, when people disagree with the sentiment of a survey. What they think they’re doing is they come back at you and they say, “I don’t agree with that.” because they don’t agree maybe with most people say this.
Bob: [25:06] Yeah, you get pushback to think that reporting something is saying that it’s true. Like with the election, there was a whole bunch of discussion or we had some research on whether the Bible says you should vote. All these pastors say yes. It’s a biblical responsibility to vote. People say, “Well, I didn’t see it in the Bible.” They want to argue that.
[25:26] You’d write, “I’m just giving you this data.” This is what these people say. Are they right? Are they wrong? I don’t know. You can argue with them. I’m going to tell that’s what they think. Whether you think it’s right or not is important, but don’t shoot the messenger. I’m just telling you here’s what we found. [25:48] What you do with that is up to you. Maybe you want to say, “Wait, maybe there is a thing to vote. Or maybe they think that because they think you’re supposed to be loving your neighbor and they see voting and being involved in society as part of loving your neighbor.” Maybe that’s how they see it. They don’t have chapter and verse to say it. You get push‑back all the time. You have a survey and you go, “They believe this.” They don’t believe that really.Scott: [26:14] Another example would be around election time, there were people who were in the minority on a couple of the questions we asked. I would hear from them. They were just adamant. “No, I do want to know who my pastor’s going to vote for.” It’s the minority of Americans that want their pastor to broadcast who they’re voting for.
[26:37] Yet, there are people who think, because they have an opinion, that it’s what the majority of people think. We definitely reported there are a lot of people like them. They just happen to be not the majority.Bob: [26:51] Those stories like that…what they call a Johnson Amendment. The pastor should endorse. There are all kinds of people who don’t endorse. There’s a difference between saying “Do I want my pastor to endorse? Is it the right thing to do? Should to be legal?” Sometimes people are arguing about the legality of it.
[27:09] What we reported on was here are people’s views on whether they’re actually endorsing anybody, whether there are people in the pew who really want to know. In both those cases, it was very low. Very few people endorsed. [27:20] Very few people really wanted to know, but there’s another argument going out in the culture about whether it’s legal or not. That’s a whole other argument. Our data is informative. It informs people about a discussion going on. It doesn’t answer that other question. It’s just more information for people. [27:43] [music]Lizette: [27:43] Thanks Bob. I appreciate you being here. We thank our listeners. Bob, how can they find you on Twitter?
Bob: [27:49] @bobsmietana.
Lizette: [27:52] You can find us @smcconn and @lizettebeard. We’ll catch you next time. Thanks for listening. We look forward to talking to you. Keep asking, learn more and do better.
Bob Smietana is senior writer at Facts and Trends magazine. He previously served as the religion reporter at The Tennessean and is an award-winning writer whose work has appeared in USA Today, The Washington Post, and Christianity Today.