Dr. Steven Zhou, Assistant Professor at Claremont McKenna College, joins hosts Richard and Tara to discuss null hypothesis significance testing and alternative statistical approaches in IO psychology. Zhou shares his perspective on teaching research methods to undergraduates, the importance of understanding traditional statistical frameworks, and emerging alternatives like Bayesian analysis and machine learning. The conversation explores how to build stronger academic communities through mentorship programs and addresses the hidden curriculum in IO psychology careers. Zhou also discusses his role coordinating the SIOP conference newcomer ambassador program and his vision for supporting early-career professionals.
Key Takeaways:
- Null hypothesis testing serves as a foundational starting point but has important limitations
- Teaching statistics effectively requires balancing traditional methods with modern alternatives
- The SIOP conference newcomer ambassador program pairs first-time attendees with experienced mentors
- Understanding researcher degrees of freedom is essential for ethical data analysis
- Small liberal arts colleges can offer exceptional research opportunities alongside quality teaching
- Visual tools for structural equation modeling help students understand analytical flexibility
- Building community in IO psychology requires intentional mentorship and reducing hidden curriculum barriers
- Academic career paths benefit from strong professional networks and early socialization into field norms
Graphical tool for drawing lavaan models: Karch, J. D. (2025). Lavaangui: A web-based graphical interface for specifying lavaan models by drawing path diagrams. Structural Equation Modeling: A Multidisciplinary Journal, 1-12.
https://lavaangui.org
Op-end training program: https://www.beyondtheivorytower.com/
Website: https://thegig.online/
Follow us on LinkedIn: https://www.linkedin.com/company/great-io/
Join our Discord here: https://discord.gg/WTzmBqvpyt
Join The GIG Email List: https://docs.google.com/forms/d/e/1FAIpQLSfVQ4hyF8MA4G9W-ERwVL8_e91a-MUMuhNvxhXmgkSFUDFatg/viewform?embedded=true%22
Transcript
[Richard Landers] (0:00 – 0:10)
Welcome to the Great IO Get-Together. On tonight’s show, quips and queries about the world of work as IO psychology comes alive. Now please welcome our hosts, Richard and Tim.
[Richard Landers] (0:11 – 0:36)
Welcome everyone to Great IO Get-Together number 38. My name is Richard, this is my co-host Tara. Today we are exploring chapter 12 of our textbook, Research Methods for IO Psychology.
And this chapter is all about null hypothesis significance testing and alternatives. So to help us make better decisions about tests on the show today, we have Dr. Steven Zhou, Assistant Professor of Psychological Sciences at Claremont McKenna College. Welcome to the show.
[Steven Zhou] (0:36 – 0:38)
Thank you so much, Richard and Tara. It’s so nice to be here.
[Richard Landers] (0:39 – 0:47)
To start us off, maybe you can share a bit about yourself and your brand new role at Claremont McKenna. What’s special about working there and how’s the transition going?
[Steven Zhou] (0:47 – 3:14)
Yeah, so for those who don’t know me, I finished my PhD at George Mason University in IO Psychology. And then I took a year staying at George Mason adjuncting. So I got to teach some graduate classes there and did some consulting, especially with nonprofits in the area.
While I was going through some more personal stuff, but just trying to figure out where we would end up moving to and the chaos of academic careers and all of that. And so that was my previous year. And then as of this fall, I started as the Assistant Professor here at Claremont McKenna College.
And for those who don’t know CMC, Claremont McKenna, we’re a small private liberal arts school in Southern California, and it’s amazing school. I know everyone says that, but let me just highlight a couple of things. I’m happy to get into detail on any of these.
A lot of schools say they’re balanced. CMC is truly a balanced institution. We teach a two-two load with 18 students per class.
And these are some of the brightest students ever. The acceptance rate here is like less than 10%. So it’s amazing students, highly, highly engaged.
I tell people they keep me on my toes. Like first of all, they’re paying so much money. So they’re expecting the best of the best in the classroom.
And so it really forces me to think, how am I really preparing them for their future careers, for their future grad school and so forth? So that’s the teaching side, amazing teaching, a huge emphasis on it. And then on the research side, I was lucky to have a couple of different options and I can share that CMC’s research support offers and opportunities here were rivaling some of the big research universities.
They have amazing funding for conducting research, lab space for conducting research. Everyone here is research active. And it’s actually one of the few small schools that actually have a thriving IELTS community.
So we’ve got David Day, Ron Riggio, Jake Honger, Jen Fatosa here. So I’m joining a really amazing IELTS team here at Claremont McKenna. Even though actually just a side note, I was actually hired as the quant faculty.
So the search was actually for a quantitative psychology professor. It just so happened that my background is quant and IO. And it actually fits for other reasons in terms of being a little bit more teaching focused in that balanced environment.
They really wanted someone who could teach quant, a little bit less on the how much research are you doing in quant, as long as you’re, of course, doing research. So that’s a broad overview of the school. And there’s so much more I could say about it, but it’s been a wonderful semester here.
I just gave my last final two days ago. So lots of grading left to do, of course.
[Richard Landers] (3:14 – 3:44)
Well, it sounds like you’re well prepared to chat about some complex topics that usually scare some people off on the quant side. This chapter is about null hypothesis significance testing, which I think we characterize as a sort of default for better and worse for a lot of folks when trying to analyze their data. So we try to explicitly talk about how there’s a time and place for it.
So is that also your view? When would you use alternatives, if you will?
[Steven Zhou] (3:44 – 7:06)
I love talking about this topic, because as you said, it is required learning in the sense of you really do need to understand it, even if it’s not the best option to use, which I fully agree, it’s often not the best option to use. And so here at CMC, I teach our intro stats class, and I basically own this class. I’m teaching it every semester for the next wherever, how many years?
I’ll have it until some sabbatical in the future, way in the future. But I own this class, right? And so all these students are coming in, and they’re usually scared of statistics, and they think it’s all math, and they don’t want to do it at all for most of these students.
And I get the pleasure of introducing them to both the math side of it, but also the philosophy side of how do we use numbers to ask questions? And that’s where the null hypothesis significance test comes in. I really am passionate about teaching it the right way to make sure students understand what it does and what it doesn’t do.
And so to your question, Richard, yes, it’s required because I phrase it as it’s the starting point. It’s the first point for any analysis where at least you can use it to say there’s not nothing here in my data set and what I’m looking for. Now, I emphasize when I teach it, it doesn’t say anything about what it actually is, like what actually is happening in your data.
It just says not nothing. And I’ve tried many different ways to teach this, to be clear, and I’m still learning the best way to teach it. But the one I used this semester was the multiverse, just talking about how this multiverse of all these different populations that a data set could come from, we’re starting as if we’re on Earth 1000 where there’s no effect happening.
And at the very least, can we say, let’s leave that Earth. Let’s leave Earth 1000. Now there’s a bunch of other Earths we can go to and we don’t know what those are, but that’s when the more advanced methods come in and alternatives to look at where do you actually go to.
And the last thing I’ll say there, continue talking about it is, pedagogically, I think it’s still, of course, I am in support of it being the foundational element of intro stats classes. I’m not saying we should get rid of teaching it by any means, because I think it gets students to finally think carefully about what probability is. We just say the word percentage probable and we throw it around, but most students don’t understand, well, there are different ways to define it.
We’re taking this frequentist approach here. I drill it in the students and say, we’re talking about long-term relative frequency. That’s how the null hypothesis works.
It’s not amount of belief in something. That’s something different, which we know is Bayesian and other approaches, but it sets a foundation. It gets them to think, this is one way to understand probability and it opens up the doors for future ways to understand it.
And the second reason I think pedagogically is so important is also just the reality that most students aren’t gonna go on to future graduate level research. And so they’re going to be doing some research here and there, of course, but they’re primarily going to be reading articles that have been published and many of them for many decades that ultimately still use the null hypothesis significance test. And so I see it as my job to lay a foundation for them to learn the better methods for when if and when they go on to do research, but also prepare them to be able to correctly interpret the vast amount of existing research that’s out there right now.
[Richard Landers] (7:07 – 7:14)
So are you teaching those alternatives to your undergrads too, the Bayesian and, or machine learning, like what else are you exposing them to?
[Steven Zhou] (7:14 – 8:46)
I had to stop somewhere, right? I tried to squeeze so much in, but at the very least they actually first, I actually first teach confidence intervals. I explain what that is and help them think through that before I go to p-values, but then I kind of admit to the students that you’re gonna end up seeing more p-values than you’ll see confidence intervals, but at least they start somewhere there.
And then I really hone in on effect sizes. They are required to use effect sizes in any of the analysis and understand the difference from your statistical practical significance and all this other stuff. And I hint towards the rest.
So I don’t, I hint towards, okay, there are different ways of understanding probability that if you’re looking at big data sets, there are different approaches to conducting research that’s not a statistical test, but it’s a visualization. It’s telling a story that I think leans towards the data mining things that’s discussed in the chapter. And I set up machine learning as advanced regression.
Now, of course, it’s more complicated than that, but that way, like when I get to regression at the end of the semester, it lays a foundation for when moving towards prediction, but I don’t get into the difference between prediction and description just yet, but I at least lay that foundation so that they can take a class that gets them the next step. And that’s really what my hope is, as the first quant psych faculty here at CMC is to build out. We only have an intro stats class right now.
So I’m launching an advanced stats class in the spring and hopefully building out more machine learning, NLP, large language model classes in the future.
[Richard Landers] (8:46 – 9:38)
That’s a very familiar set of concerns. Actually, the first class I ever taught was like a intro stats course for our psych majors. One of the things I actually struggled a lot with was where to put effect sizes, because it seemed like in some ways, they’re the most fundamental.
It’s the most fundamental question is just how different are two groups or how do you quantify this relationship you’re looking at? And it also seems to be the most common problem that for those that go into industry, like three quarters of what you do is percentages and mean changes and those kinds of effects. Do you think taken to account or how do you think about what the career paths are for the folks that aren’t going maybe academic?
Like, are you thinking of this as almost like a job training or are you thinking of it as a preparation for being statistically literate in a complex world? Like what approach do you take?
[Steven Zhou] (9:38 – 12:31)
I’d say all of the above, obviously. You know, we’re psychologists, we can say everything. But so a couple of contextual factors, CMC also unique in that most of our students actually like two thirds are Gov or Econ majors and the students really come in with a focus on applied careers.
That’s everything they’re going into when they do research and something like 70% of our students actually are involved in undergraduate research labs. It’s all for the purpose of going towards applied careers primarily. Of course, we have a handful that go on to academic careers and do very well for themselves, but it is much more career focused.
Case in point, I put out a call for people to join the lab and got resumes and you know, undergraduate resumes, what are you expecting? Out of 300 first year students who came to CMC, the Career Center hosted a resume writing workshop, completely optional and 95% of them showed up. By the time their resumes hit my desk, these were professional resumes.
I could not make any decisions based on them because they were so polished and well done. It was insane. And I had to think through different ways of finding students to work with.
That’s a bit of an aside, but it does give that context so I know these students are really eager to say, how am I going to actually apply this? And so I really have to lean towards that, but also still keep in the element of, you know, I want them to do research, I want them to think critically about things in ways that will inspire them to advance our field of knowledge, right? So I combined the two and I tried to find ways like, okay, in class examples of, I have one example assignment that I didn’t do this semester, I used to do, was they had to write an email to their boss explaining a t-test in detail.
So the idea is they’re writing it to somebody who has no idea what statistics is, but I want to make sure they’re correctly interpreting a p-value, they’re correctly interpreting effect size, and explaining what matters and what doesn’t matters. So that’s just a small example of taking that more applied route. But then on the side of both applied and also doing research, you know, they have a final project, and so they find their own data set and they analyze it and they present it.
And so on one hand, it’s preparing them for verbally explaining what they’re doing and presenting data-related findings. On the other hand, it gives them a chance to just tap into a little bit of, oh, I want to study something I’m interested in. I had one student who’s a swimmer use one of those devices that captures data on heart rate and stroke length and all this other stuff, and collected her own data set on her own swimming for hundreds of laps or something like that and analyzed it.
So that was a lot of fun. They just did their presentations last week. So I guess that’s my approach of trying to blend all of those, recognizing most of them are going on to applied consulting-related jobs.
That’s kind of a lot of people’s career paths here, but still instilling in them that sense of doing research.
[Richard Landers] (12:32 – 13:27)
I think that’s a good jumping off point to talk about the differences between what we might think of as a more stereotypical scientific concerns versus more practice concerns. And that’s not just in IO psychology, but practice in general, how these stats kind of play out in the real world. And I know that recently you wrote, you have a paper in Industrial Organizational Psychology, the journal Industrial Organizational Psychology about science practice gap.
I’m curious how you see the connections between not just an HST, but like all statistical approaches here that we’re talking about and how that facilitates or is aside from, I don’t know, what’s the relationship between stats fluency maybe and the science practice gap. You see that as something that needs to be resolved. Like you feel you’re preparing by teaching these topics this way that you’re actually preparing your students to like blend those worlds together.
I don’t know. How do you think?
[Steven Zhou] (13:27 – 15:38)
I really hope so. I hope that I’m doing that. And I do think it needs to be resolved or at least addressed for sure.
And especially on that stats fluency piece, right? It’s just, it’s sometimes disturbing how many news articles are out there that report a statistical finding and have no idea what they’re saying. And it’s, I really, I think your textbook references it.
I’ve seen other textbooks say something like 80% of existing stats textbooks don’t correctly explain a p-value. And we can only imagine how much more of confidence intervals because that’s even harder to fully explain. And that’s disturbing to me, right?
And I really hope that every student can take a stats class that enables them to read a news article that reports this correlation, that reports this significant t-test and starts asking critical questions. So I haven’t done this. I want to make like a cheat sheet for my students.
Here’s the questions you should ask the moment someone gives you a research finding, especially if it’s in a news article, right? And just things like, where did they get their data? Like starting from, did they really do a random representative sample in the first place?
And then how did they actually conduct their experiment or collect their data? Did they do any sort of manipulation check? Did they do any sort of reliability check?
Did they look at what are they actually collecting? On the analysis side, how are they reporting the null hypothesis? Difference between statistical and practical significance.
Like I said earlier, I don’t teach Bayesian, but I would have loved to put in all the issues of percentage correct. What does that actually mean in terms of positive and negative results on testing? That was obviously a big deal during COVID and there was so much misinformation being spread on how tests actually work and how accuracy actually works.
And I just really wish students would get the chance to really absorb that and understand that. I know that’s a lofty goal. And at the end of the day, not every student is going to go through a stats class, but I hope that I can do a little bit at least to really help students be prepared for a public understanding of what statistics really means.
[Richard Landers] (15:39 – 16:19)
Yeah, I mean, lofty goals aren’t bad. I’m curious what you think of, I mean, is this the last opportunity to try to correct for this sort of problem? Students, I like to think that the students take what we teach them and bring them out into the real world.
But of course, there’s gonna be a relatively smaller than 100% number of them that are gonna actually do that. But then we have a much larger group of people who never had that training in the first place. Do you see opportunities to teach or train or mentor the public, I guess, in stats consumption or stats literacy?
Or is university the last chance? What do you think about that?
[Steven Zhou] (16:19 – 19:08)
Yeah, it’s definitely not the last chance, but it is harder, of course, afterwards. And I’m still so early in my career, I’m still figuring it out as I go. So I don’t have any answers by any means.
But I’ve tried to pursue ways to build skill sets that could help towards that in the future. I can throw out a couple of things that I’ve done. There’s a really cool group called Beyond the Ivory Tower.
It’s funded through the New York Times and they bring a New York Times op-ed editor to work with a group of scholars, specifically has to be faculty members, to draft op-eds and talk about how do you write them for New York Times? How do you write them for the public? So I got the chance to participate in that group last year.
And that to me is like a step towards getting the training to try to write about statistics and, or concepts in general, it doesn’t have to be just stats, but that’s what we’re talking about here. And to bring that to a broader public so that they are getting trained in those ways. And so I hope there are paths towards that.
I think one of the challenges off the top of my head is statisticians, I say this carefully, can be, we can be difficult at times, right? Like we can talk about things that don’t make any sense to the broader public because once we get into, actually, let me say this, I’m running into this problem right now as I’m developing an advanced stats class that’s covering structural equation modeling, factor analysis, multi-level and all this for undergrads. So I’ve taught this class before at the graduate level and at the graduate level, I feel comfortable getting into the matrix algebra, having them actually calculate an eigenvalue by hand so they understand how it works.
I don’t think I can really do that in an undergraduate classroom, right? Maybe, and I’m debating that. I really am debating that with myself of how much can I bring in there and how much do I have to gloss over the details to say this is how you do something.
And I guess what I’m saying there is when we’re trying to train the public, especially those who are no longer in college and not able to learn and take an exam, but just kind of looking at, oh, what’s interesting? What’s a headliner? What’s a little bit of clickbait?
It’s really hard to take a complex statistical topic and translate that into something that a 30-year-old who hasn’t been in college for a decade will actually open up and read. I don’t know how to do that. And I think we shoot ourselves in the foot a little bit by trying to be so accurate as statisticians or applied statisticians, I should say, where we try to be so accurate that we sacrifice accessibility.
And there’s a fine line there, and I don’t know the answer to you. I just acknowledge that that line exists and makes it really difficult.
[Tara Behrend] (19:09 – 19:14)
But it’s a great point. I wanna come back to your op-ed project. Have you written an op-ed?
Where can we see it?
[Steven Zhou] (19:14 – 20:13)
I’ve written a few different ones. I had one that was about personality testing, of course, at work. That was Fast Company.
A couple in more higher ed things, like I had this piece I’m pretty proud of that was talking about how academic administrators require a completely different skillset than research and teaching, which I think we can all agree with, but I was just trying to articulate what are we actually doing if we take someone who’s a full professor who’s amazing at research and teaching and then put them in charge of a university to decide do we shut down and fire half our people during COVID or do we stay open and risk all these different hospitalizations and deaths?
So that kind of decision-making and leadership is nothing near what we do in our day-to-day research and teaching jobs. So just a couple of small examples. I’m certainly not as, there are people who do a lot more op-eds and I admire how they translate our science to a public audience.
[Richard Landers] (20:13 – 20:38)
We’ve talked a lot about how your approach is. I’d kinda like to ask the reverse. In what ways do you think you’ve changed your own thinking and approaches as a result of seeing these responses that you’re getting both from undergrads or your grad teaching or even responses to op-eds, people writing to you or connecting with you on LinkedIn, whatever it is, how has that sort of audience response affected the way you think about all of this?
[Steven Zhou] (20:39 – 23:22)
It’s just continued to help me refine and think through how do I make this more engaging, keeping accuracy, of course, but more engaging. I’ll say one example here. I just, again, finished teaching intro stats here at CMC and I realized about 2 1⁄3 of the way through that the students were sitting and just listening the whole time.
And to be fair, that has to be true to some extent in a stats class where you just have to lecture and explain here is what the null hypothesis test is and here’s how you do it. But I just realized I really could have made it a little bit more engaging to get students to take it and actually apply it in front of them. And so I tried that for the last third of the semester, a little bit more small examples, like I was teaching reliability and so showing, obviously, Chrome X Alpha, but then also inter-rater reliability.
And so I had the students watch a bunch of Dancing with the Stars clips from the dances and then rate them and submit their ratings. And then I collected it live and sent the data set, cleaned it quickly and sent it out to everyone and have them run IRR on it. And so just little things like that so that they remember doing it, which hopefully instills in them a little bit more memory of what it actually is and how to use it.
So as I’ve learned things like that to try to insert, and next semester I’ll teach this again, I’m trying to do it earlier on. So to some extent, again, the first half of a stats class, you still have to lecture a lot because it’s so much theory in the first half until you get to the application. Here’s how you actually run an ANOVA, but I can still try a little bit more.
So that’s one example. But with other concept, I’ll articulate, but I don’t have a clear vision of how this exactly plays out yet, is foreshadowing, actually. I think it’s interesting that one of the challenges of teaching intro stats is that there’s so much complicated stuff you have to simplify, right?
And this is related to what I said earlier, but just things like, we’re gonna assume normality of residuals, we have to assume that, and we don’t have time to get into all the details on what happens if it doesn’t work. And so I joke that we have to lie to our students to some extent in intro stats. We have to lie and say, this is how you do it, and then in the future, you’ll learn, here’s how not to do it, here’s how to change everything you just learned a little bit based on these other assumptions that you have to now check.
And I just realized that the foreshadowing comes in in trying to help the students know that there’s more to learn, but at the very least, they can take what they have right now and be able to do something with it.
[Richard Landers] (23:22 – 24:24)
So one area of NHST that I really worry about is related to Maslow’s hammer, right? Which is when you’re holding a hammer, everything looks like a nail. And there’s definitely been a long tendency, especially in psychology, but it seems pretty common in many fields, to just sort of automatically apply NHST in situations where it is not necessarily the best idea.
And I wonder now if we’re kind of experiencing the same problem in relation to AI solutions or generative AI solutions. And I know you had a study reporting in IO psychology, I think 30% of people were using it daily. So that’s a big influence potentially on the kind of things we’re doing and the products and the outputs we’re creating.
So yeah, I’m just wondering if you’re seeing or if you’re anticipating the same kind of challenge here. I’ve even seen people using gen AI, not just to help them figure out their stats, but to literally do their analysis for them. So are you concerned about that?
Where do you think we’re going?
[Steven Zhou] (24:25 – 28:52)
Definitely. I’m concerned in the sense of there’s so many ways to misuse it, right? And that’s the big concern.
Not concerned about it existing, or I guess it’s concerning, but it’s just a reality, okay? So I’m part of the crowd that would say, it’s here to stay, people are going to use it. So how can we use it the right way at the end of the day?
And a couple of thoughts on that. First is, even just in the intro stats classroom, so I put in my syllabus that I am open but cautious with AI. Like I think it’s necessary, it’s required, it’s part of life, but very cautious over how we use it and how we interpret it.
And at least when I tell my students, especially when they’re first learning our programming and learning how to analyze data, to not use it, and I try to incentivize them this way, I explain that most likely it’s going to spit out a bunch of analysis at you that isn’t really answering what we need you to answer in a psychology stats classroom or a social science stats classroom. And case in point, example is that it’s going to be trained on so much object-oriented programming out there that it’s going to do unnecessary for loops and basically borrowing from object-oriented principles that are, of course, great principles, but they’re just not applicable when you’re running a two-sample t-test. And so that’s one thing I tell the students, look, I can immediately tell if you’re doing it because your code’s going to be a lot messier, messy in our sense, in our psych stats sense of the term, and redundant.
And it’s not actually going to give you the answers we’re looking for, and it’s probably going to be a little bit off because it’s going to make some estimation differences versus how a t-dot test function would do it. And so I warn them that way, like it’s going to give you a right answer, but not the one that we’re trying to train you to do in this class. So that, I think, has worked so far.
I had no cases, as far as I could tell, for the first half of the semester where the students weren’t using AI, to my knowledge. Of course, they could be tricking me. I fully understand that.
But I think, I’m pretty sure they weren’t. Like, I look at their code and I can see that they’re doing the way I’m telling them to do the code and the way I’m teaching them and the textbook is teaching them, right? As we move into the second half of the semester, that’s when it opens up a little bit more because the students do need to understand that nowadays, when you’re coding, you’re going to have AI merged in with your coding platform.
That’s already true for the different platforms that we use today. I know it’s not as much in our studio right now, right? But in most platforms, like it’ll be right there, Copilot’s right there telling you how to code it.
And so it’s going to be there. How do you then use it? And how do you interpret it correctly?
And I’ve had moments where students would come to me and say, hey, I’m trying to interpret this interaction effect. And this is what Gemini said, I don’t get it. Like, it’s not helping me understand it.
And here’s where having them present their findings in front of the class is so helpful because they’re incentivized to say, they need to know it well enough to be able to articulate it. They can’t just be writing it into paper. They have to explain it.
I’ve had students come to me to say, hey, I use AI to explain it. I don’t get it, can you help me? And I’m really grateful for that.
Like, that’s what I want. I want to provide those with those opportunities to see AI is helpful, but you still need a human interpreting it effectively and correctly. And the other thought that’s somewhat disconnected from this class specifically, but just a broader topic on Maslow’s hammer in action is that in some ways, the hammer is going to be there.
So as faculty, we need to learn how to train people to use it the right way. We all have our friends working in applied contexts and I have friends working in applied contexts even outside of IOs areas, and they’re required to use AI. I had a friend, it was amusing, their job description literally said, you’re required to implement AI as part of your job.
And this was for like a basic contract negotiations job that you wouldn’t think is all that AI heavy. And so I guess the takeaway is, employers are requiring it and these students are going to graduate and they’re going to be in an internal world where they have to use AI. So unless we tell them how to use it, they’re going to engage in Maslow’s hammer to your point, Richard.
Like they’re going to just try to use it everywhere and have no idea when it’s accurate, when it’s not helpful, when it is helpful. And I think as faculty, we need to be the ones leading the charge and saying, this is how you’re supposed to use it so that they get into the workforce ready to implement.
[Richard Landers] (28:52 – 30:09)
And you know, the concerns here are always about the, we just had a faculty meeting two days ago about this, on the sort of the extreme cases, right? It’s much less talk about, well, how do we use this well in a day-to-day basis? It’s more like, oh, well, I heard about this story about this one person or this one student who submitted an article for publication and it got retracted later, because it turned out that they just asked AI to not only run their analyses, but to invent the data that the analyses would be based on.
And then the student claimed that they didn’t understand that that was like wrong. So that’s often like the scary stories and the backdrop is not, it’s not just, for me, it’s not exactly about incompetence related to AI, that’s always an undercurrent, but it’s also like grosser violations of ethics. And trying to figure out where that line is and also to convince, I found it challenging to explain to colleagues that those are different problems.
That it’s not that using AI is different than not understanding the AI is different from employing it in situations where it’s potentially damaging. These are different problems. They’re related, but they’re not the same thing.
Do you actually have these kind of ethics-ish conversations with students about how do you know for yourself when it’s okay, almost metacognitively or maybe?
[Steven Zhou] (30:10 – 32:57)
A little bit. And sometimes it’s about AI, sometimes not. Just the broader question of ethics in research and data science is really important to me and something we try to dive into.
So I’ll start with basics like data visualization. Here’s all the ways that a data visualization can go viral even though it’s cutting off the scale point or it’s manipulating the colors to make it look like one thing is more important than others and just basic stuff like that, right? So that kind of starts some of the conversation.
For the sake of brevity, let me jump to that specific example there and that you just gave. And I think what’s interesting there is I would say there’s personal ethics at play there, right? There’s a personal student does that or a faculty member does that, someone does that.
I personally would say that’s not as important in the sense of people are, they’re gonna find bad apples no matter what. So it’s less important to me to find ways to prevent bad apples. I’m more concerned with the infrastructure around the industry that is leading that to be successful.
If we have an infrastructure where a bad apple is not rewarded or is rarely rewarded, then okay, if someone’s gonna cheat, then they’re going to do it and we just have to have a system that makes it so they’re not succeeding or benefiting from that. So I guess my lens is more on the system. And so what my question there is, why would someone do that?
And I would say it comes down to they needed a publication. They needed something in that journal. And so that actually points me towards how is our system making it so that people are going to do that and are rewarded by doing that.
And that’s a combination of policy level decisions, of tenure criteria and how we evaluate scholarly output, which I don’t have any answers to. I just want to say that I think it’s problematic. It’s causing these problems, but I don’t know of a better solution.
There’s half that and then the other half is kind of back to our earlier one, public understanding of these issues. Even if it gets published, if the vast majority of journalists, of people reading these articles and citing them in news articles know the signs of bad ethics and immediately say, oh yeah, I’m not gonna cite that, I’m not gonna read that, it reduces the incentive to do such a thing. So it’s up to us to train the majority of people to get to the point where they’ll look at a paper and say, they AI generated the data and there may be cases where that’s appropriate, of course, but this is when it’s not appropriate and making sure people know that so they’re not citing it, they’re not reading it and they’re not referencing it in a news article or something like that.
[Richard Landers] (32:57 – 34:09)
The issue of incentives is a really complicated one. To me, it’s often, it’s a statement about community. It’s to say that we value X and we have created these incentives so that we get more of X in our community.
And we talk quite a bit in the book about how research in general is really about being a part of a group of people who have shared goals and are trying to understand something together using a shared set of values and frameworks and a way of understanding the world. That also plays out a little bit in the field at large. You know, you’re in charge of the ambassador program at the Sci-Op conference, which is a little bit of, I think, that same kind of idea.
It’s like a socialization into the conference, but also a socialization into the field, the community. And a lot of people will, I think, go to Sci-Op, to the Sci-Op annual conference, partly to learn, but partly to figure out like, is this a group of people that I belong to? Do they share my values?
Is this somewhere that I want to, something I want to be a part of? So I’m wondering what your thoughts are on that program. Like, how are you using, actually, I’m not totally sure when you started, just come to realize it.
[Steven Zhou] (34:09 – 34:11)
Last year, so my second year doing that.
[Richard Landers] (34:11 – 34:27)
Okay. So yeah, how have you been using that, or not, I guess, to try to build community? Like, what is your strategy for maybe incentivizing, but at least socializing people into the kind of values that you think our field should have?
About methods, but also more broadly.
[Steven Zhou] (34:27 – 37:22)
I hope it morphs into something like that. Admittedly, being my first year running the program, last year, it was just kind of do what’s been done before, so I could learn the ropes and understand how it functions. And on its own, it’s still, I hope, very encouraging.
So for those who don’t know, if you’re a newcomer to Sci-Op, this is your first time going to the conference, for free and no, it’s completely free. It’s part of the conference. Sign up for the ambassador program.
You’ll get paired with an ambassador, essentially a mentor. And we try to pair it based on interests, right? So we try to say, if you’re going and you’re interested in applied careers, we try to find an applied practitioner as your ambassador and an academic with academic.
And then even a little bit of area of interest within IO, but that’s, of course, harder to pair up on. But at the very least, academic, applied, or something else. And so we try to pair you up so that you are required to meet with them for 30 minutes on Zoom before the conference, and then you’re required to meet once during the conference, and we create space for that at a newcomer reception that we have, at coffee and cup connections that we have.
And so it’s just two touch points and that’s it. So as its own, that’s really what the program is. I hope this year we’re talking about developing it out a little bit more.
At the end of the day, there’s only so much we can ask, especially of ambassadors to give us their time. So we’re not expecting this to be a huge commitment, but to some extent, hey, a follow-up conversation, a guide to things you can talk about. In that guide, that’s where we can start inserting some of the, what is the community like?
What’s the socialization like here? And really trying to tackle, at least for me as an academic career path, the hidden curriculum, if you’ve heard that phrase of academia, right? And I was very lucky to be at George Mason, deeply embedded in the iosphere of academics and what people are and where they are and what they talk about and what journals and so forth.
And a lot of other places aren’t as much, right? And that’s not a fault of the quality of the program, that’s just the reality of the social network. And so those students at times get disadvantaged by just the basics of not knowing this person’s the editor of this journal and this is their background, this is what they like.
As simple as that actually makes a bit of a difference in an academic career. So hopefully, and again, that’s only on the academic side, I’m sure with applied practice, there’s just as much you could do there, helping newcomers get a sense of what that hidden curriculum is and exposing it so that they feel more confident and don’t have that imposter syndrome coming into the community. And so I hope that something like that is able to happen, again, recognizing it’s a limited time commitment, but at the very least, getting them to have a touchpoint with somebody is better than nothing.
It’s better than walking in completely blind to a conference. So I do hope every newcomer signs up for it and then that just makes my job harder of recruiting enough ambassadors, but I’ll deal with that challenge when the time comes.
[Richard Landers] (37:23 – 37:56)
Well, that’s great. And my only comment on that would be to return to our earlier thought of dream big. The only way we make change is through trying amazing cool things.
I’m excited to see what kind of changes you got in store. So yeah, this has been really great. To kind of wrap us up, I would love to leave our viewers with maybe a less hidden curriculum, an explicit curriculum, a specific paper, maybe in the method space, NHSC or otherwise, that you found maybe inspiring or just made a really big impression on you that you would recommend everybody take a look at.
[Steven Zhou] (37:56 – 40:06)
I mean, there’s so many good ones. I’ve also got a bit of a backlog, of course, of quite a few papers I need to go through. I get all these email notifications of new papers and I download them and save them to Zotero and I tag it as to read.
And that’s how I organize it. I need to read some of these. There’s been some really great work.
Of course, in the machine learning space and bias in training algorithms versus the algorithm itself versus the training data and so forth. So I love some of those papers. Let me give you one that’s a bit unique.
There was a paper that came out in SEM, the Structural Equation Modeling Journal, that was a teaching paper. And they developed a web-based platform for coding in Levon. I think it’s like Levon G-U-I or something like that.
And it’s a very cool drag and drop interface where you drag and drop circles and boxes and all the arrows and you actually visualize it and it will generate a Levon code for you. And I think it’s really cool. I think that’s really helpful as a teaching tool and I think I’m gonna use that for teaching next semester to help people understand, this is what the Levon code says, this is what you’re actually drawing out.
And it works both ways because then you can put in the code and you can visualize it as part of the paper you’re doing or you can draw it and play around with, well, what if I have this pointy here and this pointy here? I also think it actually, to get to the ethics topic, it also takes a nice step towards helping students realize all the degrees of freedom that exist in research itself. The phrase researcher degrees of freedom, I like to use.
Not saying that people are faking anything, some are, or that they’re intentionally crossing the wrong boundary, but that there’s a lot of flexibility in how you analyze a data set. Just simply by drawing the arrow the other direction and it takes such good, important theory and understanding of when you collect data, how you collected your data, what you’re actually measuring to be able to understand, no, the arrow needs to go this way. But I think this type of interface really makes it easy to just draw the arrow the other direction and it shows how it rapidly changes everything that you do in your analysis.
That’s an example of one that wasn’t a new method, it’s just a new platform that teaches methods effectively, I think. But there’s so many other methods papers we could talk about.
[Richard Landers] (40:06 – 40:11)
No, that’s fabulous. I think I need to find that. That’d be really useful.
[Tara Behrend] (40:11 – 40:28)
Yeah, please send it to us, we’ll link it in the notes. And as far as having a larger library of things than you could ever possibly read, that’s a virtue. Don’t ever worry about that.
I like to think of my unread library as my aspirational library. So don’t stress about that ever.
[Richard Landers] (40:29 – 40:35)
This has been really great. Thank you, Stephen, so much for taking the time with us. I think this will really help a lot of folks.
[Steven Zhou] (40:35 – 40:37)
Thanks so much for having me, it was wonderful.
[Richard Landers] (40:37 – 40:51)
That’s it for another gig. To stay in touch, subscribe on YouTube, check out our website at thegig.online, join our LinkedIn group, sign up for our email notification list and join our Discord. Thanks for joining us and see you next time for another great I-O Get Together.