This post may contain affiliate links. Please read my disclosure for details.

Why a Communication Research Class Quantitative Content Analysis Assignment?

In my opening post to the Spring 2018 semester, I reviewed several new assignments and activities I will be bringing into my classes this semester.

In this post, I will discuss the quantitative content analysis assignment that students will complete in my COMM 435 Communication Research course. The project simulates an analysis of earned media placement.

update – 12/3/18:  To see a follow up post to this post, check out Communication Research Class Media Placement Assignment, Part 2: Doing Data Entry and Creating a Data Legend.

[Here are all the posts about that class, including discussion of past projects].

Background: About my Communication Research Class

My aim with the communication research class is to offer our students experience learning about a variety of approaches to conducting research. My goal is to mix methodology (i.e., the study of research methods) with practical applications that students may run into in their careers. This course is not a graduate school prep course. It is designed for students who are planning to go into industry.

I think a struggle that many of us have is that there is a sense that we need to prepare students to be able to do the “new and cool stuff” (e.g., social media data analysis) in the research class, while balancing educating students about the research process, research ethics, designing measurements, building methods, gathering data, analyzing data, interpreting data, etc.

Unfortunately, we only have this one research class in our department as we are a small department serving a wide array of needs. I know that some other universities have advanced research or analytics courses. Thus, for me, I need to fit in both emerging methods and what some might see as traditional, evergreen methods:  content analysis, surveys, focus groups, interviews.

With this in mind, my students complete 3 assignments in the class, with each assignment focusing on a different topic. Each assignment is situated in a hypothetical, but plausible situation. I present the situation to the students via the assignment, and then we go through the steps of learning how to solve the problem put forth in the assignment. The assignments explore:

1. Content analysis of media artifacts (in the form of an analysis of earned media placement)
2. Social data analysis
3. Surveys, interviews, focus groups.

Each project is done in teams. This blog post will focus on project 1, content analysis of media artifacts.

As an side, if you’re interested in learning about the social data analysis assignment, last year I wrote a 4-part series on that assignment that I’ve gotten a lot of comments and questions about.

Project 1: The Set Up to Teaching Content Analysis

Learning research methods is a challenge for anyone new to it. Undergraduates sometimes express a strong aversion to the topic.

I’ve found that a content analysis of media artifacts is the most approachable method for introducing students to the systematic nature of doing research.

During the first few weeks of class, students are learning about research (e.g., the process, concepts such as reliability & generalizability, what research methods are, sampling, etc.).

After that, I introduce a hypothetical situation that the students will have to solve for their first project. Each year, I change up the situation a little bit. But the nuts and bolts have remained the same for the last 3 years.

I use the format from the Stacks book to set up the hypothetical situation students will address. You can see the entire text for the situation in the assignment at the bottom of this post. I will be referring to it in the paragraphs below.

The hypothetical is that the students work for an agency representing Netflix. Netflix is facing greater competition from other online streaming services like Amazon. To keep its competitive edge, Netflix is working to create shows that will appeal to a key market: 30-somethings. Stranger Things is one such show. Season 2 just launched.

Because the success of Netflix shows is widely influenced by critical acclaim from media, a media relations campaign was undertaken to position Netflix positively relative to itself competitors as a streaming service by way of the show.  The objectives of the campaign were to gain positive coverage of the premiere of season 2 of Stranger things.

The students enter the situation after the campaign has been executed and the campaign is now in the evaluation stage. Their job is to evaluate whether the media coverage was earned and what the nature of that coverage was.

Content Analysis Data Collection Activity

I used to gather a sample of news articles from LexisNexis and provide them to the students. This semester, the students will gather the data set themselves using the Meltwater social intelligence software. I’m excited about this because it gets the students into Meltwater and thinking about the use of the tool’s dashboard features. In addition, students are learning that they have the ability to pull down data for further analysis outside of Meltwater.

[You can learn more about the Meltwater University program in this blog post.]

The Meltwater software enables users to gather news articles from a given time period. Searches are conducted using keywords. Stranger things season 2 launched October 27, 2017. So the data set is built around the season premiere. I don’t have a strong research justification for the exact date range chosen. Rather, I chose it because it produced a manageable number of articles for each student to have to code.

Also, please note that I do not operationalize what “top news sources” are from the assignment objectives. Instead, for purposes of the exercise, I have students pick the top 5 sources related to their search results to analyze.

You can see the procedure for gathering the data via the Meltwater for Media Article Content Analysis lab guide I created.

Content Analysis Data Analysis for Students

There is a lot you could do with Meltwater to analyze the articles related to the launch of season 2 of Stranger Things.  If I had more time for this project, we’d dig into a lot of the dashboard tools. For now, students are only focusing on the quantitative content analysis of news articles.

I provide the students with details about specific research questions they are trying to answer related to media coverage: placement, share of voice, and whether or not the campaign’s 3 key messages made it into the press.

The data analysis is a simple quantitative content analysis of media artifacts. A simple coding sheet is provided. We discuss inter-coder reliability. And each student codes his/her media articles by hand using the coding sheet.

In class, we go over the coding sheet. And, in addition to the items on the coding sheet, which align with the research questions, students come up with their own item to code and to report in their paper. I do this exercise to get the students thinking about other things they could look for in the articles that might be useful.

It is worth noting that during a class activity earlier in the semester, students design their own coding sheets to evaluate car commercials and they learn quite a bit about the ups and downs of creating coding sheets. But, for the project, I create and provide the coding sheet. The operationalizations from the coding sheet are based on our class text, Paine’s “Measure what Matters: Online Tools for Understanding Customers, Social Media, Engagement, and Key Relationships.” For example, in class we discuss what we mean when we say that share of voice is “exclusive” or “dominant.”

While some of the coding can be done in class, students finish the coding as homework.

You can see the coding sheet that we will use this semester at the bottom of this blog post.

The students take their coded data and enter it into a spreadsheet so that we can quickly run frequency reports using SPSS.

Quantitative Content Analysis Assignment: The Write Up

With project 1, the write up that students produce is limited to providing a problem overview, the results, and a brief discussion section along with an appendix of their coding sheet. In the second and third projects students are asked to produce more and more of what a research paper might look like. But, because this is not an academic research class, I try to balance introducing students to a more academic style of research writing with a style that is more suitable to a report they might right in industry. I use a similar format to the format presented in the Stacks book.

I provide students with several handouts to help them write up their results.

One opportunity this assignment always presents, is a discussion about the limitations “simply measuring” the items on the coding sheet without looking at any context. As such, students are to go into their data and identify the features of the articles that support their results. Thus, they find example headlines and quotes to demonstrate, say, an example of a key message that was amplified.

Some Limitations

This project is, of course, limited in several rather important respects. However, I’ve created this project because it provides a great opportunity to introduce students to research, what a content analysis is, how to use a coding sheet, inter-coder reliability (invariably there are disagreements into how aspects of articles should be coded), and more.  Further, the project presents these learning opportunities within the context of learning a little bit about how one might evaluate earned media coverage. For example, students have learned about key messages in other classes. Now, they are learning about how those key messages may make their way into media articles and how the media represents them.

The project accomplishes this while situating the assignment in a campaign that is hypothetical but that is based on real events: Stranger Things Season 2 is real. Many of my students love the show and have watched it. They are reading real media articles about the show.  Further, students are situating this project as a campaign evaluation because the entire project is situated within the narrative that the students already executed the campaign and now they are evaluating it.

A Thought About the Key Messages Portion of the Coding Sheet

Each year I have changed up the Netflix show that we analyze and have thus changed the hypothetical backstory that accompanies it.

The key messages on the coding sheets are messages that I made up. They do not change much year to year, other than to bring them into the context of the show we’re evaluating. For example, this year I changed key message number two to emphasize the theme of nostalgia, which relates to the 30-something audience we are trying to target. I write the messages to be purposefully broad enough that they always end up achieving a good amount of frequency in the data set. The key messages are based on my general knowledge of Netflix. This year, it is possible that we won’t get many hits on key message number two. But, we’re bound to have some success with all three. However, if you choose to do this project and have some time, a better way to write the key messages would be to read through the data set ahead of time and develop them based on your own content analysis of the articles.

The next step is to teach students how to do data entry into a spreadsheet and to create a data legend. Read the follow up post in which I explain how to do this.

I hope that you found this blog post interesting and helpful. If you have ideas on how I can improve this project, please leave a comment or Tweet me. If you decide to use a version of this project in your own class, please stop on back and let readers know how it went or ways that you built upon it.

Don’t forget to check out the assignment below and the accompanying coding sheet.

Cheers!

– Matt

Project 1: Media Placement Assignment Handout

Project 1 Coding Sheet

Note: The hypothetical situation above uses the names of a real brand, Netflix, and its product. However, the situation is entirely made up and exists for educational purposes. Netflix logo is copyright Netflix.