Don’t look now, but the wheels of improvement are already in motion

430 responses.  Wow.

About 75% of Augustana’s full-time employees responded to the Augustana College Employee Survey over the last three weeks. Moreover, we got a great response from each segment of Augustana employees – faculty, staff, and administrators. I have to admit, after doing almost everything I could to encourage responses short of marching around campus in an sandwich board and chicken costume, I am thrilled. I would have been genuinely happy with 350 responses.

So … Congratulations! This means that the average response to each item is almost certain to closely reflect the perception of the entire employee population. As a result, we can be confident that whatever issues emerge from this data are not mere artifacts of the numbers we happen to collect. In addition, the quality of this data set will allow us to pursue all sorts of interesting analyses of various smaller segments of our employee population, further improving the potential for this study to help us legitimately improve the environment in which we all work.

Of course, this also means that if your earnestly held belief about a prevailing attitude among Augustana employees is contradicted by the findings of this study, you are going to be faced with a gnarly dilemma. Either you’ll have to accept the strong likelihood that your opinion has turned out not to be so, or you’ll have to present compelling evidence that refutes these findings. I suppose you could choose to double down on your belief, facts be damned, full speed ahead. But the reality of a 75% response rate means that, like it or not, the findings from this survey are pretty solid. And just so you don’t think that I’m trying to be some sort of righteous researcher revelling in my own rectitude (that line sounds great if you roll your R’s), I’ve already had to eat crow on one issue where the data makes it pretty clear that I was dead wrong. (Yes, it tastes about like what you’d think.)

Today (Monday, April 20th, 2015) we start collecting data for the second half of our employee engagement project. Later today I’ll send out an email with an invitation to participate in the The Gallup Employee Engagement Survey (otherwise known as the Q12 if you want to sound hip and “in the know” around other geeked out quant researchers). While our first survey was designed internally so that we could hone in on some important questions specific to Augustana College, the second survey, the Q12, gives us some comparison data that can function as a sort of grounding point to more realistically assess ourselves. Moreover, we will be able to get data from Gallup that we can use to compare ourselves to other educational organizations, giving us an even better sense of how we might realistically improve. The Gallup Q12 Survey is built on several decades of in-depth research on employee engagement. Some of the questions might strike you as unusual at first, but know that a virtual ocean of analysis has gone into developing the questions that compose this survey.

And in order to ensure that we don’t unintentionally bias the responses to the Q12, we won’t publicly release the results of our own Augustana Employee Survey until the Gallup data has been collected.  Even though the questions in both surveys are not identical, there is enough overlap that we need to be careful. Beside, this will give me a few weeks to process all of the data and turn it into something that will be a lot easier to read. As much as my inner quant geek would love it, I suspect that you don’t want me to send you a massive excel spreadsheet and call it good!

You will receive an email soon with a link to participate in the Q12. You’ll hear about this survey from me more than a few times in the next three weeks. Just like the first half of this project, your participation in the Q12 matters immensely.

430 responses to our first survey makes a giant statement about how much we value making Augustana a great place to work and a great community to join. It also means that this community made the collective commitment to improve – even if you did not individually complete the first survey. Whether you like or not, the improvement train has left the station and we’re all on it.

Make it a good day,

Mark

The race to get old started yesterday. Hurry up!

A little over a week ago the Wall Street Journal published a short piece entitled, “Today’s Anxious Freshmen Declare Majors Far Faster Than Their Elders:Weak job market and high debt loads prompt broad shift away from intellectual exploration.” They cited data from their own small but random survey of colleges and universities suggesting that more and more freshmen declare their majors earlier. While the article and those interviewed for it speculated about a variety of factors that might be driving this phenomenon, the conclusion seemed pretty clear: college is now much less about discovering yourself first and finding a career later and much more about locking into a track for a career.

I thought it would be interesting to see if our own data reflected a similar trend. We were able to examine data over a similar time period, exploring the differences between students who entered Augustana as freshmen in the fall of 2007 and students who entered Augustana as freshmen in the fall of 2013. In addition, I thought it would be interesting to expand on the Wall Street Journal analysis since they aren’t clear about when the institutional data they presented was collected (in the fall of the first year? in the spring of the first year? at the beginning of the second year?). So we compared the two freshmen cohorts noted above in three ways. First, what proportion of the class indicated that they were undecided on their major when they applied to Augustana? Second, what proportion of those undecided students had declared a major by the beginning of their second year? And third, what proportion of the entire freshman class had declared a major by the beginning of the second year?

Our Augustana results seem to parallel the findings reported by the Wall Street Journal. During the application process, 16% (111 of 713) of the 2007 first-year cohort indicated that they were undecided about their major. During the 2013 cohort’s application process, only 11% (70 of 627) selected “undecided” when asked about their intended major. Interestingly, the proportion of these initially undecided students who had chosen a major by the beginning of their second year did not change appreciably between the fall of 2008 and the fall of 2014. Of the undecided majors from the 2007 cohort, 68% (63 of 92 – the remaining 19 did not persist to the second year) had still not selected a major one year later.  From the 2013 cohort, 69% (40 of 58 – the remaining 12 did not return to Augustana) of the initially undecided remained undeclared.

The biggest difference between the two cohorts can be found in the proportion of students who had declared a major by the beginning of the second year. Remember, the position taken by the Wall Street Journal article was that students take less time for intellectual pursuits and narrow their focus on a major earlier than in previous years. At Augustana, It appears that we are seeing a similar phenomenon.  While 54% of the 2007 first-year cohort had not yet declared their major by the beginning of the second year, only 36% of the 2013 cohort were still undeclared majors by the beginning of the second year.

So . . . is this a bad thing?

Honestly, I’m not sure.  In the end, I don’t know that we will have much success telling students that they are wrong to respond to external pressures of a tight job market and high student debt by choosing their major earlier. That kind of approach is likely to come across as tone-deaf to some very real concerns. It seems to me that this data re-emphasizes the importance of timely and substantive conversations between students and all of us who impact their education (faculty, administrators, work supervisors, residence life staff, student life staff, and fellow students) that push students to develop themselves even as they are preparing for life after college. Personal and intellectual development and career preparation ought to be a “both/and” enterprise.

If we can do that, our students are likely to grow and change in just the ways that we hoped they would.

Make it a good day,

Mark

What if our students could point to their most important learning moments?

If we could make a college education work perfectly, our students would do more than learn. In addition, they would be able to point to those actual moments during their college career when an interaction, an experience, or a discovery altered their trajectory regarding their plans for life after college. Although this might sound a little dreamy aspirational, it turns out that students who can talk about their learning experiences in this way tend to have a sort of educational momentum that seems to set them apart from their peers. These are the students who do the little things to put themselves in the early running for advantageous opportunities that ultimately lead to a deeper sense of purpose and direction as well as stronger job applications and stronger graduate school applications. These students make folks like me wish I had had some of what those students have when I was their age.

That’s why it makes a lot of sense to find out what proportion of our freshmen have this kind of perspective after their first year at Augustana. Ideally, we’d like to be able to cultivate that deeper level of awareness in more of our students by figuring out if there are ways that we could make this happen in more of them. So at the end of the year we ask our freshmen to agree or disagree with the following statement: “Reflecting on the past year, I can think of specific experiences or conversations that helped me clarify my life/career goals (e.g. conversations with faculty/staff, organized activities with other students, community involvement, specific classes, etc.).”

Here’s how last years’ freshmen responded (remember that not all freshmen completed this survey):

  • Strongly disagree –  9 (4%)
  • Disagree          -     15 (6%)
  • Neutral           –       64 (28%)
  • Agree            -       100 (43%)
  • Strongly agree    -   44 (19%)

My reaction to this bit of data is a little mixed.  On the one hand, most of the students either agreed or strongly agreed with this statement.  On the other hand, 87 of our respondents can’t seem to put themselves in an affirmative category.

To be fair, it would be a little naive to think that we could hand out inspirational moments like some kind of kitschy swag. At the same time, it would be awfully useful to know whether there are things we could do to increase the likelihood that a given freshman would say that they could point to a specific experience in their first year that helped them clarify their life or career goals.

After testing a host of possibilities, we found five items that significantly increased the likelihood of this perspective among our freshmen. Interestingly, in addition to a set of experiences that come from all facets of a residential college life these items indicate a certain type of experience that provides some guidance for our work.  Here are those five items:

  • How frequently did your faculty ask you to try to understand someone else’s views by imagining how an issue looks from his or her perspective?
  • My instructors recommended specific out-of-class experiences that would enhance my learning and growth.
  • My adviser asked me about my career goals and post-graduate aspirations.
  • My out-of-class experiences helped involve me in community service off-campus.
  • About how often have you had serious conversations with students who are very different from you?

Again, as we’ve found in other analyses of our student data, the ideal college experience depends upon the work that each of us do, no matter if it is inside or outside of a classroom. But today I want to highlight the role of faculty reflected in these items. Instructors who often ask students to practice perspective-taking in order to better understand someone else’s views, instructors who take the time to recommend specific out-of-class learning experiences, and advisers (in other words, faculty who are first-year advisers) who ask students about their career goals and post-graduate aspirations all appear to significantly contribute to the quality of our students’ educational experience. Students who experience these kinds of faculty interactions seem to be more likely to be able to point to specific moments in their first year experience that helped them hone in on their post-graduate goals.

The other thing I like about this list of faculty interactions is that, no matter the course or the discipline, at least one of these items seems possible. If your course doesn’t lend itself to perspective-taking exercises, you could point students toward particularly valuable educational experiences on campus or in the community. If your class is composed of students who are already highly involved, you could engage them in perspective-taking skill development. And when students engage you outside of class, you could take a moment to ask them about their life goals beyond college.  I hope you will consider finding a way to plug one of these items into your regular interactions with students.  Good luck with your spring term!

Make it a good day,

Mark

We’ve Still Got a Long Way to Go

Most of the time, I try to write a post that includes both a deep dive into some morsel of data and a few implications that I think might be embedded in that data.  But this week, I think I’m going to try to dispense with a longer examination of implications and just lay out a set of responses to a single question from our recent survey of prospective students that we conducted in collaboration with the Hanover Research Group.

An early question in the survey asked the respondents to select the top five words that best described the college they would most like to attend.  You might recall that last week I pointed out that “affordable” was the most frequently selected word (not a big surprise, right?) and that “liberal arts” was pretty far down the list.

Although it’s certainly interesting to see the ordering of selected words from highest to lowest, it’s also potentially enlightening to look at how different subgroups of respondents respond to similar words. Parsing the responses of white and non-white respondents exposes a stark difference worth noting.

In digging deeper in the responses to this same question, the disparity between white and non-white respondents in selecting the word “diverse” really jumped out at me. White respondents selected this word 15% of the time. Non-white respondents selected this word 46% of the time. Given the substantial demographic shifts that are already underway across our primary recruiting region, this difference seems particularly important.  In addition to the moral imperative for us to continue to diversify our student body, it appears that ignoring such an imperative could increase our future economic risk as well.

While this finding is interesting, asking respondents to choose their top five words from a long list of possible options can complicate the interpretation of the results. So I want to show you another set of responses, parsed by white and non-white respondents, to a very specific question that asks respondents to indicate how important a diverse student body is to them when selecting a college.

Response Option                                                                        White              Non-White

Not at all important 11% 2%
Slightly important 19% 3%
Moderately important 40% 27%
Very important 25% 38%
Essential 5% 31%
Very important + Essential 30% 69%

As you can probably tell, non-white respondents trend toward thinking that a diverse student body matters a lot.  By contrast, it appears that white respondents trend toward thinking that a diverse student body matters some, but not nearly as much.

Yes; there is probably more than one reason for this difference in responses. And it’s not as if the difference between the two sets of responses are in complete opposition to each other. But, I hope this data will further underscore the reasons why we need to be active champions for equality. We’ve still got a long way to go.

Make it a good day,

Mark

Who are we talking to when we use the term “liberal arts”?

If a complete stranger had stumbled onto campus the weekend before last they might have thought that Augustana was the busiest college on the planet. That Saturday (January 17th), the Admissions Office hosted one of our largest annual open-house events for prospective students and families. While this event always draws large numbers, this year the number of visitors to campus (prospective students and their parents combined) may well have exceeded the actual number of Augustana students living on campus.

With the college recruiting season hurtling into the most critical few months of the year, every little bit of information that we can learn about prospective students and their parents and their decision-making process matters. To that end, we’ve been gathering data on the things that are most important to our prospective students and their parents as they evaluate, and ultimately select, a college. One way that researchers try to get at this kind of information is to ask folks to pick five words or phrases from a longer list of words or phrases that they think best describe an idyllic college experience. As you might expect from this year’s prospective students, “affordable” topped the list with 57% of the respondents choosing it.  Other words near the top of the list included “friendly” (41%), “safe” (39%), “respected,” (38%), and “career-oriented” (33%).

Much further down the list, 15th to be exact, sits the phrase “liberal arts” (just 12% of respondents thought this was a top-five word for them). Since rank ordering the words selected ends up clustering “liberal arts” with a seemingly contradictory group of terms (e.g., “small,” “large,” “rigorous,” and “flexible,”), it’s clear that we probably  shouldn’t go all Chicken Little just yet. Look on the bright side: only 6% of the respondents selected “party school.”

The question this finding raises for me, however, isn’t really about the exact ranking of the term “liberal arts.” My concern is that there seems to be a substantive gap between the degree to which we (faculty, staff, administrators, board members) use the phrase “liberal arts” to describe who we are and the level of importance that prospective students responding to this survey gave it. To make matters worse, this data doesn’t come from some general survey of potential college-going students; these responses came from students in our own inquiry pool (i.e., students who have either contacted us directly or students who fit a profile of those who might be interested in us).

Now please don’t conclude that I’m suggesting the elimination of the term or the philosophy behind it. On the contrary, I happen to think that if we are going to remain a viable college then we will have to explicitly embody a liberal arts philosophy that focuses on integrating and synthesizing preexisting knowledge. Almost exactly a year ago, I went on a three-post rant about it here, here, and here.

Rather, I suspect that the term “liberal arts” means very little of substance to prospective students. Maybe it is, like many other words that get used over and over again in marketing materials, a case where the phrase means one thing to an internal audience and something else to an external audience. When we use the term, even though we might not all agree exactly, I think we could describe relatively precisely the dispositions of a liberally educated individual.  This finding increases my worry that when an external audience, most notably prospective students, sees this term, they have a much less precise sense of its meaning. In that context, “liberal arts” might mean little more than “small” or “rigorous.” It also could end up being interpreted to mean “lots of classes in fields I’m not interested in” or, even worse, “a club that maybe we’ll let you into.”

I certainly don’t have a brilliant answer to this challenge. But I think it is worth noting that just because we have a term that we believe describes us well doesn’t mean that this term will compel others who are new to the concept of college to buy what we are selling. There’s nothing wrong with believing in what we do; even drinking our own Kool-Aid. We just better be able to spell out what we do and why it works in a way that makes sense to regular folks who seem to care a lot more about affordability.

Make it a good day,

Mark

Setting a high bar for equality in graduation

US News rankings have never been my favorite part of higher education. For many years these rankings did little more than con colleges and universities into an illusory arms race under the guise of increasing educational quality. But recently US News has started to use their data, power, and influence to prod more useful conversations that might lead to improvements at higher education institutions. Last week, US News released their rankings for “Which top-ranked colleges operate most efficiently.” Like last year Augustana appeared near the top of the list among liberal arts colleges, suggesting that we apply our limited resources effectively to educate our students. Whether conversations about “efficiency” give you a warm fuzzy or a cold shudder, I don’t think it’s particularly controversial to say that such recognition is, at the very least, more good than bad.

But in keeping with their deified status in higher education, the US News rankings giveth and the US News rankings taketh away. A few weeks ago, they released another set of rankings that I found particularly intriguing given our recent campus discussions about equality and social justice. This set of rankings focused on the graduation rates of low-income students, and contrasted the proportion of low income students who ultimately graduate from each institution with each institution’s overall graduation rate. Based on these two numbers, US News identified colleges and universities that they called “top performers,” “over performers,” and “under performers.” Sadly, Augustana appeared in the under performer group with a 13 percentage point deficit between our overall six-year graduation rate (78%) and our six-year graduation rate of low-income students (65%). Just in case  you’re wondering, these graduation rates come from students who entered college in the fall of 2007.

Because of the focused nature of this particular analysis, US News combined all institutions from their two national ranking categories (national universities and national liberal arts colleges) to create these three groups. The presence of several familiar institutions in each group suggests that there might be something to learn about graduating low-income students from other similar institutions that might in turn help us narrow our own disparity in graduation rates. 

The criteria for the “top performer” category required that the institution’s overall graduation rate was above 80% and that the graduation rate of low-income students was the same (or within a percentage point). While there were numerous national liberal arts colleges on the list, they were generally highly ranked institutions with well known pedigrees. However, two familiar institutions appeared in this category that seemed worth highlighting.

  • St. Olaf College – overall grad rate: 88%, low-income grad rate: 87%
  • Gustavus Adolphus College – overall and low-income grad rate: 82%

The criteria for the “over performer” category was simply that low-income students graduated at a higher rate than the overall student population. There were several institutions in this group that are not too different from us, particularly based on their US News overall ranking (remember, Augustana was ranked #105 this year).  These institutions include:

  • Drew University (#99) – overall grad rate: 69%, low-income grad rate: 76%
  • College of the Atlantic (#99) – overall grad rate: 69%, low-income grad rate: 75%
  • Knox College (#81) – overall grad rate: 79%, low-income grad rate: 83%
  • Lewis & Clark College (#77) – overall grad rate: 74%, low-income grad rate: 79%
  • Beloit College (#61) – overall grad rate: 78%, low-income grad rate: 83%

Interestingly, there were also some institutions in the over performer group that probably wouldn’t dare to dream of a ranking approaching the top 100. In other words, they would probably trade their place for ours in a heartbeat. A few to note include:

  • Oglethorpe University (#148) – overall grad rate: 62%, low-income grad rate: 67%
  • Illinois College (#155) – overall grad rate: 64%, low-income grad rate: 68%
  • Warren Wilson College (#165) – overall grad rate: 51%, low-income grad rate: 60%
  • Ouachita Baptist University (#176) – overall grad rate: 60%, low-income grad rate: 80%
  • Wisconsin Lutheran College (#178) – overall grad rate: 64%, low-income grad rate: 75%

Finally, the under performer group noted institutions where low-income students graduated at rates lower than the overall graduation rate. Some similar/familiar liberal arts colleges in this group included:

  • Augustana College (#105) – overall grad rate: 78%, low-income grad rate: 65%
  • Washington College (#105) – overall grad rate: 68%, low-income grad rate: 49%
  • Hampden-Sydney College (#105) – overall grad rate 62%, low-income grad rate: 43%
  • St. Mary’s College of Maryland (#89) – overall grad rate: 73%, low-income grad rate: 64%
  • Wittenberg University (#139) – overall grad rate: 63%, low-income grad rate: 49%
  • Alma College (#139) – overall grad rate: 61%, low-income grad rate: 44%

Although we ought to be careful not to jump to rash conclusions from this data alone, there are a couple of suppositions that this data seems to contradict. First, although the national graduation rates for low-income students consistently lag behind overall graduation rates, this is not necessarily so at every institution. Some institutions graduate low-income students at substantially higher rates than the the rest of their students. Second, it does not appear that institutional wealth, prestige, or academic profile guarantees graduation equity. There are institutions at both ends of the ranking spectrum that manage to graduate low-income students at a higher rate than the rest of their students. Third, geographical location doesn’t necessarily ensure success or failure. Successful institutions are located in both urban and rural locations.

I don’t know what makes each of these successful institutions achieve graduation equality. But in looking at our own disparity in graduation rates, it seems to me that we might learn something from these institutions that have found ways to graduate low-income students at rates similar to the rest of their students. We have set our own bar pretty high (our overall graduation rate of 78% is comparable or higher than all of the institutions I listed from the US News over performer category). Now it’s up to us to make sure that every student we enroll can clear that height. We shouldn’t be satisfied with anything less.

Make it a good day,

Mark

Some Key Findings from our Recent Alumni Survey

Every once in a while you get lucky enough to have multiple studies that all point pretty clearly to the same conclusions.  So in the spirit of Christmas, I give you a gift of confirmatory evidence that all of what we do at Augustana – in the classroom and outside of it – matters for student learning.  Special thanks should go to my student assistant, Melanie, who did all of the data analysis and even wrote the first draft of this post.  Thanks, Melanie!

The Recent Alumni Survey asks a cohort of graduates about their experiences in the nine months since they walked across the stage to receive their diploma. Three items in this survey are designed to get at some of the intended outcomes of an Augustana education.  Those items ask:

  • To what extent do you feel your Augustana experience prepared you to succeed in your current program?
  • To what extent do you feel your Augustana experience prepared you to succeed in your current position/job?
  • To what degree does your current professional/educational status align with your long term career goals?

The first two questions address our graduates’ perception of the quality of their preparation for their next step in adult life, be it graduate school or their first foray into the world of work. Because we care about the full arc of our graduates’ adult lives, the third question addresses the degree to which that “next step” – the one for which our mission demands that we play an important role in preparation and selection, aligns with their long term goals.

To help us improve the quality of an Augustana education, we want to determine the nature of the relationship between college experiences that we already believe to be important (gleaned from our last senior survey) and our graduates’ lives nine months after they graduated. To this end, we linked responses from our 2013 senior survey and same individuals who responded to our recent graduate survey in the winter and early spring of 2014. After identifying which senior survey items significantly predicted (in a statistical sense) these recent alumni outcomes, we expanded our analysis to account for several factors that might confound our findings: race, socio-economic status, gender and cumulative GPA. The table below shows the experiences that emerged as statistically significant positive predictors for each outcome organized by the nature of the environment in which those experiences exist.

  To what degree does your current professional/ educational status align with your long term career goals? To what extent do you feel your Augustana experience prepared you to succeed in your current program? (asked of alums in grad school) To what extent do you feel your Augustana experience prepared you to succeed in your current position/job? (asked of alums in the workforce)
Co-curricular Experiences -My out-of-class experiences have helped me develop a deeper understanding of myself -My out of class experiences helped me develop a deeper understanding of myself*-My out of class experiences helped me develop a deeper understanding of how I relate to others
Advising - How often did your major adviser ask you about your career goals and aspirations? - How often did your major adviser ask you about your career goals and aspirations? - How often did your major adviser ask you about your career goals and aspirations?-My major adviser connected me with other campus resources
Experiences           in the Major -Faculty in this major cared about my development as a whole person-In this major, how frequently did your faculty emphasize making judgments about the value of information, arguments, or methods, such as examining how others gathered and interpreted data and assessing the soundness of their conclusions?
Overall Curricular Experience -My one-on-one interactions with faculty have had a positive influence on my intellectual growth and interests in ideas -My one-on-one interactions with faculty have had a positive influence on my intellectual growth and interests in ideas

Clearly there are multiple experiences across a range of settings that influence these three outcomes. Moreover, these findings are similar to the results of prior alumni data analyses and replicate findings from analyses of senior survey data.  In short, we can be confident that the experiences noted in the table above play a critical role in shaping the success of Augustana graduates.

These findings strongly emphasize the importance of quality and purposeful faculty interactions with students. The item, “my one-on-one interactions with faculty have had a positive influence on my intellectual growth and interests in ideas,” significantly predicted students’ sense of preparedness for both those entering graduate programs and those who went into the workforce. This item focuses on more than the frequency of students’ interactions with faculty or friendliness of those interactions. Instead, this item emphasizes the nature of faculty influence; encouraging, inspiring, cajoling, pushing, prodding, and even challenging students to engage tough questions and complicated ideas while at the same time supporting students as they struggle with the implications and ramifications of their own evolving values, beliefs, and worldview.

Faculty influence was again evident in the advising relationship. The question, “How often did your major adviser ask you about your career goals and aspirations?” significantly predicted all three outcomes. In addition, for graduates in the workforce faculty attention to connecting students with other campus resources also influenced the graduates’ sense of preparedness. Furthermore, faculty impact on our graduates’ success is apparent in the major experiences that predicted students’ sense of preparation for their career. Two items were significantly predictive: “Faculty in this major cared about my development as a whole person,” and “In this major, how frequently did your faculty emphasize making judgments about the value of information, arguments, or methods, such as examining how others gathered and interpreted data and assessing the soundness of their conclusions?” In addition to confirming the caring aspect of quality and purposeful faculty interactions with students, this finding also highlights the value of classroom experiences that cultivate higher order thinking skills.

It is also worth noting the importance of out-of-class experiences in predicting our graduates’ success. Again, the importance of the developmental quality of these experiences is paramount. Instead of items that denote participation in particular types of organizations or activities, the items that proved predictive emphasize that the experiences that matter are ones that help students develop in two ways. First, they help students develop a deeper understanding of themselves.  Second, they help students develop a deeper understanding of how they relate to others. Obviously, these skills are critical for success in every manner of adult life.  The key for Augustana is to ensure that every out-of-class experience contributes – directly or indirectly – to this kind of growth.

The goal of this analysis was not to determine which experiences (faculty interactions or co-curricular experiences) play a larger role in shaping Augustana graduates’ outcomes. Instead, it is clear that all facets of the Augustana education contribute to our students’ success.  It is also clear that not all graduates experience Augustana in a way that maximizes the potential impact of quality and purposeful faculty interaction or developmental out-of-class activities.  Throughout the institution, we can use these findings as principled guidelines to improving the work that we do with our students.

Make it a good day (and a great holiday break),

Mark

When does a stereotype lose it’s margin of truth?

As we tumble down the back side of the fall term, I know that the potential value of a long and involved blog post drops like a stone in the face of the ten things that have to be solved right now(!).  So I’m gonna just roll out one big-picture data point and let you mull it over when you get a chance to breathe.

Remember that economic collapse that wrecked the economy and scared the pants off of tuition dependent college like us?  Yeah, sorry – not the best way to start the week.  Although there are a few reasons to think that the American economy might be slowly pulling out of its nose dive, we all know that the ripple effects haven’t abated much.

I’ve been trolling our three years of senior survey data lately to look for trends that might be worth noting (see last week’s post on a couple of general education items).  This morning one set of numbers really jumped out at me; the increasing proportion of our students who qualify for Pell Grants – a group of students who almost certainly wouldn’t be at Augustana if it weren’t for need-based financial aid from the federal government and the state of Illinois through the MAP grant program.

The table below shows the increase in this aspect of our student population over the past three classes.  I’ve included the actual numbers in parentheses to add some perspective.

Cohort % Pell Recipients as entering freshmen % Pell Recipients as graduating seniors
Fall of 2008 – Spring of 2012 12.5% (80/641) 10.7% (51/476)
Fall of 2009 – Spring of 2013 17.5% (108/616) 14.2% (66/465)
Fall of 2010 – Spring of 2014 24.6% (185/753) 22.4% (119/532)

You can see that over a relatively short period of time, we’ve roughly doubled the proportion of students who receive Pell Grants (and therefore also received MAP Grants from the state of Illinois).  If you look closely, you can see a hint of this lower socioeconomic status on retention and graduation, since the proportion of these students shrinks over the course of four years.

Just in case you were wondering, 26.2% of our student body (655/2500) of our current overall student body is receiving a Pell Grant.  And among our newest class of freshmen, 28.1% are receiving Pell Grants.

The headline of this blog referred to the stereotype that we like to throw around about our students coming from the Chicago suburbs.  Sometimes I hear that stereotype dressed up with healthy dose of wealth, homogeneity, and entitlement.  I’m not trying to suggest that those students don’t exist at Augustana.  Of course they do.  But the thing about these numbers that hit me was this:

  • We have a trend in our data that suggests a growing proportion (now over a quarter of our student body) of our students that, at least on one critical dimension, don’t conform to that stereotype.
  • Are we adapting our expectations and interactions with our students to match what we know about from whence they come?

Make it a good day,

Mark

What is the role of general education? Some ominous shadows in the data

Despite a genuine commitment to a liberal arts mission, at times it seems easier said than done. On one hand, public fretting (some of it well founded) about unemployed and apparently unemployable college graduates has made some suggest that a college education should focus more of its coursework on preparation for a specific career. On the other hand, the proliferation of knowledge and sub-disciplines within many academic fields translates into more knowledge that faculty often believe (sometimes rightly) need to be added to the range of concepts covered within a particular major. Both of these tangible pressures bolster the argument for expanding the footprint of the major. By comparison, the counter-arguments for maintaining a robust general education program tend to be more abstract and sadly, rarely stand a chance.

Two trends (one macro and one micro) highlight the declining clout of general education. First, the number of U.S. colleges classified as liberal arts colleges has dropped substantially in the last several decades (from 212 to 130). Most of this change involves institutions that expanded their educational offerings into more vocational and pre-professional programs. Closer to home, the proportion of Augustana students who earn at least two majors continues to increase (over 45% of graduates in 2014). In the case of Augustana students, our double majors don’t stay in college longer than everyone else, they just concentrate more of the credits they earn in specific areas.

During last year’s conversations about the relative impact of general education and potential improvements that could be made, some seemed to suggest that our general education program was not in need of revisions. One of the questions posed was whether or not our senior survey data might provide evidence to inform this conversation. Now that we have a third year of senior survey findings, I thought it might be useful to explore the responses to the survey’s general education items and look for any patterns or hints of trends. I’m not sure that the findings below provide definitive answers, but I hope they will further inform the discussion and direction of the general education conversation at Augustana.

The Augustana Senior Survey includes six questions intended to assess the nature of students’ experiences in their non-major or general education courses. Interestingly, the lowest average response score over the last three years came from the 2014 seniors on five of the six questions. Further analysis indicated that the drop from highest to lowest score was statistically significant for four of those questions. They are listed in the table below.

Senior Survey Gen Ed Question 2012 2013 2014
The skills I learned in my general education courses helped me succeed in my major courses. (response options – strongly disagree, disagree, neutral, agree, strongly agree) 3.55 3.43 3.38
My classes outside my major(s) challenged me to produce my best academic work. (response options – strongly disagree, disagree, neutral, agree, strongly agree) 3.57 3.53 3.44
In your non-major courses, about how often were you asked to include different perspectives (different races, religions, genders, political beliefs, etc.) in class discussions or writing assignments? (response options – never, rarely, sometimes, often, very often) 3.50 3.52 3.41
About how often did you discuss ideas from your non-major courses with faculty members outside of class? (response options – never, rarely, sometimes, often, very often) 2.88 2.82 2.76

I fully admit that three years of data is not nearly enough to make predictive claims or produce some sort of smoking gun. However, it is enough data to begin triangulating these findings with others (everything from observational to rigorously quantitative data) and look for evidence of multiple findings moving in the same direction. This can be a particularly effective way to identify early “shadows” in the data and give us time to consider their implications in a less reactive environment.

It would be entirely reasonable to expect some fluctuation on average response scores for individual items across multiple years. But it struck me as curious that the responses to so many of the general education items – questions that I think represent the way that we imagine our general education courses functioning at a liberal arts college – moved together in a negative direction.

What might explain this phenomenon? Is it a function of our students feeling an increased pressure to focus on career preparation? Could it be a function of our own subtle leanings toward areas of our own expertise? Or could it be that we lack a clear sense of exactly how our general education requirements link together to form the kind of integrated breadth of understanding that would ultimately produce the ideal liberally educated student?

I don’t know the answer to any of these questions. But these findings did make me think again about our discussion of the role of general education and the degree to which we may need to revisit our commitment to 1) the role of general education and 2) the way we ensure that our general education program helps students develop all of the knowledge, skills, and dispositions that we know are critical to their success after graduation.

Make it a good day,

Mark

 

Retention, part deux: Freshman survey data to the rescue!

In last week’s Delicious Ambiguity post we dove into the deep end of the data pool regarding retention and the complexity of the problem. Our institutional data shows that the factors shaping our student’s decisions to withdraw or persist are influenced by characteristics that students bring with them to college as well as experiences that they have during their first year. Moreover, it’s clear that addressing this issue requires “all hands on deck” if we are to make any demonstrable progress.

But knowing that information by itself leaves us far short of actually knowing what to do differently. For us to improve our retention rates we need to know which student experiences matter most in shaping their decision to persist. We need to identify specific experiences over which we have substantial and concrete influence. Information about more general experiences, even if they are specific to one aspect of the college experience, is not enough. For example, both of the items below predict our students’ general sense of belonging on campus.

  • “My day to day experiences in my residence hall have helped me feel like I fit in at Augustana.”
  • “I know that my Community Adviser (CA) cares about how I am doing at Augustana.”

Although both findings might appear interesting, the item addressing our CA’s impact on students’ sense of well-being is more specifically prescriptive, providing tangible guidance for designing the role of CAs as well as the way that we select, train, and assess their efforts.

Similarly, if we can collect and link granular experience data to bigger picture retention data, we will be more likely to glean specific direction from our data analyses that ultimately helps us improve retention. This was our big aspiration when we altered our freshman survey’s design last year, gathering more specific experience data about academic acclimation and social integration midway through the first year. After analyzing last year’s responses, I’m excited to share several specifically actionable findings that appear to increase the likelihood of persistence.

After last week’s examination of some retention trends involving race, gender, socioeconomic status (SES), and pre-college preparation, we applied those findings to run some statistical models that combined data from our institutional data, our student readiness survey, and our mid-year freshman survey. All of the findings I share below hold true after taking into account race, gender, SES, pre-college preparation (ACT). We then added items from the freshman data that might influence retention above and beyond those four pre-college characteristics. In the end, two items produced statistically significant effects.

The first item producing a statistically significant positive effect on retention addressed a very specific aspect of the LSFY/Honors experience.

  • My LSFY/Honors instructor helped me develop at least one specific way to be a more successful college student.

Just as so many other researchers have found previously, students often need guidance in figuring out how to successfully navigate college. This means so much more than just knowing how to react to trouble in a class or with a roommate. Instead, this means knowing how to take control of the experience in order to make the most of it in preparation for life next month, next term, or next year.  Our finding suggests that LSFY and Honors instructors who taught students at least one specific way to proactively engage college as a student actually contributed directly to student persistence. And this finding held regardless of incoming ACT score, suggesting that this kind of learning is valuable for all students no matter their pre-college academic preparation.  As LSFY continually explores ways to make that course more effective, this finding seems well worth incorporating into that discussion.

The second item that produced a statistically significant effect addressed a more general sense of social integration.

  • I feel like I belong on campus.

Although this is interesting, we needed to dig further to come up with more concrete guidance toward future improvement. After peeling back another layer of the onion, we found that very kind of guidance.

This time, in addition to accounting for race, gender, SES, and ACT score, we decided to add comfort with social interaction to the mix. Interestingly, in the end comfort with social interaction still produced a statistically significant positive effect, suggesting that despite everything that we might do to influence students’ sense of belonging, more reserved students are likely to still feel less of a sense of belonging than more outgoing students at the midpoint of the first year. However, this effect appears to vanish by the end of the first year, supporting the contention that more reserved students may simply need more time to find their niche on campus.

Under these analytic conditions, we found granular experience guidance for faculty, both as instructors and as advisers, and for student affairs professionals that appear to influence student’s sense of belonging. The two items addressing faculty interactions with students were:

  • My first year adviser made me feel like I could succeed at Augustana.
  • How often have your instructors pointed out something you did well on an assignment or in class?

Again, these findings held even after accounting for incoming ACT score. In other words, regardless of a student’s academic “ability,” faculty communicating to students that they can succeed and pointing to something that they have done well appears to contribute to a student’s sense of belonging on campus. This doesn’t mean that faculty should pull punches or tell students that they are doing well when they are not. Instead, this suggests to me the faculty play a critical role in contributing to student’s belief that they can succeed and then finding positive reinforcement to show them the way.

We found two items predicting a sense of belonging on campus that provide some concrete guidance for working with students outside of class.

  • Fall Connection provided the start I needed to succeed academically at Augustana.
  • I know that my Community Adviser cares about how I am doing at Augustana.

Although the item addressing Fall Connection (now called Welcome Week) seems fairly general, I think it further emphasizes the importance of the changes introduced this year to increase the emphasis on academic preparation. Based on last year’s data (prior to this elevated academic emphasis), this aspect of Fall Connection mattered significantly. In addition, I am particularly intrigued by the nature of the CAs impact on students’ sense of belonging. This kind of guidance provides pretty clear direction in designing the nature of CAs conversations with students.

All of these findings together simply confirm that we all play a significant role in shaping our students’ decision to persist at Augustana College. I hope we can find ways to further convert these findings into concrete action. As with so many other aspects of college students’ experience, it’s not what they do; it’s how they experience what they do.

Make it a good day,

Mark