What if early feedback made your students work harder? (Spoiler Alert)

One of the ways that we come up with questions for our freshman and senior surveys is by asking advisers about the concerns that students express to them. If we start to hear a particular theme, we try to find a way to capture that issue in a single question. Then, with the data from that item, we can test to see (1) if it really is a common problem or concern, and (2) whether that issue correlates with other experiences or the broader learning outcomes that we know are important for student success.

This is how we developed the freshman survey item to which students agreed or disagreed on a five point scale, “I had access to my grades and other feedback early enough in the term to adjust my study habits or seek help as necessary.” Numerous students had claimed that they didn’t know that they were struggling in one or more of their classes until very late in the term. They suggested that it wasn’t as if they had known their grades early in the term but hadn’t sought out academic support, but rather that they had either not been assigned enough (or more often, any) graded work to know their academic standing or that their homework wasn’t graded and returned or posted until well after the middle of the term. By the time they received their first substantive grade it was often well past the drop or withdrawal date and there was little time to recover. Moreover, these students often felt the late term pressure and in most cases had emotionally “given up” on the class.

So we inserted this item into last year’s freshman survey. Of the 286 responses, the answers were distributed like this:

  • Strongly Disagree – 30 (10%)
  • Disagree – 64 (22%)
  • Neutral – 77 (27%)
  • Agree – 89 (31%)
  • Strongly Agree – 22 (8%)

After seeing an even more “normal” distribution of responses (i.e., a bar graph of the responses that looks like a bell curve) from this year’s new mid-year freshman survey, I decided that it would be worth looking in more depth at this data to see if this item might be predictive of any other important important student-faculty interactions.

In fact, there were several student-faculty interaction items that were significantly influenced by early access to grades and feedback.  In each case, the more strongly students agreed with having access to grades and feedback, the more strongly they agreed with each of the following statements.

  • My one-on-one interactions with faculty have had a positive influence on my intellectual growth and interest in ideas.
  • My professors were interested in helping students grow in more than academic areas.
  • Faculty and staff at Augustana treated me like an individual.

Moreover, these statistically significant relationships held even after we controlled for incoming ACT score, parents’ education, and the Student Readiness Survey scales for academic habits, academic confidence, and persistence and grit.

By itself, these findings are important. But we found one other result that seems particularly useful. Even after controlling for the effects of the pre-college characteristics listed above, early access to grades and other feedback produced a statistically significant effect to a question that asked, “How often did you work harder than you have in the past in order to meet your instructor’s expectations?”

In other words, students who had access to grades and other feedback early enough in the term to adjust their study habits also indicated that they worked harder than they had in the past to meet their instructor’s expectations.

Why might this be? It might be because the early feedback gives students a chance to check themselves and reflect on their current level of effort. Furthermore, if the feedback comes in the form of an actual grade, it’s data that is difficult to ignore (we all have seen students demonstrate impressive levels of denial or willful blindness). This finding might also reflect the likelihood of an educational environment that emphasizes a continual exchange between the teacher and the student. This kind of environment is much more likely to result in a deeper level of student engagement in a course.

Yes, this finding isn’t absolute proof.  And yes, this is a statistically significant finding from only one cohort of freshmen.  And no, you’ll never get smoking gun evidence that one pedagogical practice will transform your most curmudgeonly student into a singing apostle for your discipline. However, this finding does comport with a compellingly large body of research on effective pedagogy for engaged student learning.

So you want your students to work harder in your class? Insert a substantive graded assignment early in the term. Turn that assignment around quickly with some extended formative feedback. Students will use that feedback to better grasp the degree to which they are working hard enough to meet your expectations. With that information they can adjust their efforts or seek out that academic resources might help them increase their likelihood of success.

Make it a good day,

Mark

Refining the way we deliver the liberal arts

First of all, thanks to everyone who reads Delicious Ambiguity for putting up with a three-part series of posts.  Somehow it seems to me like it requires an extra dose of ego to think that any idea is so big or important that it deserves three separate posts, so I feel sort of sheepish for even trying to pull it off.  More likely, this endeavor could just mean that I’m so damn long-winded that I can’t say anything of substance within my self-imposed word limit.

With that said, this is the “refining” post and is to be the last of the three-part bit I promised on reframing, refocusing, and refining the way that we operationalize the liberal arts. Partly because I’m late with this post and partly because it would be ironic to write a painfully long post about the act of refining, I’m going to make an extra effort to be concise.

I think that we should refine the way that we deliver the liberal arts because the quality of any educator’s effort (i.e., the degree to which an educator influences learning) can’t be separated from the amount of time that an educator has to dedicate to this effort. We know that helping students develop the skills and dispositions necessary to succeed in life requires substantial personal interaction. Over the last decade, it appears that we’ve become a very busy college.  We’ve added a host of legitimate educational and administrative responsibilities at every level, revamped programs and curricula, and seem to constantly on the brink of deciding to change something big. Yet we have taken very little away even as we’ve taken on all of these entirely defensible programs, pedagogies, and policies.  During the last twelve months of strategic planning discussions, the question that many raised, “But what are we going to take away?” seemed sometimes to be as much a lament as it was a serious question.

So I offer a set of findings from one part of last year’s senior survey and a few musing on some potential implications of these findings in the hope that it will at least fuel continued discussion of the things that we should take away.  I know there will always be some who are personally vested in the things that we might decide to take away, but (1) if we are ultimately about the experience of the students, and (2) we know that our ability to provide the best learning experience for students requires us to be efficient in the time we allot to each educational aspect of that experience, then we have to be brave enough as a community to make these choices.

Last spring, after the faculty had approved our college-wide learning outcomes earlier in the fall, I thought it might be useful to get some sort of baseline sense of where students think they develop these skills – not as evidence of where they actually gain these skills, but rather some reflection of the degree to which the experience we offer is as holistic as we’d like to think it is.  Thus, I inserted a set of questions in the senior survey that asked students to identify the experiences where they think they developed each of these learning outcomes.  For each learning outcome, students were given a list of college experiences and were allow to choose as many as applied.  Here are the results.

Disciplinary Knowledge
Major(s) or Minor(s) Courses 93.7%
General Education Courses (AGES) 45.9%
Augie Choice Programs (Study Abroad, Internship, or Undergraduate Research) 45.9%
Residence Life Experience 11.3%
Participation in Student Organizations and Clubs 28.5%
Working On or Off Campus 29.5%
Informal Interactions with Faculty and Staff 34.3%
Informal Interactions with Peers 25.4%
Volunteering in the Community 19.8%
Senior Inquiry 55.6%
None of the Above 0.4%
Critical Thinking and Information Literacy
Major(s) or Minor(s) Courses 91.3%
General Education Courses (AGES) 58.4%
Augie Choice Programs (Study Abroad, Internship, or Undergraduate Research) 36.4%
Residence Life Experience 10.7%
Participation in Student Organizations and Clubs 23.0%
Working On or Off Campus 25.9%
Informal Interactions with Faculty and Staff 29.7%
Informal Interactions with Peers 26.7%
Volunteering in the Community 11.9%
Senior Inquiry 57.8%
None of the Above 0.4%
Quantitative Literacy
Major(s) or Minor(s) Courses 77.6%
General Education Courses (AGES) 50.5%
Augie Choice Programs (Study Abroad, Internship, or Undergraduate Research) 15.2%
Residence Life Experience 3.2%
Participation in Student Organizations and Clubs 8.7%
Working On or Off Campus 16.0%
Informal Interactions with Faculty and Staff 10.9%
Informal Interactions with Peers 8.1%
Volunteering in the Community 3.4%
Senior Inquiry 39.0%
None of the Above 3.4%
Collaborative Leadership
Major(s) or Minor(s) Courses 62.0%
General Education Courses (AGES) 33.1%
Augie Choice Programs (Study Abroad, Internship, or Undergraduate Research) 37.4%
Residence Life Experience 27.3%
Participation in Student Organizations and Clubs 65.5%
Working On or Off Campus 46.7%
Informal Interactions with Faculty and Staff 29.9%
Informal Interactions with Peers 43.2%
Volunteering in the Community 40.0%
Senior Inquiry 24.6%
None of the Above 1.4%
Intercultural Competency
Major(s) or Minor(s) Courses 47.1%
General Education Courses (AGES) 48.5%
Augie Choice Programs (Study Abroad, Internship, or Undergraduate Research) 46.9%
Residence Life Experience 26.5%
Participation in Student Organizations and Clubs 38.6%
Working On or Off Campus 27.9%
Informal Interactions with Faculty and Staff 19.0%
Informal Interactions with Peers 40.0%
Volunteering in the Community 37.0%
Senior Inquiry 17.2%
None of the Above 5.1%
Communication Competency
Major(s) or Minor(s) Courses 84.4%
General Education Courses (AGES) 60.6%
Augie Choice Programs (Study Abroad, Internship, or Undergraduate Research) 41.4%
Residence Life Experience 20.2%
Participation in Student Organizations and Clubs 41.6%
Working On or Off Campus 35.2%
Informal Interactions with Faculty and Staff 40.2%
Informal Interactions with Peers 37.2%
Volunteering in the Community 22.4%
Senior Inquiry 51.3%
None of the Above 1.2%
Creative Thinking
Major(s) or Minor(s) Courses 79.6%
General Education Courses (AGES) 59.2%
Augie Choice Programs (Study Abroad, Internship, or Undergraduate Research) 36.6%
Residence Life Experience 13.3%
Participation in Student Organizations and Clubs 36.6%
Working On or Off Campus 26.1%
Informal Interactions with Faculty and Staff 23.0%
Informal Interactions with Peers 27.1%
Volunteering in the Community 17.4%
Senior Inquiry 49.3%
None of the Above 2.6%
Ethical Citizenship
Major(s) or Minor(s) Courses 59.4%
General Education Courses (AGES) 45.7%
Augie Choice Programs (Study Abroad, Internship, or Undergraduate Research) 40.8%
Residence Life Experience 25.5%
Participation in Student Organizations and Clubs 42.0%
Working On or Off Campus 36.6%
Informal Interactions with Faculty and Staff 31.9%
Informal Interactions with Peers 39.6%
Volunteering in the Community 42.6%
Senior Inquiry 26.1%
None of the Above 4.8%
Intellectual Curiosity
Major(s) or Minor(s) Courses 87.1%
General Education Courses (AGES) 59.2%
Augie Choice Programs (Study Abroad, Internship, or Undergraduate Research) 49.5%
Residence Life Experience 9.3%
Participation in Student Organizations and Clubs 27.7%
Working On or Off Campus 23.6%
Informal Interactions with Faculty and Staff 42.0%
Informal Interactions with Peers 34.9%
Volunteering in the Community 19.8%
Senior Inquiry 51.1%
None of the Above 2.2%

To some degree, the implications of these findings may lie in the eye of the beholder.  But what jumps out to me is the degree to which major and minor courses seem to do so much of the heavy lifting.  If this is really so, then it is no wonder that faculty would be pressed for time to do much else besides teach students.  However, if we would like to create a more holistic educational experience and do so in a way that allows all of us to be equally involved in the development of our students, then there might be some logic in considering ways for faculty to pass on some of the responsibilities of learning in specific areas.

But another way to read this data is to consider the degree to which students don’t indicate experiences outside of the academic realm more often.  And this might be a more useful way to think about designing a comprehensive college experience that is more effective and more efficient.  In an odd way, we might find that the refinement we need to consider is not that the responsibility for students’ learning across each of the learning outcomes is distributed more exclusively among individual experiences, but rather that each of us explicitly understands our role in contributing (1) a small but crucial building block to students’ development and (2) the sinew that connects that building block to another block that can only be gained through a different experience offered elsewhere on our campus (or an off-campus experience facilitated by someone on campus).

Someone once said that planning to do something is the easy part; the hard part was actually doing it.  I suspect for us this will also be true.  The strategic planning was the easy part.  The hard part will be implementing this plan in a way that accomplishes the learning goals and student success to which we aspire.

So let’s roll up our sleeves.  We’ve got work to do.

Make it a good day,

Mark

Refocusing on the Connections Instead of Just Making Better Parts

This is the second of three posts about our need to reframe, refocus, and refine the way that we operationalize (i.e., deliver) the liberal arts. Near the end of last week’s post (where I suggested reframing how we deliver the liberal arts around enabling our graduates to thrive in the midst of change) I suggested:

So the learning experiences that matter the most may in fact be the things that we consider the least. Right now we focus the most time, resources, and energy on the classes we offer, the activities we organize, the experiences we sponsor. Reframing the way that we deliver the liberal arts means placing increased focus on the way that students connect these experiences and apply the ideas from one experience to succeed in another. Moreover, it means guiding students to strategically set up the ideal set of inter-experience connections that best prepare them to achieve their post-graduate aspirations.

When Augustana was founded, the connections between classes (at the time considered the primary, if not only, learning experiences offered by the college) were a foregone conclusion because the curriculum was virtually identical for every student.  New content assumed the delivery of prior content and students moved in lockstep from beginning courses to advanced seminars.  Only near the end of their schooling were students allowed to deviate from the central educational path to take courses that fit their vocational intentions in law, medicine, the clergy, civil service, etc.  Furthermore, extra-curricular experiences weren’t seen as potential learning experiences since they didn’t have anything to do with the content delivered through the curriculum.

This earlier version of a liberal arts education and the one that we now endeavor to provide could not be more different.  While the curriculum of yesteryear was almost entirely predetermined both in terms of the courses one took and when one took them, today (although some majors are more prescribed than others) I doubt you could find two students who took the same courses in the same order during the same year – let alone throughout an entire undergraduate degree.  In addition, we now know that students develop, learn, and grow at least as much through their courses as they do through their out-of-class experiences, resulting in the wide support for everything from student organizations to study abroad.

In the context of all of this curricular and co-curricular opportunity, it’s no wonder that students’ effort to convey the impact of their college experience on a resume often devolves into a list where the length of that list is assumed to convey something about an individual’s preparation and potential for future success.  Of course, those considering recent college graduates for graduate school, employment, or long-term service have figured out that these lists are a ruse and can be a red flag for someone who is more surface than substance.

But if we refocus the way that we operationalize the liberal arts so that students highlight “why” they chose the experiences they chose and how they took charge of constructing the person they have become, the grocery list of college experiences (AKA resume) suddenly comes to life as a story of perpetual improvement. This doesn’t mean that they are perfectly constructed when they receive their diploma.  But it does mean that those students can probably tell their own story in a way that shows an emerging clarity of purpose and an accelerating sense of momentum toward it.  We all know from our own experiences that those students stand out even when they aren’t trying to make an impression.

So how might we operationalize the way that we deliver the liberal arts to highlight this new focus?

First of all, we don’t need to go back to the days of an overly prescribed college experience.  With the diversity of our students’ pre-college experiences, learning needs and interests, and post-graduate aspirations, treating our students as if they were all the same would be stunningly foolish.

Instead, we begin by mapping every activity and every course that students can take in terms of what learning is intended to emerge from that experience and how that learning contributes to the larger learning goals and mission of the college.  Since this mapping is intended to be an iterative experience, the exercise may well result in adapting, adding, or even subtracting some courses or experiences.  It might also result in altering some experiences to more specifically meet certain learning goals.  The primary result of this exercise is not just to produce a complete catalog of the learning experiences in which students can engage.  Instead, the goal is to produce customize-able flow charts that show the variety of ways that different types of students can identify a sequence of experiences that together cultivate the learning that each student need to fully prepare them to succeed after they graduate.

These maps become the primary tools for the college to help students construct a college experience that builds upon their pre-college experiences and abilities, fills in the areas in which they need additional opportunities to learn and grown, and gives them the best chance to be the kind of person they aspire to be when they graduate.  Ultimately, the totality of each student’s college experience can be conveyed through a cohesive narrative that tells the story of his or her college journey from start to finish.

The challenges to making such a refocus work are not without consequence.  Most important, we have to actually enact our commitment to student growth and development in everything that we do.  That likely means changing something that we currently do (even if it is something we really like to do a certain way) to make it more educationally effective for students.  We often ask the question during planning conversations, “But what are we going to take away?”  This mapping exercise often identifies things that we could and probably should take away.  The challenge is whether we are willing give those things up.

In broader terms, this means that we have be able to “zoom out” and see the forest instead of the trees.  There will be a myriad of ways that a student could put together the learning experiences necessary for post-graduate success. The most important goal here is that the students can lay out their path, retrace their steps and explain why they took each one of them, situating the reasons for their choices in the context of their post-graduate aspirations. Of course there will likely be students who, despite all of our best efforts, don’t follow the guidance that we provide for them. But if all of our students learn the value of thinking about their own lives as a strategic effort to grow and develop, the chances are pretty good that they will all be on their way to succeeding in life and embodying the results of a liberal arts education when they walk across the stage to accept their diploma.

Make it a good day,

Mark

Reframing How We “Deliver” the Liberal Arts

Last week I promised to spend the next several posts unpacking what I meant by “reframing, refocusing, and refining the way that we operationalize the liberal arts.” At the outset, let me emphasize that the focal point of these posts isn’t some new-fangled definition of the liberal arts. Instead, I want to concentrate on the way we make that learning happen; the way that we deliver the liberal arts.

To give this post some context, let’s begin with some data. (What did you expect!). Earlier this year, I asked my two student-workers to review 25 years of Augustana alumni data and identify those people who were working in a profession unrelated to any of their majors. Over four weeks we examined 10,680 files from the Augustana Advancement Office. As you can imagine, this is an inexact science, so we decided to be conservative in deciding whether someone worked in a “matched” profession (something that aligned with one of their majors) or not. For example, we assumed that a person who majored in art and was listed as a teacher was likely an art teacher and therefore we counted them as a match. Conversely, if an English major was listed as a park ranger? Bingo.

After categorizing and counting all of these alumni, we found that 4,137 of our 10,680 Augustana alums are working in professions completely unrelated to their majors – about 39%. In addition, while we found that alums from pre-professional majors like CSD, education, pre-med were working in a corresponding field somewhat more often than alums from traditional liberal arts majors like philosophy, history, english (for somewhat obvious reasons), we have a healthy dose of “mismatched” alums coming from virtually every major. For example, 41% of our former accounting majors are no longer accountants, 33% of physics majors aren’t physicists, and 20% of education majors aren’t teaching (the same is true for pre-med majors).

Yet these individuals are anything but failures.  On the contrary, many of them are quite successful, by any definition, and are clearly in positions of substantial responsibility. Others seem to be doing quite well in fields that simply did not exist when they graduated from Augustana (e.g., an early 1990s grad who now works in cell phone technology). Still others appear to have lived remarkably interesting and satisfying lives. I have to admit, as we looked through this list of alums I sometimes felt little pangs of envy, imagining the stories that some of these people probably have to tell.

So what does this data point mean? First of all, I think it suggests that the inclination to measure our institutional success by linking the major per se to a job and a salary is fundamentally misguided. Our graduates are supposed to leave Augustana with an array of opportunities that they did not have when they arrived. Moreover, those opportunities aren’t supposed to sit along a single path, nor are they supposed to avail themselves all at once. A metric that treats a major as a narrowing mechanism just doesn’t match our institutional mission.

Second, the fact that 40% of our alums are in professions other than their major corroborates what we know about present day professional life. The very concept of the career as 40-plus years in a single profession is almost extinct. Although the claim that a person will have seven careers a lifetime is up for some debate, it is clear that most people will change jobs on more than one occasion – especially during the first decade after college. So it seems to me that our educational goal for all graduates should ultimately be to prepare them to thrive in the midst of change. Sometimes that means adapting to new responsibilities and shifting conditions. Sometimes that means making the leap to a new and completely different opportunity. And sometimes it means resiliently responding to disappointment (or worse) in a way that turns lemons into lemonade.

And what does all of this have to do with reframing the way that we deliver the liberal arts?

A liberal arts environment is an ideal setting for developing students’ ability to thrive in the midst of change, but only if we use the breadth of disciplines and the epistemologies within them as starting points for learning rather than an end in and of itself. This breadth of disciplines should become the tools that we use to develop our students’ ability to see a hypothesis, a claim, a problem, or an opportunity from a variety of perspectives and deepen their understanding of the issues at hand. While the knowledge of content within the discipline is a necessary precursor to understanding, as our graduates’ lives move them further from their final term at Augustana, the skills they developed to thrive in the midst of uncertainty or change will inevitably become more important than the content knowledge they acquired.

So the learning experiences that matter the most may in fact be the things that we consider the least. Right now we focus the most time, resources, and energy on the classes we offer, the activities we organize, the experiences we sponsor. Reframing the way that we deliver the liberal arts means placing increased focus on the way that students connect these experiences and apply the ideas from one experience to succeed in another. Moreover, it means guiding students to strategically set up the ideal set of inter-experience connections that best prepare them to achieve their post-graduate aspirations.

Finally, reframing the way that we deliver the liberal arts won’t matter much unless students grasp the value of this reframing. This means that students need to know, from the very beginning, the strategy for why they should consider one choice over another – or even to engage that choice at all. While an increased focus on the spaces and connections between experiences might seem daunting, fully developing our ability to help students understand this new way of thinking about a college experience may be even more difficult.

But if we don’t roll up our sleeves and take on this challenge, it will continue to be more and more difficult to explain why a family or a student shouldn’t just accumulate the necessary number of course credits by any means necessary (AKA for the cheapest cost possible) in order to obtain an undergraduate degree.

Make it a good day,

Mark

 

Wait . . . why focus on life after college?

In the last few years several prominent liberal arts colleges have made post-graduate success a central measure of the institution’s educational quality (for example, see such plans or programs at St. Olaf College and The College of Wooster). Despite what some old school moon-howlers might have you believe, this move isn’t driven by a rejection of the liberal arts or an administrative coup d’etat. (If we’d given up on the liberal arts, we’d have all gone for higher paying gigs at some fly-by-night for-profit a long time ago, and – with all due love and respect for my administrative compatriots – we aren’t nearly smart enough to pull of a decent coup d’etat.)

Rather, what most of us have come to realize is that in order for the liberal arts to thrive into the next century, we have to reframe, refocus, and refine the way that we operationalize the liberal arts in the context of our current social, cultural, and economic conditions. Just like the approach that drove the successful emergence of small liberal arts college during the 18th and 19th centuries, we are again faced with the need to adapt our commitment to developing creative and innovative problem-solvers through interdisciplinarity and the synthesis of great ideas.

In the next three posts, I’d like to explain in more depth what I mean by reframing, refocusing, and refining the way that we operationalize the liberal arts in a 21st century context.  But for now, I’d like to highlight one data point that I think underscores a need to strengthen our focus on preparing all students equally for life after college.

In the 2013 administration of our recent graduate survey (alumni surveyed nine months after they’ve graduated from Augustana), we asked our alums about the degree to which they thought Augustana prepared them to succeed in their current endeavors.  Since we had already organized the survey to take those who had chosen to go to graduate school and those who had pursued immediate employment to separate subsets of questions, they answered two different versions of this questions, each worded to more specifically get at the relationship between their Augustana education and their current path.

The difference between the proportion of alums who felt they were prepared well depending upon whether they went to grad school or went to work seemed large enough to consider further.  Among those who went to grad school, 78% said that they were “fairly well” or “very well” prepared for their grad program.  By contrast, among those who went to full-time employment, only 65% said that they were “fairly well” or “very well” prepared for their first job. (The other response options were “somewhat,” “a little,” and “not at all.”)

Why might this be?  In digging further into the data, it appears that some initial suppositions don’t always hold true.  Pre-professional majors don’t always feel well prepared for their subsequent path (whether it be work or grad school) and more traditional liberal arts majors don’t always feel less well-prepared for their subsequent path.  We didn’t find a strong correlation between a student’s final GPA and their sense of preparation. Moreover, both groups of respondents (grad students and full-time employees) were scattered across a range of programs or professions, so it didn’t appear that any specific undergraduate programs were driving the results one way or the other.

Interestingly, it appears that the students who got a preview of what life would be like during the next phase of their own post-graduate path were the ones who felt best prepared by their Augustana experience.  Alums currently employed in a job who also participated in an internship while at Augie tended to feel better prepared than those who did not do an internship.  Similarly, the alums currently in grad school who also participated in a research project either with faculty or on their own tended to feel better prepared than those who did not have any research experience.

In general, this data point suggests that we might need to improve the degree to which we prepare students who go directly into the workforce after college.  In some ways, it doesn’t seem particularly surprising that an institution largely governed by faculty with Ph.D.s from elite research universities would be better at preparing students to succeed in grad school than in full-time employment.  This might be simply a case of something where we have to put a more concerted effort into preparing the students who aim for a full-time job right out of college just because our frame of reference would likely advantage graduate school preparation.

This finding also seems to provide some insight into the kinds of advice that we ought to give students based on their post-graduate goals.  Students intending to go into the workforce might be better suited for internships while students with plans for grad school might be ideal candidates for an extended research experience.  And while some of these experiences might be credit-bearing, in many cases they are outside the scope of the curriculum; another reason why it is important for advising to be conceived as a means of developing students holistically instead of merely selecting courses.

In the end, I don’t know that this finding demands a wealth of professional development for faculty – although if done right such assistance is not a bad thing.  Instead, resolving this gap may require little more than recognizing the extent of our own experiences, adjusting for our biases, and explicitly connecting students with the  right resources that uniquely fit with their post-graduate aspirations.

Welcome back to campus!  It’s lonely around here without you.

Make it a good day,

Mark

The Holiday Wish List for a Measurement Geek

Sincerely apologies to anyone who tried to find a new post on my blog yesterday. Apparently our server went “walk-about” over the weekend and our IT folks have been working day and night to salvage everything that was no longer operational.  I think that we are in the clear today, so I’ll try to put this post up a day late.

______________________________________________________________________

This is the week where I can’t help but overhear all the talk of the holiday gifts that people are getting for their spouses, partners, kids, friends, or in-laws.  And it struck me that there aren’t nearly enough suggestions for measurement folks who need to just own their geekdom and go big with it.  So here are a few ideas, discoveries, and possibilities.

  • Statistics ties.  Any formula, pie chart, or dumb stats pun on a tie.  Because nothing bludgeons humor to death better than a stupid stats pun.
  • The children’s book Magnus Maximus, a Marvelous Measurer.  It’s a pretty fun book with wonderful illustrations.  And it’s never too early to stereotype your profession.
  • The world’s largest slide rule.  Of course, it’s located in Texas.
  • The complete DVD set of the TV show NUMB3RS. This show managed to tease my people with the hope that someday complex math skills could really save a life. And yet, to this day I’ve never been in a public venue where someone suddenly yelled frantically, “Is there a statistician in the house!?”
  • A Digicus. They were made in the late 70s and early 80s by the electronic’s company Sharp. Apparently many Japanese were suspicious of the digital calculator when it was first introduced, so the Digicus was created to allow people to check their calculator results against an abacus. And you thought higher ed types were skeptical of change???
  • And last but not least, anything by the band Big Data. Yes, there is a band called Big Data. They describe themselves as a “paranoid electronic music project from the internet.”  Okey dokey.

Make it a good holiday break,

Mark

For the want of a response, the data was crap

Any time I hear someone use data from one of the new freshman, senior, or recent graduate surveys to advocate for a particular idea, I can’t help but smile a little.  It is deeply gratifying to see faculty and administrators comfortably use our data to evaluate new policy, programming, and strategic direction ideas.  Moreover, we can all point to a growing list of data-driven decisions that we know have directly improved student learning.

So it might seem odd, but that smile slips away almost as quickly as it appears. Because underneath this pervasive use of data lies a deep trust in the veracity of those numbers. And the quality of our data depends almost entirely upon the participation of 18-22 year-olds who are . . . . let’s just say “still developing.”  Data quality is like milk – it can turn on you overnight. If the students begin to think that survey questions don’t really apply to them or they start to suspect that the results aren’t valued by the college, they’ll breeze through the questions without giving them much thought or blow off the survey entirely. If that happens on a grand scale . . . . I shudder to think about it.  So you could say that I was “mildly concerned” as I organized fall IDEA course feedback forms for processing a few weeks ago and noticed several where the only bubbles colored in were “fives.”  A few minutes later I found several where the only darkened bubbles were “ones.”

Fortunately, a larger sampling of students’ IDEA forms put my mind at ease.  I found that on most forms the distribution of darkened circles varied and, as best as I could tell, student’s responses to the individual questions seemed to reflect at least a minimal effort to provide truthful responses.  However, this momentary heart attack got me wondering: to what degree might student’s approach to our course feedback process impact the quality of the data that we get?  This is how I ended up in front of Augustana’s student government (SGA) earlier this week talking about our course feedback process, the importance of good data, the reality of student’s perceptions and experiences with these forms, and ways that we might convince more students to take this process seriously.

During this conversation, I learned three things that I hope you’ll take to heart.  First, our students really come alive when they feel they are active participants in making Augustana the best place it can be.  However, they start to slip into passive bystanders when they don’t know the “why” about processes in which they are expected to be key contributors.  When they become bystanders, they are much less likely to invest their own emotional energy in providing accurate data.  Many of the students honestly didn’t think that the IDEA data they provided on the student form was used very often – if ever. If the data doesn’t really matter anyway, so their thinking goes, the effort that they put in to providing it doesn’t matter all that much either.

Second, students often felt that not all of the questions about how much progress they made on specific objectives applied in all classes equally.  As I explained to them how the IDEA data analysis worked and how the information that faculty received was designed to connect the objectives of the course with the students’ sense of what they learned, I could almost hear the light bulbs popping on over their heads.  They were accustomed to satisfaction-type surveys in which an ideal class would elicit a high score on every survey question.  When they realized that they were expected to give lower scores to questions that didn’t fit the course (and that this data would be useful as well), their concern about the applicability of the form and all of the accompanying frustrations disappeared.

Third, even though we – faculty, staff, and administrators – know exactly what we mean when we talk about learning outcomes, our students still don’t really know that their success in launching their life after college is not just a function of their major and all the stuff they’ve listed on their resume.  On numerous occasions, students expressed confusion about the learning objectives because they didn’t understand how they applied to the content of the course.  Although they may have seen the lists of skills that employers and graduate schools look for, it seems that our students think these are skills that are largely set in stone long before they get to college, and that college is mostly about learning content knowledge and building a network of friends and “connections.”  So when they see learning objectives on the IDEA forms, unless they they have been clued in to understand that these are skills that the course is designed to develop, they are likely to be confused by the very idea of learning objectives above and beyond content knowledge.

Although SGA and I plan to work together to help students better understand the value of the course feedback process and its impact on the quality of their own college experience, we – faculty, staff, and administrators – need to do a much better job of making sure that our students understand the IDEA course feedback process.  From the beginning of the course, students need to know that they will be learning more than content.  They need to know exactly what the learning goals are for the course. Students need to know that faculty want to know how much their students’ learned and what worked best in each class to fuel that learning, and that satisfaction doesn’t always equate to learning.  And students need to know how faculty have used course feedback data in the past to alter or adapt their classes.  If you demonstrate to your students how this data benefits the quality of their learning experience, I think they will be much more willing to genuinely invest in providing you with good data.

Successfully creating an evidence-based culture of perpetual improvement that results in a better college requires faculty, staff, and administrators to take great care with the sources of our most important data.  I hope you will take just a few minutes to help students understand the course feedback process.  Because in the end, not only will they benefit from it, but so will you.

Make it a good day,

Mark

 

 

 

 

Could a focus on learning outcomes unwittingly sacrifice process for product?

A central tenet of the learning outcomes movement is that higher education institutions must articulate a specific set of skills, traits, and/or dispositions that all of its students will learn before graduation. Then, through legitimate means of measurement, institutions must assess and publicize the degree to which its students make gains on each of these outcomes. Although many institutions have yet to implement this concept fully (especially regarding the thorough assessment of institutional outcomes), this idea is more than just a suggestion. Each of the regional accrediting bodies now requires institutions to identify specific learning outcomes and demonstrate evidence of outcomes assessment as a standard of practice.

This approach to educational design seems at the very least reasonable. All students, regardless of major, need a certain set of skills and aptitudes (things like critical thinking, collaborative leadership, intercultural competence) to succeed in life as they take on additional professional responsibilities, embark (by choice or by circumstance) on a new career, or address a daunting civic or personal challenge. In light of the educational mission our institutions espouse, committing ourselves to a set of learning outcomes for all students seems like what we should have been doing all along.

Yet too often the outcomes that institutions select to represent the full scope of their educational mission, and the way that those institutions choose to assess gains on those outcomes, unwittingly limits their ability to fulfill the mission they espouse. For when institutions narrow their educational vision to a discrete set of skills and dispositions that can be presented, performed, or produced at the end of an undergraduate assembly line, they often do so at the expense of their own broader vision that would cultivate in students a self-sustaining approach to learning. What we measure dictates the focus of our efforts to improve. As such, it’s easy to imagine a scenario in which the educational structure that currently produces majors and minors in content areas is simply replaced by one that produces majors and minors in some newly chosen learning outcomes. Instead of redesigning the college learning experience to alter the lifetime trajectory of an individual, we allow the whole to be nothing more than the sum of the parts – because all we have done is swap one collection of parts for another. Although there may be value in establishing and implementing a threshold of competence for a bachelor’s degree (for which a major serves a legitimate purpose), limiting ourselves to this framework fails to account for the deeply-held belief that a college experience should approach learning as a process – one that is cumulative, iterative, multi-dimensional, and, most importantly, self-sustaining long beyond graduation.

The disconnect between our conception of a college education as a process and our tendency to track learning as a finite set of productions (outcomes) is particularly apparent in the way that we assess our students’ development as life-long learners. Typically, we measure this construct with a pre-test and a post-test that tracks learning gains between the years of 18 and 22 – hardly a lifetime (the fact that a few institutions gather data from alumni five and ten years after graduation doesn’t invalidate the larger point). Under these conditions, trying to claim empirically that (1) an individual has developed and maintained a perpetual interest in learning throughout their life, and that (2) this life-long approach is direct attributable to one’s undergraduate education, probably borders on the delusional. The complexity of life even under the most mundane of circumstances makes such a hypothesis deeply suspect. Yet we all know of students that experienced college as a process through which they found a direction that excited them and a momentum that carried them down a purposeful path that extended far beyond commencement.

I am by no means suggesting that institutions should abandon assessing learning gains on a given set of outcomes. On the contrary, we should expect no less of ourselves than substantial growth in all of our students as a result of our efforts. Designed appropriately, a well-organized sequence of outcomes assessment snapshots can provide information vital to tracking student learning over time and potentially increasing institutional effectiveness. However, because the very act of learning occurs (as the seminal developmental psychologist Lev Vygotsky would describe it) in a state of perpetual social interaction, taking stock of the degree to which we foster a robust learning process is at least as important as taking snapshots of learning outcomes if we hope to gather information that helps us improve.

If you think that assessing learning outcomes effectively is difficult, then assessing the quality of the learning process ought to send chills down even the most skilled assessment coordinator’s spine. Defining and measuring the nature of process requires a very different conception of assessment – and for that matter a substantially more complex understanding of learning outcomes. Instead of merely measuring what is already in the rearview mirror (i.e., whatever has already been acquired), assessing the college experience as a process requires a look at the road ahead, emphasizing the connection between what has already occurred and what is yet to come. In other words, assessment of the learning that results from a given experience would include the degree to which a student is prepared or “primed” to make the most of a future learning experience (either one that is intentionally designed to follow immediately, or one that is likely to occur somewhere down the road). Ultimately, this approach would substantially improve our ability to determine the degree to which we are preparing students to approach life in a way that is thoughtful, pro-actively adaptable, and even nimble in the face of both unforeseen opportunity and sudden disappointment.

Of course, this idea runs counter to the way that we typically organize our students’ postsecondary educational experience. For if we are going to track the degree to which a given experience “primes” students for subsequent experiences – especially subsequent experiences that occur during college – then the educational experience can’t be so loosely constructed that the number of potential variations in the ordering of different students’ experiences virtually equals the number of students enrolled at our institution. This doesn’t mean that we return to the days in which every student took the same courses at the same time in the same order, but it does require an increased level of collective commitment to the intentional design of the student experience, a commitment to student-centered learning that will likely come at the expense of an individual instructor’s or administrator’s preference for which courses they teach or programs they lead and when they might be offered.

The other serious challenge is the act of operationalizing a concept of assessment that attempts to directly measure an individual’s preparation to make the most of a subsequent educational experience. But if we want to demonstrate the degree to which a college experience is more than just a collection of gains on disparate outcomes – whether these outcomes are somehow connected or entirely independent of each other – then we have to expand our approach to include process as well as product.  Only then can we actually demonstrate that the whole is greater than the sum of the parts, that in fact the educational process is the glue that fuses those disparate parts into a greater – and qualitatively distinct – whole.

Make it a good day,

Mark

Athletes, Enrollment, and Retention

It’s becoming more and more clear that the way we have thought about retention in the past is just too simplistic. Too often we use terms like “levers” or “buttons” in suggesting that if we could only identify the right thing to change, then retention would improve. However, when we don’t take the time to fully match our metaphor to the complexity of our circumstances, we run the real risk of putting in a lot of effort for very little improvement. For example, if we like the idea of one or more “levers” that we think we can move to systematically impact our retention rate, our metaphor can’t assume that the levers under our control are independent from each other. As we all know, the educational endeavor in which we are involved is much too complex. For our metaphor to be accurate (and therefore useful in identifying a course of action that has the best chance of producing positive results), we have to understand that each lever over which we have control is welded to other levers. In essence, moving one lever will automatically re-position others that also affect the long-term health of the college.

One example of this complexity became more apparent recently as we were examining our retention data among athletes.  Over the years we’ve found that typical first-to-second year retention rates among students who self-report as athletes are higher than our college average, and four, five, and six year graduation rates of athletes don’t differ between athletes and non-athletes. However, in digging a little deeper we found that about 45% of the students who left during the 2012 school year (a subset of the all the students who leave sometime between their first and second fall terms) started that academic year as athletes, a much higher proportion than the overall percentage of students who identify as athletes at the end of the year (about 30%). Unlike prior retention analysis where we used student self-reports of athletic status, for this analysis we looked at all of the students who were listed on all sports team’s initial rosters – including all the students who quit their sport before the end of the season and therefore didn’t report themselves as athletes on the end-of-the-year survey.

At first, one might think that this is a problem for athletics to solve (stereotypes of the hard-nosed dictator/coach chasing off less capable athletes might come to mind). However, further exploration exposes the degree to which our levers are welded together. You’ll forgive me if I borrow from my decade of experience in college athletics here to make my point.

It is no secret that our investment in athletics is, at least in part, based on the reality that athletic opportunity is a potent enrollment draw.  Our coaches play a significant role in encouraging perspective students to come to Augustana, both by initiating recruiting relationships and by offering opportunities to those students who inquire. This is clearly evident in the size of many of our sports’ initial rosters; especially among men’s sports. However, in the same way that the student-faculty ratio matters in creating a high touch, personalized college experience, the athlete-coach ratio matters too.  Large rosters can make it more difficult for a coach to connect with each player. And especially among younger athletes who may have less opportunity to compete due to the presence of older, more skilled players, this can exacerbate feelings of uncertainty and self-doubt that sometimes produce a decision to leave the team – and even the college. In the end, the way that we are using one lever (athletics) to meet enrollment goals may be increasing the likelihood of attrition among a certain subset of students.

Please understand that I am not advocating that we change anything.  Instead, given the the number of sports we offer and the way that we current organize our athletics programs, I am simply pointing out an example in which a lever we use quite effectively to meet one goal (enrollment) might well be creating an obstacle that limits our ability to meet another goal (retention).  I suppose one could argue that we should consider offering an additional sport or two so that athletics could still recruit the same overall number of students while reducing the average roster size of the individual sports. However, that depends on whether the increased costs of an additional sport (salaries, facilities, operating funds) would be offset by a potential increase in retention of students who came to Augustana with the intention of playing a sport. Obviously this is a pretty sticky question without a clear answer.

Again, my point here is only to highlight a trade-off – one that might be entirely legitimate – where we meet one set of goals in a way that potentially increases the difficulty of meeting another set of goals. Optimizing our retention rate is about finding our sweet spot. It’s not just about moving individual levers. That is what makes it so incredibly challenging – especially when we are trying to squeeze the last drops of optimizing out of something that we already do comparatively pretty well.

Make it a good day.  And enjoy the holiday.

Mark

The Fallacy of Matching Majors with Careers

It seems that most of the talk in recent months about the ROI (return on investment) of a college degree from a given institution has been focused on the degree to which new graduates from that institution can get well-paying jobs related to their major.  For liberal arts colleges and those of us who believe in the importance of a well-rounded education, the whole idea of assuming an inherent connection between major choice and career seems problematic.  Not only are there plenty of majors that don’t have a natural correlate on the job market (e.g., philosophy majors come to mind), but we are also regularly bombarded with the claims that individuals in today’s world will hold multiple jobs in multiple professions over the course of their working careers. Thus it seems odd to suggest that a college’s effectiveness could be pinned to the proportion of graduates who have landed jobs in their field within six months of graduation.

One data point from our survey of recent graduates seems to highlight this conundrum. Nine months after a class of seniors graduates, we ask them to complete a survey that asks a variety of questions about their current status, the degree to which their Augustana experience helped prepare them for their present circumstance, and the degree to which they believe that they are on the right long-term path.

One of the questions we asked our 2012 graduates last spring (about nine months after they had received their BA degrees from Augustana) was:

“Have your long-term professional goals changed since you graduated from Augie?”

The distribution of responses was revealing.

Not at all

48%

A little

21%

Somewhat

20%

Substantial

4%

Completely

3%

In other words, fewer than 50% of the 2012 graduating class considered themselves on the exact same long-term path that they were on when they walked across the stage to collect their diplomas.  In addition, over a quarter of the respondents said that their long-term goals had changed “somewhat,” “substantially,” or “completely.”

I believe the result of this single question holds critical implications for our efforts to best prepare our students to succeed after college.  First of all, this finding supports what we already know to be true – many of our students are going to change their long-term goals during their first several years after graduation. This is what happens to young people during their first foray into the world of working adulthood. We would be foolish to tie ourselves too tightly to a data point that doesn’t allow for these natural developments in the life a young adult.

Second, rather than mere job or graduate school placement, we would be smart to begin thinking about our students’ post-graduate success in terms of direction and momentum. Our students need to develop a clear sense of direction in order to decide what the best “next step” is for them. In addition, our job is to help them know when to take that “next step,” whether it be getting into the right graduate school or finding the right job or taking advantage of a once-in-a-lifetime opportunity that will better position them to move in the direction they have chosen for themselves. If we can do that, then no matter what happens to our students in the years after they graduate, they will be better able to succeed in the face of life’s inevitable challenges.

In concert with a sense of direction, our students need momentum.  This momentum should be self-perpetuating, cultivated by the right mix of motivations to handle setbacks and success. More importantly, it needs to be strong enough to thrive in the midst of a change in direction. This means that we develop their ability to be autonomous while holding themselves to high standards.  It means that they know how to be strategic in staying true to themselves and their goals no matter the distractions that might appear.

This doesn’t mean that we shouldn’t care about our students’ success in applying to graduate school or entry-level jobs in a given profession. On the contrary, we absolutely should care about statistics like these – especially if they support a student’s chosen direction and momentum.  But we should remember that a successful life isn’t etched in stone upon graduation from college.  And we should have the courage to track our students’ life trajectory in a way that doesn’t limit both us and them.

Make it a good day,

Mark