The Problem with Aiming for a Culture of Assessment

In recent years I’ve heard a lot of higher ed talking heads imploring colleges and university to adopt a “culture of assessment.” As far as I can tell (at least from a couple of quick Google searches), the phrase has been around for almost two decades and varies considerably in what it actually means. Some folks seem to think it describes a place where everyone uses evidence (some folks use the more slippery term “facts”) to make decisions, while others seem to think that a culture of assessment describes a place where everyone measures everything all the time.

There is a pretty entertaining children’s book called Magnus Maximus, A Marvelous Measurer that tells the story of guy who gets so caught up measuring everything that he ultimately misses the most important stuff in life. In the end he learns “that the best things in life are not meant to be measured, but treasured.” While there are some pretty compelling reasons to think twice about the book’s supposed life lesson (although I dare anyone to float even the most concise post-modern pushback to a five year old at bedtime and see how that goes), the book delightfully illustrates the absurdity of spending one’s whole life focused on measuring if the sole purpose of that endeavor is merely measuring.

In the world of assessment in higher education, I fear that we have made the very mistake that we often tell others they shouldn’t make by confusing the ultimate goal of improvement with the act of measuring. The goal – or “intended outcome” if you want to use the eternally awkward assessment parlance – is that we actually get better at educating every one of our students so that they are more likely to thrive in whatever they choose to do after college. Even in the language of those who argue that assessment is primarily needed to validate that higher education institutions are worth the money (be it public or private money), there is always a final suggestion that institutions will use whatever data they gather to get better somehow. Of course, the “getting better” part seems to always be mysteriously left to someone else. Measuring, in any of its forms is almost useless if that is where most or all of the time and money is invested. If you don’t believe me, just head on down to your local Institutional Research Office and ask to see all of the dusty three-ring binders of survey reports and data books from the last two decades. If they aren’t stacked on a high shelf, they’re probably in a remote storage room somewhere.

Measuring is only one ingredient of the recipe that gets us to improvement. In fact, given the myriad of moving parts that educators routinely deal with (only some of which educators and institutions can actually control), I’m not sure that robust measuring is even the most important ingredient. An institution has no more achieved improvement just because they measure things than a chef bakes a cake by throwing a bag of flour in an oven (yes I know there are such things as flourless tortes … that is kind of my point). Without cultivating and sustaining an organizational culture that genuinely values and prioritizes improvement, measurement is just another thing that we do.

Genuinely valuing improvement means explicitly dedicating the time and space to think through any evidence of mission fulfillment (be it gains on learning outcomes, participation in experiences that should lead to learning outcomes, or the degree to which students’ experiences are thoughtfully integrated toward a realistic whole), rewarding the effort to improve regardless of success or failure, and perpetuating an environment in which everyone cares enough to continually seek out things that might be done just a little bit better.

Peter Drucker is purported to have said that “culture eats strategy for lunch.” Other strategic planning gurus talk about the differences between strategy and tactics. If we want our institutions to actually improve and continually demonstrate that, no matter how much the world changes, we can prepare our students to take adult life by the horns and thrive no matter what they choose to do, then we can’t let ourselves mistakenly think that maniacal measurement magically perpetuates a culture of anything. If anything, we are likely to just make a lot more work for quantitative geeks (like me) while excluding those who aren’t convinced that statistical analysis is the best way to get at “truth.” And we definitely will continue to tie ourselves into all sorts of knots if we pursue a culture of assessment instead of a culture of improvement.

Make it a good day,

Mark

 

A little thing happened while you were away . . .

Welcome back to campus! I hope you enjoyed a restful winter break. Although I was able to find a few days of legitimate relaxation (I actually read fiction for fun!), a little thing happened at the end of last week that yanked me back into focus and kept my mind spinning over the weekend.

Friday morning’s big reveal from the higher ed press was the announcement from President Obama that he is proposing a program to make community college free.  The details and the obligatory range of reactions was dutifully reported here and here, and by this morning it seems that almost every news outlet with an education beat has polled the usual suspects for comment, analysis, and knee-jerk reaction. The chatter about this policy proposal doesn’t need any more faux smart people to weigh in, so I’ll refrain from adding an unfocused “thumbs-up” or “thumbs-down” to the mix.  However, I think that the mere emergence of this policy proposal holds a couple of important implications that could matter a lot for those of us at Augustana College (as well as other small liberal arts colleges).

First, a big part of this proposal turns on the caveat that “Community colleges will be expected to offer … academic programs that fully transfer credits to local public four-year colleges and universities.”  This sounds great, except for the fact that the destination institution is the one that determines whether academic programs or credits transfer fully, not the individual community college from whence the student originates.  Whether or not the President’s policy proposal comes to fruition, I think it reflects a increasingly common belief that students should be able to move seamlessly between higher education institutions, no matter if they are moving between two-year or four-year institutions (not to mention the individual online courses, degree programs, or prior learning credits).

If I’m right here, then we will continue to see more and more students transfer credits to and from Augustana as they become less associated with a particular institution and more connected to the degree they are intending to earn or career they intend to pursue.  Again, if I’m right, that will make it even more difficult for us to know, a) if our graduates have learned everything that we believe an Augustana degree represents, and b) if the students sitting in front of us on the first day of the term already possess the prerequisite knowledge and skills to succeed in each class. However we respond to this issue (for example, offering remediation services for students who struggle, signing articulation agreements with individual community colleges to assure some degree of vetting prior coursework for transfer students, or designing competency-based assessments for students to demonstrate their readiness for advanced academic work and graduation), the challenges that emerge when students increasingly enter and depart colleges and universities at times other than the beginning and the end of that institution’s designed educational experience are, as a 2012 study suggests, likely to become more prevalent.

Second, if this proposal does in fact signal that earning credits from multiple institutions to complete a degree is gaining in both numbers and legitimacy, then we would be smart to take a hard look at all of the ways in which our institutional practices might subtly dissuade transfer students from considering Augustana.  Since our study of transfer students’ experience a couple of years ago, we’ve already made some changes to make Augustana a better destination for transfer students. But we still have some work to do – not because we have dropped the ball in responding to our findings, but because this kind of work is just plain hard.

Third, it seems to me that this trend further emphasizes the degree to which we need to be able to show that the totality of the Augustana experience – not just the academic coursework – produces the critical learning that we intend for our students.  Otherwise, we are likely to fall victim to the external framing of what constitutes a college education (aka an accumulation of academic credits that are equally valuable as a whole or a sum of their parts), making it even more difficult to differentiate ourselves in product or perception.

I’m sure that you can think of specific issues that we ought to examine if transfer students are going to become an increasingly large segment of the college-going public.  As the number of high school graduates in the Midwest continues to decrease over the next decade or so, it seems that this question becomes that much more important.  If you have some thoughts, please feel free to post them in the comment section below.  Maybe we can have a conversation without having to brave the frigid temperatures outside?

Make it a good day,

Mark

What is your definition of a “Plan B?”

I often get pegged as “the numbers guy.” Even though the words themselves seem pretty simple, I’m never really sure how to interpret that phrase. Sometimes people seem to use it to defer to my area of expertise (and that feels nice). But sometimes it seems vaguely dismissive, as if they’re a little surprised to find that I’ve escaped from my underground statistical production bunker (that doesn’t feel so nice).

With data points, it’s not the numbers by themselves that make the difference; it’s the meaning that gets assigned to them. The same is true with phrases that we all too often toss around without a second thought. I stumbled into a prime example of this issue recently while talking to several folks about the way that they think about helping students prepare for life after college. It turns out that we can run ourselves into a real buzzsaw of a problem if we don’t mean the same thing when we talk to students about developing a “Plan B.”

Essentially, a Plan B is simple – it’s a second plan if the first plan doesn’t work out. But underneath that sort of obvious definition lies the rub. For what purpose does the Plan B exist? Is it to get to a new and different goal, or is it to take an alternative path to get to the original goal?

For some, helping a student construct a Plan B means identifying a second career possibility in case the student’s first choice post-graduation plan doesn’t work out. For example, a student who intends to be a doctor may not have the grades or references to guarantee acceptance into med school. At this point, a faculty adviser might suggest that the student investigate other careers that might match some of the student’s other interests (maybe in another health field, maybe not). This definition of a Plan B assumes a career change and then begins to formulate a plan to move toward that new goal.

But for others, helping a student construct a Plan B doesn’t mean changing career goals at all. Instead, this definition of a Plan B recognizes that there are often multiple pathways to get into a particular career. For the aspiring med school student who may not have slam-dunk grades in biology or chemistry but still wants to be a doctor, one could envision a Plan B that includes taking a job at a hospital in some sort of support role, retaking specific science courses at a local university or college, then applying to medical school with stronger credentials, potentially better references, and more experience. In this case the end goal didn’t change at all. The thing that changed was the path to get there.

In no way am I suggesting that one definition of a Plan B is better than another. On the contrary, both are entirely appropriate. In fact, the student would probably be best served by laying out both possibilities and walking through the relevant implications. But the potential for a real disaster comes when two people (maybe a faculty member and a career adviser) are separately talking to the same student about the need to devise a Plan B, yet the faculty member and the adviser mean very different things when they use the same phrase.

As you can imagine, the student would probably feel as though he or she is getting conflicting advice. In addition, she might well think that the person encouraging a different career choice just doesn’t believe in her (and that the person suggesting an alternate path to her original career goal is the one who really cares about her). Moreover, the person encouraging the student to explore another career choice might feel seriously undermined by the person who has suggested to the student an alternative way to continue toward the original career goal. In the end, a student’s trust in our ability to guide them accurately and effectively is seriously eroded and a rift has likely developed between the two individuals who both genuinely care about the student in question.

Absolutely, there are times when we have to tell students that they need to explore alternative career plans. We do them no favors by placating them. At the same time, we all know students who, although they seemed to lack motivation and direction when they were at Augustana, kicked it in after graduation and eventually found a way into the career they had always wanted to pursue.

I’m certainly not suggesting that we should adopt one official definition of the phrase “Plan B.” Rather, my suspicion is that this is one of those phrases that we use often without realizing that we might not all mean the same thing. If our goal is to collectively give students the kind of guidance that they need to succeed after graduation, we probably ought to make sure that in each case we all mean the same thing when we talk to a student about a Plan B.

Make it a good day,

Mark

 

“Lean” in and learn something new

I think it’s fair to say that most educators cringe at the idea of applying practices from the world of business to education. So many times we’ve read or heard someone talk about education as if it were a cursory transaction where students or parents simply purchase a product as an investment toward future earnings. Of course, one only needs to spend a few days trying to get students to learn something that contests their prior assumptions to know that viewing education through such a transactional lens leads to a gross misunderstanding of what we do and how education works. I’d love to see a list of all the times when a business framework was misapplied to an educational setting with disastrous results.

So, does this mean that everything developed in a business setting is guaranteed to fail in an educational setting? It’s okay if you’re inclined to say “yes” (especially if you’ve been down that road a few times). When President Bahls suggested that we could apply principles of lean management to improve a variety of processes at Augustana College, I’ll admit that I shuddered. Maybe like you, I imagined an internal apocalypse: budget cuts and position reductions with no changes in expectations. But after reading up on the concept of lean management and spending last week as a member of the first Rapid Improvement Team, I have to admit that my shudder was merely emblematic of my own ignorance. While lean management has its own set of terminology that might seem foreign to educators, the values embedded in a lean management philosophy embody the same values that we aspire to uphold in a collaborative and transparent organization dedicated to educating students. I found the framework and the process to be deeply gratifying and potentially applicable to the range of domains in which we operate.

First, “lean” doesn’t mean thinner.  It’s not about losing weight, downsizing, or cutting out the fat. It’s not an acronym. The term refers to the degree to which processes are conducted efficiently while best serving the needs of the beneficiary (i.e., anyone who benefits from that process).

Second, lean management philosophy asserts that the people best positioned to make improvement happen are those who are intimately involved in that particular job or process. Not only do those folks know the ins and outs of that work better than anyone else; they also need to believe in the efficacy of any identified changes in order to give those changes the best chance of turning into demonstrable and lasting improvements. For these reasons, any attempt to improve a process must genuinely involve the people who do that work.

Third, lean management philosophy argues that improvement of a process is exemplified in those who benefit from that process. Although the beneficiaries of our work are often students, we often conduct operations that benefit more than just students. The beneficiaries of payroll are anyone who gets paid. The beneficiaries of the salad bar are anyone who eats a salad. As a result, the way to determine if we have improved a process is to identify clear means of demonstrating an improved impact on the beneficiaries of that process.

Fourth, lean management starts with the belief that the collective ability of an organization’s people can find and put in place substantial improvements to a process.  Effective lean management begins by collaborating to develop a shared understanding of the current state of a process or problem.  Only after the problem is fully understood as something worthy of improvement would an improvement team begin to consider potential solutions.

Fifth, lean management philosophy focuses on continual improvement, not perfection. There are simply too many external and unpredictable influences to expect perfection.  Furthermore (especially in the work that we do), just when we find that a particularly education practices works well, the students change and we have to continue to adjust.

Everything the Rapid Improvement Team did last week reflected all of these values.  I was impressed with the way the process was designed to keep them at the forefront while moving us toward a set of suggestions that were extremely likely to improve the process.

If you would like to see a recording of the presentation from the Rapid Improvement Team from last Friday, you can see it here.

Ultimately, the lesson I learned from this process was that it is possible (shocking, I know) for something that has been developed in the business world over the last several decades to be applied successfully in an educational institution in a way that actually strengthens our ability to enact the values we espouse.  In addition, I (re)learned that we have some amazing people at Augustana who are willing to put their hearts into doing what we do better. I’m lucky to be a part of it.

Make it a good day,

Mark

 

A taste of my own medicine

One of the fundamental tenets of Delicious Ambiguity has been that all of us who contribute to the development of students are at our best when we approach our work through a lens of “positive restlessness.” That phrase was introduced into the lexicon of higher education writers by George Kuh and his colleagues in their book Student Success in College, describing a pervasive philosophy that his research team saw at colleges that always seemed to be seeking out ways to get better no matter how successful they already were. Anyone who knows me recognizes that I relish the chance to look for ways to improve. But I think it is an entirely fair criticism to suggest that I might have an overly rosy view of change and that I should be forced to get elbow-deep in the down-and-dirty work of actually fixing a complicated and convoluted process.

So this week, if you’ve ever thought that I needed a dose of your version of reality, you are in luck. My “comeuppance” has appeared in the form of participation in a weeklong, immersive Rapid Improvement Event (RIE). I’ll be joining a team of Augustana employees trying to wrangle a portion of the payroll process and hopefully improve it. I don’t know much about payroll – so they tell me I’m “perfect” for the job.

So here goes!

Make it a good day,

Mark

 

Hey; what’s this I hear about the Winding Path Study?

Some of you have heard me mention a study that we (AKA our massive juggernaut of an IR office better known to most of you as Kimberly and Mark) started last spring called the Winding Path Study. In short, this study was designed to gather information from all living Augustana alumni (at least those for whom we had working email addresses) about the nature of their adult lives from the time they entered Augustana up until last spring.

During the twelve months of strategic planning discussions, one of the things that stood out to me was how much we really don’t know about the long-term impact of an Augustana education. Don’t get me wrong; we have lots of wonderful stories about Augustana graduates excelling in all sorts of professional and personal pursuits. But we don’t know nearly as much as we would like about the nature of our alum’s lives after college: the ways that they have handled success and failure, the adjustments they have had to make when life throws them a curveball, or the ways that their Augustana experiences might have influenced twists and turns in their life’s path right after graduation or much later in life. This information matters because, if we are preparing students to succeed throughout their adult lives, we need to know how those lives play out across personal and professional domains and as our alums grow and change over time.

After looking through all of the different ways that colleges have tried to survey their alumni, we couldn’t find any approach that matched our conceptual frame or addressed the questions we had constructed. So we rolled up our metaphorical sleeves and built a study from scratch based on the sociological theory of Life Course Perspective, a construct that describes the life course as a series of trajectories, transitions, and turning points.

In this post I’d like to share a few summary findings just to give you a flavor for what we’ve seen from the almost 2,800 responses we received last spring in the first stage of this project.

The first two questions explored the nature of our alum’s path when entering Augustana and moving through their undergraduate years.

  • Did you have a specific career goal or major in mind when you came to Augustana?
    • 53% – Yes; I was pretty sure I knew what I wanted to do
    • 33% – Somewhat; I had some ideas but wasn’t set on anything in particular
    • 12% – No; I didn’t know what I wanted to do at all
  • Did you change majors or career goals while you were an undergraduate?
    • 38% – Yes
    • 61% – No
  •  What path did you take right after graduation from Augustana?
    • 26% – Went to grad school in the same field that I studied
    • 8% –   Went to grad school in a different field than I studied
    • 42% – Took a job or volunteered in the same field that I studied
    • 15% – Took a job or volunteered in a different field than I studied
    • 2% –   Took time off to pursue other interests
    • 7% –   Other

The next set of questions explored the varied nature of our graduates’ adult lives. Although we couldn’t have possibly captured every facet of an adult life, our goal was to gather a first glimpse that could be explored in more detail later.

  • How many times have you changed jobs since you graduate from Augustana?
    • 16% – None
    • 27% – 1-2
    • 28% – 3-4
    • 16% – 5-6
    • 11% – 7 or more
  • How many of those job changes occurred because of a professional opportunity that you chose to pursue?
    • 22% – None
    • 36% – 1-2
    • 22% – 3-4
    • 11% – 5-6
    • 5% –   7 or more
  • How many of those job changes occurred because of a professional disruption (downsizing, bankruptcy, termination, etc.)?
    • 67% – None
    • 23% – 1-2
    • 3% –   3-4
    • 1% –   5-6
    • 0% –   7 or more
  • Were any of your job changes influenced by family considerations?
    • 39% – Yes
    • 56% – No
  • Were any of your job changes influenced by personal considerations?
    • 54% – Yes
    • 41% – No

(Please note that some folks didn’t respond to every question, resulting in some proportions equaling less than 100%).

These findings deepened our understanding of the variety of pathways that students pursue after college.  Almost one quarter of our graduates, immediately after college, entered graduate school or took a job in a different field than their major.  These findings also strengthened our belief that preparing students for successful lives after college goes far beyond one’s major or minor and extends long past the first job, first graduate school degree, or whatever the first thing a student chooses to do after college might be.

As you can also see, many if not most Augustana alumni have likely led adult lives that look more like winding paths than straight lines. These findings – even if they might seem fairly obvious to anyone who has lived through the reality of an unpredictable life – have shaped our thinking as we continue to design a college experience that prepares every student to carve through life after college – no matter what comes out of the woodwork.

The first stage of this study concluded with a question at the end of the survey asking if the respondent would be willing to participate in a half-hour interview.  Based on prior research experience, we expected 100-200 positive responses.  We received over 1,400 positive responses!  So the next step for us, after spending the last six months analyzing all of the open-ended responses, is to develop a framework for the interviews and how we will select potential interviewees.  We would like to interview as many as possible, but frankly, the specter of 1,400 interviews is a bit daunting! Moreover, because we also asked other categorizing information like the year the respondent graduated, their major, and their current profession, we have all kinds of ways that we can organize and analyze this data.

Over the rest of the academic year, I hope I’ll have another update on the results of the second phase of this study. In the mean time, enjoy the last week of the fall term!

Make it a good day,

Mark

Data doesn’t have to be numbers

As some of you might know, every once in a while I get asked to talk to other colleges or universities about my study abroad research. Yesterday I was fielding some questions at one such workshop, when a faculty member who takes students on study abroad trips told a story of her experiences talking to her students about what they had learned during their trip abroad. She talked at length about the students’ description of their own growth but ended her statement by saying, “Of course, I don’t have any data on this.”

I hear that line so often when talking with faculty or student affairs staff about their experiences with students.  And although I’m sure I’ll say this again at some point (and hopefully in a kind and caring way!), I just wanted to quickly say to anyone reading this blog today . . . Data is information.  Sometimes information comes in the form of numbers. Sometimes information comes in the form of comments.  Sometimes information comes in the form of student assignments.  Data does not have to be numbers. In fact, sometimes the worst data out there comes in the form of numbers. So if you have information about students that comes from data you have gathered, then you have data. It might be indicative of something that a broader swath of students experience, or it might just be illustrative of the small group of students that provided that data.  But it’s ALL data.

The subtitle of this blog is “using evidence to improve student learning.”  Evidence comes in all forms, and information that qualifies as evidence must go through a vetting process that doesn’t have to be numerical. So if you have data that you think qualifies as evidence – bring it!  Please don’t hesitate or worse, short-change yourself and your work, by thinking that it only qualifies if it comes in numbers.  That’s just not so.

Make it a good day,

Mark

Swimming in the 2014 Senior Survey data!

I think I’ve resigned myself to the fact that it is nearly impossible to explain to someone who hasn’t experienced Augustana’s 10-week terms what it feels like to go from standing still less than a month ago to flying by the seat of your pants at week four. But here it is verging on mid-term time, and I’m hurtling through space trying everything I can to get my bearings!

So, to show that the Institutional Research and Assessment staff (AKA Kimberly and I) doesn’t just sit around dreaming up ways to collect more data, I thought I’d share with you . . . more data. (I guess this doesn’t really debunk my assumptions about your assumptions, does it.)

Every spring, we ask our graduating seniors to complete a survey that asks all sort of questions about their experiences at Augustana. In addition, we ask a few important questions that we’ve found to be useful outcome questions (would you choose Augustana again, is your post-graduate plan a good fit for who you are and where you want your life to go, and do you already have a job or grad school place).

It takes a lot of work to process this data into a readable report, but it’s finally finished and posted on the IR web page.  Here is the direct link to the 2014 Senior Survey results.

2014 Senior Survey Report and Findings

Now you could jump on that link right away and start swimming in the data – percentages, averages, and standard deviations (oh, my!). And you might survive the experience, although your eyes will probably start to glaze over as you look at mean score after mean score and your brain will likely start to go soft wondering (rightly so) what exactly each average score means – is it good? is is bad? is it just right? (sort of like Goldilocks in the house of the three bears . . . replacing the bowls of porridge with excel spreadsheets, of course).

So if I may, let me make a suggestion that I hope will make some of this data more meaningful.  Instead of looking at the numbers first, put a sheet of paper over the numbers and look at the questions first.  Reflect on why each question might matter for students and what might be the “about right” distribution of responses.  Pick out 3-5 questions that seem particularly interesting to you.

THEN take away the sheet of paper covering the numbers. Do your musings match up with the average score or the distribution of responses?  What more would you like to know that might help you get a better handle on what we could do to improve, if the difference between your reflections and the actual score suggests that institutional improvement might be valuable?

This data isn’t of much use if it doesn’t help us get better at what we do. And you – the people on the ground floor who are working with students every day – are the ones who are ideally suited to tackle this data, jump into this process, and benefit from the results of your efforts.

If you have any questions, comments, suggestions, or criticisms (I prefer to think of it as constructive feedback!) about the senior survey, PLEASE PLEASE PLEASE contact us – Kimberly or me – in the IR office. Nothing we have built is so important that it can’t be changed . . . especially if those changes make the survey better.

Make is a good day,

Mark

What happens when you build assessment around improvement?

Well hello, everyone!

It’s great to feel the energy on campus again. And it’s exciting to restart my Delicious Ambiguity blog: Season 4 (I’ve been renewed!). Each week I share some tidbit (data that comes from statistics or focus groups) from our own Augustana student data that will help you do what you do just a little bit better the next time you do it. If you’re new to Delicious Ambiguity, you might also want to know that you can search the three years of previous posts (about 100 in all) for everything from athletes to introverts, Greeks to geeks. In addition to a ton of useful findings, you might even find a few funny quips (AKA bewildering side comments).

By now you’ve probably heard me say on at least one occasion that building assessment efforts around genuine improvement, as opposed to doing assessment to find out what’s already happened (i.e., to prove what you think you are already doing), thoroughly changes every part of the assessment process.  More importantly, it’s the only way to actually get better because:

  1. You’ve backward-designed the entire project around finding out what you need to do to get better instead of just finding out what happened, and
  2. You’ve humbled yourself to the possibility of improvement and thereby matched your efforts with the way that educational processes actually work.

I’d like to share an example of one program at Augustana that has clearly benefited from an “assessment for improvement” approach.  My goal here isn’t to brag, but rather to walk you through an example of such a process in the hope that something I share might be applicable to your own unique context.

Augustana has run some version of freshman orientation for a very long time. And by and large, it’s been a pretty successful program. Yet everyone involved has always wondered what they might do to make it just a little bit better. Much of our prior data only told us the proportion of students who were “satisfied” with the experience. Although we could pat ourselves on the back when the numbers were decent (which they were virtually every year), we had no way of turning that information into specific changes that we could trust would actually make the experience demonstrably more effective.

So a few years ago, folks from Student Affairs, Academic Affairs, and the IR office began applying an improvement-centered assessment approach to orientation. First, we talked at length about drilling down to the core learning goals of freshman orientation. Sure, we have lots of things we’d love to result from orientation, but realistically there are only so many things you can do in four days for a group of 18-year-olds who are some combination of giddy, overwhelmed, and panicked – none of which makes for particularly fertile learning conditions.

So with that in mind, we needed to strip our goals down to a “triage” set of learning goals for orientation.  We settled on three concepts.

  • Welcome Week will connect new students with the people necessary for success.
  • Welcome Week will connect new students with the places necessary for success.
  • Welcome Week will connect new students with the values necessary for success.

The people we identified included all of the individuals who might influence a student’s first year experience – other students, student affairs and residential life staff, and specific faculty members. The concept of place involved a) knowing EXACTLY how to walk to one’s classes and specific first-year resources, and b) finding other places on campus that a student might use for emotional rejuvenation as well as intellectual work. The values we discussed focused on clarifying a strategy for getting the most out of a liberal arts college setting. This meant introducing students to a mindset that focuses on actively participating in a process of learning and growth and show them how this approach will increase their likelihood of success, both in the first year and beyond.

Once we spelled out our goals for Welcome Week, then we could set about our work from two directions. First, we could start to alter the design of the experience to meet those goals. Second, we could build a survey that examined the degree to which students came away from Welcome Week connected to the people, places, and values that substantially increase the likelihood of success in the first year.

Over the last two years the survey findings have provided a number of interesting insights into the degree to which certain experiences were already meeting the goals we had set. More importantly, the survey data has become a critical conversation guide for specific improvements. Because the questions were built around specific experiences, it has given everyone – particularly peer mentors – a clear target to shoot for with each student. For example, if the goal was to ensure that each student would say that they knew exactly how to get to their classes on the first day, then the peer mentor could shift from merely pointing at buildings while walking around campus to creating some way for new freshmen to walk right up to the door of the room where their class would be.

At the same time that we were using data to guide specific adjustments, the folks planning Welcome Week examined the design of the entire program. This led them to introduce several changes, including the Saturday morning concurrent sessions titled “Augie 101,” focusing on all kinds of issues that would specifically increase the likelihood of successful academic acclimation.

We will survey the freshmen in the next week or so to find out how the most recent set of changes impacted their experience during Welcome Week. But even without that data, I suspect that the new programming this year improved the experience. My confidence comes from one particularly compelling data point that isn’t a number (I know – sit down and take a deep breath!). During the Augie 101 sessions, peer mentors and other older students who were assisting with Welcome Week kept saying, “I wish we would have had something like this during my Welcome Week experience.”  To me, that is a powerful endorsement of our efforts.

We will likely never be perfect, but we have mounting evidence that we keep getting better at what we do. That doesn’t mean that we have any reason to brag or rest on our laurels.  It just means that we are doing things right.  And that’s what makes doing this work so much fun.

Make it a good day,

Mark

A new (and maybe better) way to understand the impact of an Augustana education

As you probably know by now, the new Augustana 2020 strategic plan places our graduates’ success after college at the center of our institutional mission.  In real terms, this means that what our students learn in college matters to the degree that it contributes to their success after college.  Put another way, even if our students learn all kinds of interesting knowledge and complicated skills, if what they have learned can’t be effectively and meaningfully applied to life after college, then we haven’t really done our job.

Now whether you think that this is the last nail in the liberal arts coffin or the long-awaited defibrillator to revive liberal arts education, our own success hinges on something else that I’m pretty sure we haven’t thought much about. Exactly what are we talking about when we talk about a successful life after college? Do we have a working definition of what might make up a successful life for an Augustana graduate? In order to grapple with those vertigo-inducing questions, we have to know a lot more about what happens to our graduates after college.  But do we have anything more than vague notions about our own graduates’ lives?

I’m afraid that the answers to those questions are probably no, no, and no. In part, it’s because these are big, hard questions.  And to be fair, I don’t know of a college that has tried to get a real handle on these ideas.  So . . . . here we go . . . .

This is the kind of research project that can keep you up at night.  Because it isn’t just about getting data to figure out the relationship between one thing (an Augustana college experience) and another thing (a successful life after college).  For starters, these are two monstrously complicated constructs.  Distilling them down to some essential qualities may well be impossible.  I’m not saying that it’s NOT possible; I’m just admitting to the fact that I’m intimidated by the very idea of trying to identify a set of valid essential qualities. And as if that weren’t enough, we (higher education researchers writ large) have yet to have developed a conceptual framework that is complex enough to account for the almost infinite range of ways in which people’s lives evolve. To date, every effort to link alumni success to their college experience has presumed a straight line – even when we know that very few of us traveled a straight path to get to where we are now.

So over the past six months or so, Kimberly and I have built a multi-stage study in an attempt to get at some of these questions.  We settled on calling it “The Winding Path Study” (all credit to Kimberly for the title) and we have organized it around two initial stages, with room for additional exploration.  First, we had to find a conceptual framework that fit the way that people live their lives.  We found one that I think works that comes out of sociology and anthropology called Life Course Perspective. Essentially, this framework describes lives as amazingly complex and almost infinitely unique, yet full of three common elements – trajectories, transitions, and turning points. While Life Course scholars have extended definitions for each of these terms that I won’t try to summarize here, I think we all know what these terms mean just because we can likely point to moments in our own lives where the impact of these concepts became clear.

Next, we built a survey (but of course!) to try to get a better sense of the range of trajectories, transitions, and turning points that our graduates have experienced.  I hoped that we might get 1000 responses.  From these respondents, I hoped that we might find 100 that were willing to participate in a 30 minute interview.

Well, apparently we struck a chord.  We got 1000 responses from Augustana alumni in the first 12 hours of the survey, and finished with 2,792.  In addition, over 1200 respondents said that they would be willing to participate in a 30 minute interview.

I’ll share more about this project in the next several months as we pore through the data. One thing that jumped out at me as I began to watch the data coming in was the extent to which people were willing to tell us surprisingly personal details about their lives.  Our respondents wrote and wrote and wrote. We now have a treasure trove of data that we have to read through and organize.  At the end of this project, however, we will likely have a much greater understanding of the range of life courses that our alums have taken. Better yet, we hope to find some patterns that will help us think about the way that we guide our students during college.

The goals of the Augustana 2020 strategic plan are lofty and complicated.  I’m not sure we even realized how challenging this plan would be when the Board approved it in the winter or when we designed it last fall.  But now that we’ve started to roll up our sleeves, I think we already have information on our graduates that most colleges could only wish that they had.  Now comes the fun part!

Make it a good day,

Mark