Swimming in the 2014 Senior Survey data!

I think I’ve resigned myself to the fact that it is nearly impossible to explain to someone who hasn’t experienced Augustana’s 10-week terms what it feels like to go from standing still less than a month ago to flying by the seat of your pants at week four. But here it is verging on mid-term time, and I’m hurtling through space trying everything I can to get my bearings!

So, to show that the Institutional Research and Assessment staff (AKA Kimberly and I) doesn’t just sit around dreaming up ways to collect more data, I thought I’d share with you . . . more data. (I guess this doesn’t really debunk my assumptions about your assumptions, does it.)

Every spring, we ask our graduating seniors to complete a survey that asks all sort of questions about their experiences at Augustana. In addition, we ask a few important questions that we’ve found to be useful outcome questions (would you choose Augustana again, is your post-graduate plan a good fit for who you are and where you want your life to go, and do you already have a job or grad school place).

It takes a lot of work to process this data into a readable report, but it’s finally finished and posted on the IR web page.  Here is the direct link to the 2014 Senior Survey results.

2014 Senior Survey Report and Findings

Now you could jump on that link right away and start swimming in the data – percentages, averages, and standard deviations (oh, my!). And you might survive the experience, although your eyes will probably start to glaze over as you look at mean score after mean score and your brain will likely start to go soft wondering (rightly so) what exactly each average score means – is it good? is is bad? is it just right? (sort of like Goldilocks in the house of the three bears . . . replacing the bowls of porridge with excel spreadsheets, of course).

So if I may, let me make a suggestion that I hope will make some of this data more meaningful.  Instead of looking at the numbers first, put a sheet of paper over the numbers and look at the questions first.  Reflect on why each question might matter for students and what might be the “about right” distribution of responses.  Pick out 3-5 questions that seem particularly interesting to you.

THEN take away the sheet of paper covering the numbers. Do your musings match up with the average score or the distribution of responses?  What more would you like to know that might help you get a better handle on what we could do to improve, if the difference between your reflections and the actual score suggests that institutional improvement might be valuable?

This data isn’t of much use if it doesn’t help us get better at what we do. And you – the people on the ground floor who are working with students every day – are the ones who are ideally suited to tackle this data, jump into this process, and benefit from the results of your efforts.

If you have any questions, comments, suggestions, or criticisms (I prefer to think of it as constructive feedback!) about the senior survey, PLEASE PLEASE PLEASE contact us – Kimberly or me – in the IR office. Nothing we have built is so important that it can’t be changed . . . especially if those changes make the survey better.

Make is a good day,

Mark

What happens when you build assessment around improvement?

Well hello, everyone!

It’s great to feel the energy on campus again. And it’s exciting to restart my Delicious Ambiguity blog: Season 4 (I’ve been renewed!). Each week I share some tidbit (data that comes from statistics or focus groups) from our own Augustana student data that will help you do what you do just a little bit better the next time you do it. If you’re new to Delicious Ambiguity, you might also want to know that you can search the three years of previous posts (about 100 in all) for everything from athletes to introverts, Greeks to geeks. In addition to a ton of useful findings, you might even find a few funny quips (AKA bewildering side comments).

By now you’ve probably heard me say on at least one occasion that building assessment efforts around genuine improvement, as opposed to doing assessment to find out what’s already happened (i.e., to prove what you think you are already doing), thoroughly changes every part of the assessment process.  More importantly, it’s the only way to actually get better because:

  1. You’ve backward-designed the entire project around finding out what you need to do to get better instead of just finding out what happened, and
  2. You’ve humbled yourself to the possibility of improvement and thereby matched your efforts with the way that educational processes actually work.

I’d like to share an example of one program at Augustana that has clearly benefited from an “assessment for improvement” approach.  My goal here isn’t to brag, but rather to walk you through an example of such a process in the hope that something I share might be applicable to your own unique context.

Augustana has run some version of freshman orientation for a very long time. And by and large, it’s been a pretty successful program. Yet everyone involved has always wondered what they might do to make it just a little bit better. Much of our prior data only told us the proportion of students who were “satisfied” with the experience. Although we could pat ourselves on the back when the numbers were decent (which they were virtually every year), we had no way of turning that information into specific changes that we could trust would actually make the experience demonstrably more effective.

So a few years ago, folks from Student Affairs, Academic Affairs, and the IR office began applying an improvement-centered assessment approach to orientation. First, we talked at length about drilling down to the core learning goals of freshman orientation. Sure, we have lots of things we’d love to result from orientation, but realistically there are only so many things you can do in four days for a group of 18-year-olds who are some combination of giddy, overwhelmed, and panicked – none of which makes for particularly fertile learning conditions.

So with that in mind, we needed to strip our goals down to a “triage” set of learning goals for orientation.  We settled on three concepts.

  • Welcome Week will connect new students with the people necessary for success.
  • Welcome Week will connect new students with the places necessary for success.
  • Welcome Week will connect new students with the values necessary for success.

The people we identified included all of the individuals who might influence a student’s first year experience – other students, student affairs and residential life staff, and specific faculty members. The concept of place involved a) knowing EXACTLY how to walk to one’s classes and specific first-year resources, and b) finding other places on campus that a student might use for emotional rejuvenation as well as intellectual work. The values we discussed focused on clarifying a strategy for getting the most out of a liberal arts college setting. This meant introducing students to a mindset that focuses on actively participating in a process of learning and growth and show them how this approach will increase their likelihood of success, both in the first year and beyond.

Once we spelled out our goals for Welcome Week, then we could set about our work from two directions. First, we could start to alter the design of the experience to meet those goals. Second, we could build a survey that examined the degree to which students came away from Welcome Week connected to the people, places, and values that substantially increase the likelihood of success in the first year.

Over the last two years the survey findings have provided a number of interesting insights into the degree to which certain experiences were already meeting the goals we had set. More importantly, the survey data has become a critical conversation guide for specific improvements. Because the questions were built around specific experiences, it has given everyone – particularly peer mentors – a clear target to shoot for with each student. For example, if the goal was to ensure that each student would say that they knew exactly how to get to their classes on the first day, then the peer mentor could shift from merely pointing at buildings while walking around campus to creating some way for new freshmen to walk right up to the door of the room where their class would be.

At the same time that we were using data to guide specific adjustments, the folks planning Welcome Week examined the design of the entire program. This led them to introduce several changes, including the Saturday morning concurrent sessions titled “Augie 101,” focusing on all kinds of issues that would specifically increase the likelihood of successful academic acclimation.

We will survey the freshmen in the next week or so to find out how the most recent set of changes impacted their experience during Welcome Week. But even without that data, I suspect that the new programming this year improved the experience. My confidence comes from one particularly compelling data point that isn’t a number (I know – sit down and take a deep breath!). During the Augie 101 sessions, peer mentors and other older students who were assisting with Welcome Week kept saying, “I wish we would have had something like this during my Welcome Week experience.”  To me, that is a powerful endorsement of our efforts.

We will likely never be perfect, but we have mounting evidence that we keep getting better at what we do. That doesn’t mean that we have any reason to brag or rest on our laurels.  It just means that we are doing things right.  And that’s what makes doing this work so much fun.

Make it a good day,

Mark

A new (and maybe better) way to understand the impact of an Augustana education

As you probably know by now, the new Augustana 2020 strategic plan places our graduates’ success after college at the center of our institutional mission.  In real terms, this means that what our students learn in college matters to the degree that it contributes to their success after college.  Put another way, even if our students learn all kinds of interesting knowledge and complicated skills, if what they have learned can’t be effectively and meaningfully applied to life after college, then we haven’t really done our job.

Now whether you think that this is the last nail in the liberal arts coffin or the long-awaited defibrillator to revive liberal arts education, our own success hinges on something else that I’m pretty sure we haven’t thought much about. Exactly what are we talking about when we talk about a successful life after college? Do we have a working definition of what might make up a successful life for an Augustana graduate? In order to grapple with those vertigo-inducing questions, we have to know a lot more about what happens to our graduates after college.  But do we have anything more than vague notions about our own graduates’ lives?

I’m afraid that the answers to those questions are probably no, no, and no. In part, it’s because these are big, hard questions.  And to be fair, I don’t know of a college that has tried to get a real handle on these ideas.  So . . . . here we go . . . .

This is the kind of research project that can keep you up at night.  Because it isn’t just about getting data to figure out the relationship between one thing (an Augustana college experience) and another thing (a successful life after college).  For starters, these are two monstrously complicated constructs.  Distilling them down to some essential qualities may well be impossible.  I’m not saying that it’s NOT possible; I’m just admitting to the fact that I’m intimidated by the very idea of trying to identify a set of valid essential qualities. And as if that weren’t enough, we (higher education researchers writ large) have yet to have developed a conceptual framework that is complex enough to account for the almost infinite range of ways in which people’s lives evolve. To date, every effort to link alumni success to their college experience has presumed a straight line – even when we know that very few of us traveled a straight path to get to where we are now.

So over the past six months or so, Kimberly and I have built a multi-stage study in an attempt to get at some of these questions.  We settled on calling it “The Winding Path Study” (all credit to Kimberly for the title) and we have organized it around two initial stages, with room for additional exploration.  First, we had to find a conceptual framework that fit the way that people live their lives.  We found one that I think works that comes out of sociology and anthropology called Life Course Perspective. Essentially, this framework describes lives as amazingly complex and almost infinitely unique, yet full of three common elements – trajectories, transitions, and turning points. While Life Course scholars have extended definitions for each of these terms that I won’t try to summarize here, I think we all know what these terms mean just because we can likely point to moments in our own lives where the impact of these concepts became clear.

Next, we built a survey (but of course!) to try to get a better sense of the range of trajectories, transitions, and turning points that our graduates have experienced.  I hoped that we might get 1000 responses.  From these respondents, I hoped that we might find 100 that were willing to participate in a 30 minute interview.

Well, apparently we struck a chord.  We got 1000 responses from Augustana alumni in the first 12 hours of the survey, and finished with 2,792.  In addition, over 1200 respondents said that they would be willing to participate in a 30 minute interview.

I’ll share more about this project in the next several months as we pore through the data. One thing that jumped out at me as I began to watch the data coming in was the extent to which people were willing to tell us surprisingly personal details about their lives.  Our respondents wrote and wrote and wrote. We now have a treasure trove of data that we have to read through and organize.  At the end of this project, however, we will likely have a much greater understanding of the range of life courses that our alums have taken. Better yet, we hope to find some patterns that will help us think about the way that we guide our students during college.

The goals of the Augustana 2020 strategic plan are lofty and complicated.  I’m not sure we even realized how challenging this plan would be when the Board approved it in the winter or when we designed it last fall.  But now that we’ve started to roll up our sleeves, I think we already have information on our graduates that most colleges could only wish that they had.  Now comes the fun part!

Make it a good day,

Mark

 

Refocusing on the Connections Instead of Just Making Better Parts

This is the second of three posts about our need to reframe, refocus, and refine the way that we operationalize (i.e., deliver) the liberal arts. Near the end of last week’s post (where I suggested reframing how we deliver the liberal arts around enabling our graduates to thrive in the midst of change) I suggested:

So the learning experiences that matter the most may in fact be the things that we consider the least. Right now we focus the most time, resources, and energy on the classes we offer, the activities we organize, the experiences we sponsor. Reframing the way that we deliver the liberal arts means placing increased focus on the way that students connect these experiences and apply the ideas from one experience to succeed in another. Moreover, it means guiding students to strategically set up the ideal set of inter-experience connections that best prepare them to achieve their post-graduate aspirations.

When Augustana was founded, the connections between classes (at the time considered the primary, if not only, learning experiences offered by the college) were a foregone conclusion because the curriculum was virtually identical for every student.  New content assumed the delivery of prior content and students moved in lockstep from beginning courses to advanced seminars.  Only near the end of their schooling were students allowed to deviate from the central educational path to take courses that fit their vocational intentions in law, medicine, the clergy, civil service, etc.  Furthermore, extra-curricular experiences weren’t seen as potential learning experiences since they didn’t have anything to do with the content delivered through the curriculum.

This earlier version of a liberal arts education and the one that we now endeavor to provide could not be more different.  While the curriculum of yesteryear was almost entirely predetermined both in terms of the courses one took and when one took them, today (although some majors are more prescribed than others) I doubt you could find two students who took the same courses in the same order during the same year – let alone throughout an entire undergraduate degree.  In addition, we now know that students develop, learn, and grow at least as much through their courses as they do through their out-of-class experiences, resulting in the wide support for everything from student organizations to study abroad.

In the context of all of this curricular and co-curricular opportunity, it’s no wonder that students’ effort to convey the impact of their college experience on a resume often devolves into a list where the length of that list is assumed to convey something about an individual’s preparation and potential for future success.  Of course, those considering recent college graduates for graduate school, employment, or long-term service have figured out that these lists are a ruse and can be a red flag for someone who is more surface than substance.

But if we refocus the way that we operationalize the liberal arts so that students highlight “why” they chose the experiences they chose and how they took charge of constructing the person they have become, the grocery list of college experiences (AKA resume) suddenly comes to life as a story of perpetual improvement. This doesn’t mean that they are perfectly constructed when they receive their diploma.  But it does mean that those students can probably tell their own story in a way that shows an emerging clarity of purpose and an accelerating sense of momentum toward it.  We all know from our own experiences that those students stand out even when they aren’t trying to make an impression.

So how might we operationalize the way that we deliver the liberal arts to highlight this new focus?

First of all, we don’t need to go back to the days of an overly prescribed college experience.  With the diversity of our students’ pre-college experiences, learning needs and interests, and post-graduate aspirations, treating our students as if they were all the same would be stunningly foolish.

Instead, we begin by mapping every activity and every course that students can take in terms of what learning is intended to emerge from that experience and how that learning contributes to the larger learning goals and mission of the college.  Since this mapping is intended to be an iterative experience, the exercise may well result in adapting, adding, or even subtracting some courses or experiences.  It might also result in altering some experiences to more specifically meet certain learning goals.  The primary result of this exercise is not just to produce a complete catalog of the learning experiences in which students can engage.  Instead, the goal is to produce customize-able flow charts that show the variety of ways that different types of students can identify a sequence of experiences that together cultivate the learning that each student need to fully prepare them to succeed after they graduate.

These maps become the primary tools for the college to help students construct a college experience that builds upon their pre-college experiences and abilities, fills in the areas in which they need additional opportunities to learn and grown, and gives them the best chance to be the kind of person they aspire to be when they graduate.  Ultimately, the totality of each student’s college experience can be conveyed through a cohesive narrative that tells the story of his or her college journey from start to finish.

The challenges to making such a refocus work are not without consequence.  Most important, we have to actually enact our commitment to student growth and development in everything that we do.  That likely means changing something that we currently do (even if it is something we really like to do a certain way) to make it more educationally effective for students.  We often ask the question during planning conversations, “But what are we going to take away?”  This mapping exercise often identifies things that we could and probably should take away.  The challenge is whether we are willing give those things up.

In broader terms, this means that we have be able to “zoom out” and see the forest instead of the trees.  There will be a myriad of ways that a student could put together the learning experiences necessary for post-graduate success. The most important goal here is that the students can lay out their path, retrace their steps and explain why they took each one of them, situating the reasons for their choices in the context of their post-graduate aspirations. Of course there will likely be students who, despite all of our best efforts, don’t follow the guidance that we provide for them. But if all of our students learn the value of thinking about their own lives as a strategic effort to grow and develop, the chances are pretty good that they will all be on their way to succeeding in life and embodying the results of a liberal arts education when they walk across the stage to accept their diploma.

Make it a good day,

Mark

For the want of a response, the data was crap

Any time I hear someone use data from one of the new freshman, senior, or recent graduate surveys to advocate for a particular idea, I can’t help but smile a little.  It is deeply gratifying to see faculty and administrators comfortably use our data to evaluate new policy, programming, and strategic direction ideas.  Moreover, we can all point to a growing list of data-driven decisions that we know have directly improved student learning.

So it might seem odd, but that smile slips away almost as quickly as it appears. Because underneath this pervasive use of data lies a deep trust in the veracity of those numbers. And the quality of our data depends almost entirely upon the participation of 18-22 year-olds who are . . . . let’s just say “still developing.”  Data quality is like milk – it can turn on you overnight. If the students begin to think that survey questions don’t really apply to them or they start to suspect that the results aren’t valued by the college, they’ll breeze through the questions without giving them much thought or blow off the survey entirely. If that happens on a grand scale . . . . I shudder to think about it.  So you could say that I was “mildly concerned” as I organized fall IDEA course feedback forms for processing a few weeks ago and noticed several where the only bubbles colored in were “fives.”  A few minutes later I found several where the only darkened bubbles were “ones.”

Fortunately, a larger sampling of students’ IDEA forms put my mind at ease.  I found that on most forms the distribution of darkened circles varied and, as best as I could tell, student’s responses to the individual questions seemed to reflect at least a minimal effort to provide truthful responses.  However, this momentary heart attack got me wondering: to what degree might student’s approach to our course feedback process impact the quality of the data that we get?  This is how I ended up in front of Augustana’s student government (SGA) earlier this week talking about our course feedback process, the importance of good data, the reality of student’s perceptions and experiences with these forms, and ways that we might convince more students to take this process seriously.

During this conversation, I learned three things that I hope you’ll take to heart.  First, our students really come alive when they feel they are active participants in making Augustana the best place it can be.  However, they start to slip into passive bystanders when they don’t know the “why” about processes in which they are expected to be key contributors.  When they become bystanders, they are much less likely to invest their own emotional energy in providing accurate data.  Many of the students honestly didn’t think that the IDEA data they provided on the student form was used very often – if ever. If the data doesn’t really matter anyway, so their thinking goes, the effort that they put in to providing it doesn’t matter all that much either.

Second, students often felt that not all of the questions about how much progress they made on specific objectives applied in all classes equally.  As I explained to them how the IDEA data analysis worked and how the information that faculty received was designed to connect the objectives of the course with the students’ sense of what they learned, I could almost hear the light bulbs popping on over their heads.  They were accustomed to satisfaction-type surveys in which an ideal class would elicit a high score on every survey question.  When they realized that they were expected to give lower scores to questions that didn’t fit the course (and that this data would be useful as well), their concern about the applicability of the form and all of the accompanying frustrations disappeared.

Third, even though we – faculty, staff, and administrators – know exactly what we mean when we talk about learning outcomes, our students still don’t really know that their success in launching their life after college is not just a function of their major and all the stuff they’ve listed on their resume.  On numerous occasions, students expressed confusion about the learning objectives because they didn’t understand how they applied to the content of the course.  Although they may have seen the lists of skills that employers and graduate schools look for, it seems that our students think these are skills that are largely set in stone long before they get to college, and that college is mostly about learning content knowledge and building a network of friends and “connections.”  So when they see learning objectives on the IDEA forms, unless they they have been clued in to understand that these are skills that the course is designed to develop, they are likely to be confused by the very idea of learning objectives above and beyond content knowledge.

Although SGA and I plan to work together to help students better understand the value of the course feedback process and its impact on the quality of their own college experience, we – faculty, staff, and administrators – need to do a much better job of making sure that our students understand the IDEA course feedback process.  From the beginning of the course, students need to know that they will be learning more than content.  They need to know exactly what the learning goals are for the course. Students need to know that faculty want to know how much their students’ learned and what worked best in each class to fuel that learning, and that satisfaction doesn’t always equate to learning.  And students need to know how faculty have used course feedback data in the past to alter or adapt their classes.  If you demonstrate to your students how this data benefits the quality of their learning experience, I think they will be much more willing to genuinely invest in providing you with good data.

Successfully creating an evidence-based culture of perpetual improvement that results in a better college requires faculty, staff, and administrators to take great care with the sources of our most important data.  I hope you will take just a few minutes to help students understand the course feedback process.  Because in the end, not only will they benefit from it, but so will you.

Make it a good day,

Mark

 

 

 

 

Could a focus on learning outcomes unwittingly sacrifice process for product?

A central tenet of the learning outcomes movement is that higher education institutions must articulate a specific set of skills, traits, and/or dispositions that all of its students will learn before graduation. Then, through legitimate means of measurement, institutions must assess and publicize the degree to which its students make gains on each of these outcomes. Although many institutions have yet to implement this concept fully (especially regarding the thorough assessment of institutional outcomes), this idea is more than just a suggestion. Each of the regional accrediting bodies now requires institutions to identify specific learning outcomes and demonstrate evidence of outcomes assessment as a standard of practice.

This approach to educational design seems at the very least reasonable. All students, regardless of major, need a certain set of skills and aptitudes (things like critical thinking, collaborative leadership, intercultural competence) to succeed in life as they take on additional professional responsibilities, embark (by choice or by circumstance) on a new career, or address a daunting civic or personal challenge. In light of the educational mission our institutions espouse, committing ourselves to a set of learning outcomes for all students seems like what we should have been doing all along.

Yet too often the outcomes that institutions select to represent the full scope of their educational mission, and the way that those institutions choose to assess gains on those outcomes, unwittingly limits their ability to fulfill the mission they espouse. For when institutions narrow their educational vision to a discrete set of skills and dispositions that can be presented, performed, or produced at the end of an undergraduate assembly line, they often do so at the expense of their own broader vision that would cultivate in students a self-sustaining approach to learning. What we measure dictates the focus of our efforts to improve. As such, it’s easy to imagine a scenario in which the educational structure that currently produces majors and minors in content areas is simply replaced by one that produces majors and minors in some newly chosen learning outcomes. Instead of redesigning the college learning experience to alter the lifetime trajectory of an individual, we allow the whole to be nothing more than the sum of the parts – because all we have done is swap one collection of parts for another. Although there may be value in establishing and implementing a threshold of competence for a bachelor’s degree (for which a major serves a legitimate purpose), limiting ourselves to this framework fails to account for the deeply-held belief that a college experience should approach learning as a process – one that is cumulative, iterative, multi-dimensional, and, most importantly, self-sustaining long beyond graduation.

The disconnect between our conception of a college education as a process and our tendency to track learning as a finite set of productions (outcomes) is particularly apparent in the way that we assess our students’ development as life-long learners. Typically, we measure this construct with a pre-test and a post-test that tracks learning gains between the years of 18 and 22 – hardly a lifetime (the fact that a few institutions gather data from alumni five and ten years after graduation doesn’t invalidate the larger point). Under these conditions, trying to claim empirically that (1) an individual has developed and maintained a perpetual interest in learning throughout their life, and that (2) this life-long approach is direct attributable to one’s undergraduate education, probably borders on the delusional. The complexity of life even under the most mundane of circumstances makes such a hypothesis deeply suspect. Yet we all know of students that experienced college as a process through which they found a direction that excited them and a momentum that carried them down a purposeful path that extended far beyond commencement.

I am by no means suggesting that institutions should abandon assessing learning gains on a given set of outcomes. On the contrary, we should expect no less of ourselves than substantial growth in all of our students as a result of our efforts. Designed appropriately, a well-organized sequence of outcomes assessment snapshots can provide information vital to tracking student learning over time and potentially increasing institutional effectiveness. However, because the very act of learning occurs (as the seminal developmental psychologist Lev Vygotsky would describe it) in a state of perpetual social interaction, taking stock of the degree to which we foster a robust learning process is at least as important as taking snapshots of learning outcomes if we hope to gather information that helps us improve.

If you think that assessing learning outcomes effectively is difficult, then assessing the quality of the learning process ought to send chills down even the most skilled assessment coordinator’s spine. Defining and measuring the nature of process requires a very different conception of assessment – and for that matter a substantially more complex understanding of learning outcomes. Instead of merely measuring what is already in the rearview mirror (i.e., whatever has already been acquired), assessing the college experience as a process requires a look at the road ahead, emphasizing the connection between what has already occurred and what is yet to come. In other words, assessment of the learning that results from a given experience would include the degree to which a student is prepared or “primed” to make the most of a future learning experience (either one that is intentionally designed to follow immediately, or one that is likely to occur somewhere down the road). Ultimately, this approach would substantially improve our ability to determine the degree to which we are preparing students to approach life in a way that is thoughtful, pro-actively adaptable, and even nimble in the face of both unforeseen opportunity and sudden disappointment.

Of course, this idea runs counter to the way that we typically organize our students’ postsecondary educational experience. For if we are going to track the degree to which a given experience “primes” students for subsequent experiences – especially subsequent experiences that occur during college – then the educational experience can’t be so loosely constructed that the number of potential variations in the ordering of different students’ experiences virtually equals the number of students enrolled at our institution. This doesn’t mean that we return to the days in which every student took the same courses at the same time in the same order, but it does require an increased level of collective commitment to the intentional design of the student experience, a commitment to student-centered learning that will likely come at the expense of an individual instructor’s or administrator’s preference for which courses they teach or programs they lead and when they might be offered.

The other serious challenge is the act of operationalizing a concept of assessment that attempts to directly measure an individual’s preparation to make the most of a subsequent educational experience. But if we want to demonstrate the degree to which a college experience is more than just a collection of gains on disparate outcomes – whether these outcomes are somehow connected or entirely independent of each other – then we have to expand our approach to include process as well as product.  Only then can we actually demonstrate that the whole is greater than the sum of the parts, that in fact the educational process is the glue that fuses those disparate parts into a greater – and qualitatively distinct – whole.

Make it a good day,

Mark

The Fallacy of Matching Majors with Careers

It seems that most of the talk in recent months about the ROI (return on investment) of a college degree from a given institution has been focused on the degree to which new graduates from that institution can get well-paying jobs related to their major.  For liberal arts colleges and those of us who believe in the importance of a well-rounded education, the whole idea of assuming an inherent connection between major choice and career seems problematic.  Not only are there plenty of majors that don’t have a natural correlate on the job market (e.g., philosophy majors come to mind), but we are also regularly bombarded with the claims that individuals in today’s world will hold multiple jobs in multiple professions over the course of their working careers. Thus it seems odd to suggest that a college’s effectiveness could be pinned to the proportion of graduates who have landed jobs in their field within six months of graduation.

One data point from our survey of recent graduates seems to highlight this conundrum. Nine months after a class of seniors graduates, we ask them to complete a survey that asks a variety of questions about their current status, the degree to which their Augustana experience helped prepare them for their present circumstance, and the degree to which they believe that they are on the right long-term path.

One of the questions we asked our 2012 graduates last spring (about nine months after they had received their BA degrees from Augustana) was:

“Have your long-term professional goals changed since you graduated from Augie?”

The distribution of responses was revealing.

Not at all

48%

A little

21%

Somewhat

20%

Substantial

4%

Completely

3%

In other words, fewer than 50% of the 2012 graduating class considered themselves on the exact same long-term path that they were on when they walked across the stage to collect their diplomas.  In addition, over a quarter of the respondents said that their long-term goals had changed “somewhat,” “substantially,” or “completely.”

I believe the result of this single question holds critical implications for our efforts to best prepare our students to succeed after college.  First of all, this finding supports what we already know to be true – many of our students are going to change their long-term goals during their first several years after graduation. This is what happens to young people during their first foray into the world of working adulthood. We would be foolish to tie ourselves too tightly to a data point that doesn’t allow for these natural developments in the life a young adult.

Second, rather than mere job or graduate school placement, we would be smart to begin thinking about our students’ post-graduate success in terms of direction and momentum. Our students need to develop a clear sense of direction in order to decide what the best “next step” is for them. In addition, our job is to help them know when to take that “next step,” whether it be getting into the right graduate school or finding the right job or taking advantage of a once-in-a-lifetime opportunity that will better position them to move in the direction they have chosen for themselves. If we can do that, then no matter what happens to our students in the years after they graduate, they will be better able to succeed in the face of life’s inevitable challenges.

In concert with a sense of direction, our students need momentum.  This momentum should be self-perpetuating, cultivated by the right mix of motivations to handle setbacks and success. More importantly, it needs to be strong enough to thrive in the midst of a change in direction. This means that we develop their ability to be autonomous while holding themselves to high standards.  It means that they know how to be strategic in staying true to themselves and their goals no matter the distractions that might appear.

This doesn’t mean that we shouldn’t care about our students’ success in applying to graduate school or entry-level jobs in a given profession. On the contrary, we absolutely should care about statistics like these – especially if they support a student’s chosen direction and momentum.  But we should remember that a successful life isn’t etched in stone upon graduation from college.  And we should have the courage to track our students’ life trajectory in a way that doesn’t limit both us and them.

Make it a good day,

Mark

 

 

How does student learning happen?

Since it’s finals week, I’ll be quick.  However, I hope you’ll take some time to think about this little tidbit below as our strategic planning conversations address examine how we are going to make sure that every student develops the ability to integrate ideas to solve complex problems.

I saw George Kuh give a talk on Saturday afternoon in which he showed the following cartoon.  Even though the whole audience found it funny, the point he was trying to make about the degree to which we often fail to ensure that students learn what we say we teach them was dead serious.

We claim that a liberal arts education teaches students how to integrate disparate ideas from a wide range of disciplines and contexts to solve complex 21st century problems.  At the same time, however, the experiences we require are specific to individual disciplines or topics while the truly integrative experiences remain optional add-ons . . . if they exist at all outside of the major.

So the question I’d ask you to think about is this:  How do we know that every student participates in a rigorously designed activity that explicitly develops the ability to integrate knowledge from multiple fields of study to solve substantive, complex problems? And how could we design a college experience where we could demonstrate that every student participated in such an activity?

Make it a good day.  And have a great fall break.

Mark

Sometimes assessing might be the wrong thing to do

Because of the break-neck pace of our work lives, we tend to look for pre-determined processes to address problems instead of considering whether or not there is another approach that might increase the chances of a successful long-term solution.  This makes sense since pre-determined processes often feel like they help to solve complicated problems by giving us a vetted action plan.  But if we begin defaulting to this option too easily, we can sometimes create more work for ourselves just because we absentmindedly opted for “doing it the way we’re supposed to do it.”  So I thought it might be worthwhile to share an observation about our efforts to improve our educational effectiveness that could help us be more efficient in the process.

We have found tremendous value in gathering evidence to inform our decisions instead of relying on anecdotes, intuition, or speculation.  Moreover, the success of our own experiences seems to have fostered a truly positive sea-change both in terms of the frequency of requests for data that might inform an upcoming discussion or decision as well as the desire to ask new questions that might help us understand more deeply the nature of our educational endeavors.  So why would I suggest that sometimes “assessing might be the wrong thing to do?”

First, let’s revisit two different conceptions of “assessment.”  One perceives “assessment” as primarily about measuring.  It’s an act that happens over a finite period of time and produces a finding that essentially becomes the end of the act of measuring.  Another conception considers assessment as a process composed of various stages: asking a question, gathering data, designing an intervention, and evaluating the effectiveness of that intervention.  Imagine the difference between the two to mirror the difference between a dot (a point in time) and a single loop within a coil (a perpetually evolving process).  So in my mind, “measurement” is a singular act that might involve numbers or theoretical frameworks. “Assessment” is the miniature process that includes asking a question, engaging in measurement of some kind, and evaluating the effectiveness of a given intervention.  “Continuous improvement” is an organizational value that results in the perpetual application of assessment.  The focus of this post is to suggest that we might help ourselves by expanding the potential points at which we could apply a process of assessment.

Too often, after discovering the possibility that student learning resulting from a given experience might not be what we had hoped, we decide that we should measure the student learning in question.  I think we expect to generate a more robust set of data that confirms or at least complicates the information we think we already know. Usually, after several months of gathering data (and if all goes well with that process) our hunch turns out to be so.

I’d like to suggest a step prior to measuring student learning that might get us on track to improvement more quickly.  Instead of applying another means of measurement to evaluate the resultant learning, we should start by applying what we know about effective educational design to assess whether or not the experience in question is actually designed to produce the intended learning.  Because if the experience is not designed and delivered effectively, then the likelihood of it falling short of its expectations are pretty high.  And if there is one truth about educating that we already know, it’s that if we don’t teach our students something, they won’t learn it.

Assessing the design of a program or experiences takes a lot less time than gathering learning outcome data.  And it will get you to the fun part of redesigning the program or experience in question much sooner.

So if you are examining a learning experience because you don’t think it’s working as it should, start by tearing apart its design.  If the design is problematic, then skip the measuring part . . . fix it, implement the changes, and then test the outcomes.

Make it a good day,

Mark

 

 

Planning, Doing, Being

Unless you’ve been holding your breath at the bottom of the slough for the past six months, you know that we are smack in the middle of developing a new strategic plan for Augustana College.  This weekend our Board of Trustees hold their annual fall meetings during which President Bahls and Dean Lawrence will provide an update to the board, answer questions, address criticisms and concerns, and work with board members to refine the strategic directions that will be prioritized in the final plan.  If you haven’t done so already, I’d highly recommend that you take some time to look at the current state of this process here.

After living in the inner sanctum of this process for the last six months, I’ve been struck by how difficult it is to effectively link the abstract aspirations of vision, mission, and strategic direction with the concrete actions, specific tactics, and measurable moments that we think will prove whether or not we have accomplished our plans.  If we lean too hard to one side, we could end up with little more than strategery – a word I use in all seriousness here because it manages to capture what happens when vision gets disconnected from any actual means of demonstrating its achievement on the ground (click here to see the origins of this word – we are in your debt, Will Ferrell.)  And if we lean too far to the other side, we can fall into the trap of simply adding a host of new programs, policies, activities, and experiences under the flawed belief that busy is always better.  If we’re honest with ourselves, I suspect we’d have to admit that we’ve driven over both of these potholes in recent years as we’ve genuinely tried to make Augustana better – in the present and for the future.

In the face of these difficulties, I understand the temptation to be silly about it and throw the strategic planning baby out with the tactical bathwater.  But that would be – in a word – stupid.  A primary reason why higher education is in such trouble these days is because so many institutions believed that they didn’t really have to plan ahead (or that anything might change over time) because they thought there would always be lots of students who would pay whatever the institution charged to sit at the feet of masters and learn whatever was taught.

Frankly, I really like a lot of what is going to be proposed and discussed this weekend. However, we are always faced with the challenge of following through.  How are we going to walk this thing out to its fullest completion, and will we really have chosen the right metrics to demonstrate the degree to which we have achieved the goal we set out to accomplish?

All of these thoughts were bouncing around in my head as I watched two TED Talks by Derek Sivers over the weekend.  Although both of them are only about three minutes long, they made me think a lot about how we might go from the laudable abstractions of mission, vision, and strategic directions to the simple, sustainable, and concrete evidence that will demonstrate to everyone whether we have reached the goals we set for ourselves.

The first TED Talk focuses on a key element of success for individuals who set goals for themselves.  The crux of his point is that those who talk too much about what they intend to accomplish can sometimes fool themselves into thinking that they have already accomplished it.  I’ve often heard a nearby college’s strategic plan described as, “Fake it ’til you make it.”  Yet there are a myriad of colleges and universities that became more selective simply by declaring themselves to be more selective.  In the end, the quality of the education they provided didn’t change a bit.  In terms of making our strategic plan something worth the kilobytes it’s saved on, we might be careful to talk more about the things we need to do or be today in order to achieve our long-term goals, and talk less about publicizing the institution we will become and the prestige we will acquire as if we were already well on our way to getting there.

The second TED Talk teases out a critical and oft overlooked moment in the origins of a social movement.  Sivers shows a video of an impromptu dance party on a hillside.  The point he makes seems to be particularly applicable to our work once the strategic plan is finalized.  Essentially, he emphasizes the leadership effect of the first follower – the individual who finds something great and has the guts to jump up and join in.

I’m sure there are several other potentially important take-aways from these clips.  I wanted to share them with you in the hopes that something from them might help us move from planning to doing to being.

Make it a good day,

Mark