Sunday, May 24, 2015

Ignorance is Not an Excuse

I've been reflecting a lot on the issue of professors and education technology of late.  In particular, I've been thinking about the exact issue raised by the tweet pasted above: we have reached a point where it is no longer acceptable to be unable to use classroom technology and the extensive suite of learning tools available to support student learning.  We are horrified if a professor shows up to class unprepared; yet regularly turn a blind eye to colleagues who refuse to invest the time and energy necessary to master the latest ed tech tools and research; and integrate them into their classrooms.  Let me be clear: I'm not suggesting that all faculty need to learn how to code.  Rather, I'm suggesting--to give one obvious example--that it is malpractice to continue to teach by lecture and high stakes midterms when we now know that there are many other, more effective ways to teach large classes; and we now have the technological capacity to implement at least some of those tools (e.g. online practice quizzes; pre-recorded content lectures; student response systems like i>clicker).

Likewise, if one is in any kind of leadership role, with decision-making authority, it is simply inexcusable to be unaware of the vigorous conversation about the role of education technology in higher education.  In particular, it is essential for our college/university leaders to grasp what is at stake when a decision is made about, for instance, outsourcing online course development to the private sector.  Higher education, especially public education, has become deeply politicized.  Debates about the role of education technology are central to the politics of higher education, in part because there is a lot of money at stake.  As campus IT evolves into a complex (and expensive) part of the campus infrastructure, more and more resources are directed away from instructional budgets and towards IT departments (in various configurations).  This includes the creation of entirely new units in such areas as Learning Sciences and Data Analytics.  This isn't a bad thing; it has the potential to be quite a good thing.  But it is a big change.

This reallocation of resources is  largely invisible to most teaching faculty and even department chairs; yet an awareness of it is essential.  For one thing, it might finally demonstrate to departments that our universities are in the midst of significantly redefining the way they accomplish their mission.  Rank and order faculty remain ignorant of this important conversation at their own risk.  Departments with graduate programs have an ethical obligation to ensure that their graduate students are trained in the latest best practices, including a strong knowledge of online course design and implementation; and a basic knowledge of important ed tech tools.  The graduate students of today will be expected to have this skill set when they apply for jobs, and many of them will be asked to teach both in classrooms and online.  It is inexcusable to not prepare those who pursue academic jobs for this reality.

For many decades, it was possible for faculty, and especially for our department leadership, to be disconnected from the larger, national discussions about higher education.  It is surprising to me how few faculty have any real sense of the vigorous debates currently happening, especially around the issue of the private sector elbowing its way into both K-12 and higher education.  It is simply impossible for departments (and Colleges within universities) to make wise decisions about where to direct resources if you don't understand this encroachment by the private sector--and what factors are fueling it.  Most faculty have little sense of where their students are taking classes.  They don't realize that, over the past three years, the number of students taking courses outside of the university--either online, at community colleges, or even at other 4 year institutions--has increased exponentially, to the point that a college like mine (Liberal Arts) has already outsourced an unacceptable amount of our General Education/Core Curriculum courses.  We should be having vigorous conversations about how to reverse this trend, including how to integrate our own, high-quality online courses into our curriculum.  It is frustrating to me to see that, if these conversations are happening, they are happening at very high levels and largely exclude faculty.  It is equally frustrating, though, to see my faculty peers express so little interest in these important conversations.

I appreciate that faculty are busier than ever these days.  It would be helpful if institutions did more to ensure that faculty are educated about policy conversations but also, that faculty are held accountable for using best practices in their classrooms.  In any other profession, an individual who refused to remain up to date on basic technological developments would no longer be able to work in that field.  This has become a real issue in medicine, as more and more sophisticated diagnostic tools come out (e.g the Da Vinci robot for abdominal surgeries).  Physicians in procedure-intensive specialties regularly have to learn how to use new devices--otherwise they would lose patients and business.  Likewise, it would be considered malpractice for a physician to be unaware of the latest research and medications for various conditions.  Imagine a rheumatologist who ignored all the new studies about the dangers of long-term use of prednisone and continued to prescribe it at high doses to his patients!

Out of date or sub-par teaching won't kill anyone, to be sure. Eventually, though, students and their parents will become more savvy and less willing to tolerate poor learning experiences.  The result won't be that tenured faculty are fired for poor teaching, however.  Instead, the persistence of faculty resistance to ed tech and best practices will be used in higher education debates to persuade the leaders of our institutions that the teaching of, especially, first and second year courses can't be left to the luddite faculty who refuse to update their practices.  This will (and, in some places, already has) lay the foundations for outsourcing these courses to private-sector companies; and to the continued redirection of resources away from departmental instructional budgets.  If departments want to hire more positions, the best thing they can do is figure out how to reclaim the credit hours that we have lost--and will continue to lose in escalating numbers.  We are approaching a tipping point.  Soon it will be too late, if it isn't already too late.

One of the best historians of education technology is Audrey Watters.  She regularly posts her public presentations to her site (see The Golden Lasso of Education Technology for a recent example).  I am also a huge fan of this collection of her essays, The Monsters of Education Technology (a bargain at $4.99 on Kindle; 9:99 for pb).  Another fantastic way to get caught up on the conversation about higher education and the politics of education technology?  Twitter.  Really.

Tuesday, May 19, 2015

Another Funny Thing Happened....

With thanks to Laura Gibbs for sending me this perfect cartoon!
You know something is up when you wake up at 7:30 am to an overflowing inbox and a large number of notifications on Twitter.  All my early bird friends let me know that Steve Kolowich at the Chronicle of Higher Education had picked up some bits of the Online Rome staffing saga.  The article has the somewhat odd title of "When your Online Course is Put up for Adoption."  It opens as follows: "Jennifer V. Ebbeler always knew that somebody else would end up teaching her online Roman-history course. But that didn’t make giving it up any easier."  Let me be clear, in case it hasn't been utterly clear to everyone who reads this blog: I have never taught the online course--I only designed and built it.  I never intended to teach it except randomly and in the distant future.  I had no problem giving it up; in fact, I couldn't wait to unload it. Indeed, I learned of the department's bizarre decision to hire a non-Romanist as course coordinator from a Facebook post in which I celebrated handing it off.  There was no regret, no second thoughts.  I WAS DONE. I bought myself a pair of expensive earrings to celebrate. 

It had been a long and difficult two years, made much more challenging by severe health issues over this past year.  Despite those issues, I felt like I owed it to my team to continue the project (thus letting them remain employed).  Especially this spring, it was incredibly difficult to find the energy to do the work on the course.  I motivated myself by thinking about the big picture: we needed the course to be finished this spring so that funding and staffing could be in place for the Fall 2015 semester, when the course would be in the Classics Department's possession.  I had also worked hard with my PM and the Asst Dean who oversees our development studio (LAITS) to get the funding in place.  This required a great deal of work and time and politics, especially on the part of Joe TenBarge, the Asst Dean.  The department was totally uninvolved in getting the funding lines in place.  But, when I finished the last of the development, it seemed like everything was finally coming together.  It felt like it was worth all the horrible days of working on it this spring.

Because I had designed the course for others to run and teach, I also put in place a transition plan.  It's not a typical "sage on the stage" MOOC-model course.  There are no video lectures; I appear nowhere in the course.  It focuses on active learning and, especially, the use of primary source materials.  Students work through modules and create their own narrative through a kind of Socratic method of course design.  We draw on documentaries, primary sources, and other things for content.  But thinking and questioning are key features of the course.  One of the reasons students like it, and why it produces high levels of student learning, is precisely because of this unusual model (unusual not because of the focus on active learning but because we do this at a scale of 1 instructor/100 students).

Let me be clear that my issue has nothing to do with wanting control or with wanting "my chosen successor" appointed.  In fact, what I suggested in a document dated 9 March is that the department continue the current instructor's appointment.  As I said, he couldn't succeed me in a job I never did--it's always been his job.  But, as the course transitioned to the department, I was no longer able to appoint him.  That became the responsibility of the department chair.  The main motivation for my suggestion was an interest in ensuring a smooth transition.  The plan I suggested would have had the current instructor be "course coordinator."  One thing that is not clear to most people: this is one course, but with multiple sections, taught to somewhere between 300-500 students (depending on how many sections are run; the College is requiring at least three/semester to keep the funding for the position).

So, in truth, it's about running a small business, working with at least some grad students who aren't content experts and have no experience in online teaching but who will be "instructing" sections of 100 students.  It's a model that can work, I think, but it needs for certain key components to be in place.  One of those is a course coordinator/mentor/supervisor who knows the course inside and out and knows how to teach online.  Of course, over the year, I expected that others would learn how to run the course through the experience of training under an experienced course coordinator.  My goal, always, was sustainability.  That requires many people with the ability to teach the course at any given time.

I am the only member of my department to have taught a course over 250 or so students.  For two years I taught a 400 student Intro to Ancient Rome class.  It was a huge production and logistical challenge.  It's not just like teaching 200 student classes times two (which is the common assumption of those who have never done it).  It's about ten times the work and complexity.  The current instructor dealt with this in Fall 2014, when we had over 300 students in the course.  It was his first experience of such a set-up, which included managing graders; it was a real challenge and he learned a lot by trial and error.  My concern is that, now, the Classics Department is asking someone with no experience in any of the key components required to run the course to step in and a. run a huge, multi-section course; b. know how to manage, support, and train the instructors of the individual sections.

Thankfully, LAITS had a kind of back-up plan in place.  They have hired the current instructor to teach the Extended Campus version of the course in the Summer and Fall 2015.  In the Summer, he will be focused entirely on converting the 15 week course into a five week course while a graduate student from the department is the instructor.  He will work closely with that grad student and mentor him as much as possible on the challenges and tricks of online instruction.  Likewise, he will work with the course coordinator in the fall semester.  I was relieved to learn of this plan and I think it will do a lot to keep things from being a total disaster in the fall.  But it depends on the fact that the current instructor is extremely gracious--far more than I would be in a similar circumstance.  It also leaves unaddressed the larger issue that the course coordinator is far from a content expert or online teaching expert, and yet is going to need to mentor several graduate instructors each semester.  I would understand if there had been no other option but that just wasn't the case.

I'm a bit disappointed that the article's melodramatic (and oddly gendered) tone encourages comments about me instead of about the issues at stake here.  This isn't about me or my course design.  In fact, a key component of the project from the start was figuring out a sustainable model at scale--all without overly compromising on the quality.  It was a huge challenge but we did it.  The course is an excellent example of a quality online course that is "efficient"--or at least more efficient than our current campus version; and works at a 1/100 scale.  It also, ideally, provides graduate students the opportunity to teach online under the supervision of an experienced course coordinator.  Finally, it created two lecturer positions for my department for next year, with money from my college as well as LAITS.

In return for taking on this project, I received nothing.  I was not paid a stipend.  My time was paid for, including summer funding; but, for instance, the 20 hours/week this spring didn't come close to paying for the 60 hour weeks the project required.  It's not at all clear that the creation of this course will "count" for anything professionally.  After all, it's not a monograph.  I get no royalties, which I would get if this were a published textbook.  I didn't even get a thank you from anyone.  In essence, I donated two potentially productive years of my career to a project that I cared deeply about and was intellectually challenging to me; but which benefited my career and bank account not at all.  I'd have been better off to design courses for online programs for a negotiated stipend (if it was the money that I cared about).

The real lesson of my experience is not that your baby can be taken away (ugh!); or that instructors can be separated from their courses (duh!);  but that, at the moment, institutions are still playing catch-up in terms of policy and infrastructure around the delivery of online courses to campus-based students.  What I found, over and over, was that we were inventing policy and procedures as we went. A big part of the problem is the fact that the development of these courses is done in non-academic units, but then the courses are handed over to academic units to manage.  Sometimes this is ok.  But if the department isn't up to date on the latest best practices of online education; or if a chair decides to make unilateral decisions and not collaborate with the people who are familiar with the course and its design, it can lead to real problems.

The reason I have made such a sensitive issue public is to try to encourage more conversation (and action) on the policy front.  If administrators are going to ask faculty to take risks, to sacrifice years of their time to projects like these, then there need to be clear and rational policies.  For instance, despite the tradition that departments control all staffing decisions, with online courses (which are very expensive to develop and keep updated), this is probably not the best procedure, especially if the main concern is ensuring a quality course and learning experience for our students.  I want to encourage my smart and talented colleagues to take on a project like this instead of focusing entirely on work that is great for their career but doesn't do much for the larger community (i.e. writing monographs and articles).  At the same time, for this to happen on a larger scale, we need to get policies in place.  Faculty want to take risks, they want to take on projects like this.  I think most of us have no problem handing our courses over to others.  What we want are some assurance that the course we spent two years building will be treated well and, hopefully, run as it was designed to run.  This seems like a reasonable expectation to me vis-a-vis Online Rome, given that I have essentially donated two years of work to my department and have received no extra compensation.

As we venture forward into this brave new world, we also need to make sure that all decision-makers are educated in the areas about which they are making important decisions; and ensure that there is campus support for all online instructors/course coordinators.  This brave new world of online course design and delivery is, potentially, a fun, stimulating, and surprising one.  It is also one that doesn't quite work like our traditional campus education.  Online teaching requires a lot of different skills than does classroom teaching.   I'm confident that, eventually, all these pieces will get into place on our campuses.  My aim in discussing my experience is entirely to provide a kind of "case study" for why it matters that we get these policies in place sooner rather than later.

Monday, May 18, 2015

Administering Exams in Online Classes

One of the persistent challenges in teaching online is designing assessments that are credible, reliable, and resistant to academic dishonesty.  In smaller courses, this can be done by avoiding exams and turning to papers, blogs, creative writing, and other kinds of activities that can be gathered into a final portfolio.  As much as I would love to assess students using something other than quizzes and exams in Online Rome, it just isn't feasible given the large enrollments that we are required to maintain.  We do make a point of making written work some part of the grade, in the form of essays and short answer exam questions.  But, ultimately, about 50% of the final grade is determined by the students' performance on quizzes and exams. 

I've been waiting for a reliable and affordable online proctoring service to come on the market.  With today's announcement that Instructure (Canvas) has partnered with Verificient, an online proctoring service, it looks like we are making some progress on this front.  This was an obvious next move for Instructure, now that their LMS is being used as a platform for online course delivery of all sorts, including MOOCs.  The absence of a proctoring service was a serious gap that, it seems, is now being filled.  We'll see how effective it is--I never underestimate the creativity of students and their ability to outsmart any monitoring system.  But, at least, it is a good start.  As online courses become more common, we need a way to protect the integrity of the grade.  The system doesn't need to be perfect.  After all, academic dishonesty happens all the time in face to face classes.  But instructors need to feel reasonably confident that the grades they are awarding were, in fact, earned.  This is especially true as universities move towards a policy of not distinguishing between online and classroom-based courses on transcripts.

The quizzes, which appear at the end of each module, are not proctored.  We tried to disincentivize dishonesty by writing questions that are application of facts rather than regurgitation of facts (and therefore difficult to Google or find in a textbook).  The quizzes are timed and each question has three variations, so it is unlikely that students working together will have very many questions in common.  The quizzes are also not worth all that much of the final grade.  Ultimately, even if a student cheats on 1-2 questions/quiz, it is unlikely to make any difference in the final course grade. 

Still, I would have preferred for the quizzes to be mastery quizzes rather than graded quizzes.  When we tried that approach in the Fall 2014 version of the course, however, students didn't study for them and, ultimately, were not learning the material as well as they needed to in order to perform well on midterm exams.  By switching to graded quizzes, we were able to get the students to take them seriously.  The performance data looked almost identical to the data produced by my classroom-based students, which suggests to me that there was not much cheating going on.  Likewise, we saw significant improvements in the exam scores--another indication that the students were studying for the quizzes.  Finally, there were no obvious cases where a student had high quiz scores and low exam scores--not a sure sign of academic honesty, but a decent indication that any cheating was small-scale.

Online Rome also had three midterm exams that were administered on campus, in a proctored environment.  We included these, in part, as a way to ensure the course's credibility.  I suspect that, at some point, these midterms can be eliminated.  I suspect that the quiz data will demonstrate that it is a reliable indicator of a student's mastery of the content; and that the instructor can design other, more engaging activities that require students to make connections across the course.  A key skill in the study of Ancient Rome is the ability to see patterns and connections across time.  Studying for midterm exams is one way that students begin to see broader patterns--but there are many other ways that this could happen.

For now--and, I suspect, the next several years, midterm exams will be a part of the course.  For all sorts of reasons, it would be easier to administer these exams via Canvas.  This spring, we required that all students--even non-UT students--take the exam on campus or in an accredited testing center.  We had to resort to this after several problems with academic dishonesty in the fall semester, when we did allow distance students to take the exams online (there was an oral component to the exam, which was largely just a pain for everyone and did not work especially well).  Since the majority of the students who currently take Online Rome are Austin-based, it has been fairly easy to administer the exams on campus.  As we make a real effort to expand the enrollment to non-Austin based students, however, it would be great to have an effective proctoring service built into Canvas.

Sunday, May 17, 2015

It's about the Relationships, Always

Image source:
I can't emphasize enough how true this statement is.  One of the major reasons why the delivery of effective online education presents such a challenge to institutions is the failure to understand this basic principle.  Online courses need to be well-designed and accessible; they require a different set of logistical skills to manage than does a classroom course.  They do not especially require a lot of bells and whistles.  It turns out that students don't much care if the video is high quality so long as it is acceptable quality.  While it is convenient to have the resources of a production studio, as I did when I built Online Rome, it is certainly not a sine qua non.  There are plenty of tools available gratis on the internet that can be used to record video and/or audio and even live stream video/audio.

Without an instructor who is a. capable of creating and nurturing relationships with the students; and b. capable of creating a learning environment that provides opportunities for students to connect to and learn from one another, an online course is very unlikely to be truly successful.  It certainly will never rival the classroom experience.  Faculty who oppose online education generally rationalize their opposition by claiming that the online medium cannot replicate the intimacy of the classroom, the ability of the classroom space to nurture the kinds of relationships that support learning.  This is a dangerous failure of imagination.  In fact, a skilled and experienced online instructor knows exactly how to nurture these crucial relationships--and often can do so much more deeply and with many more students than can a classroom instructor.  That's the irony in all of this.

These days, most of the attention of institutions is on the courses. There is a tendency to get wowed by fancy videos, animations, and the like.  None of these things matter much for student learning if the course lacks an instructor who knows how to build relationships and also provide space (and incentivizes) for the students to build relationships with each other.  This latter task is especially challenging.  Discussion boards are the old fallback but, in my experience, they don't actually do much to encourage student to student interaction.

One of the areas where Online Rome could use more development is in the area of supporting peer to peer learning. Steve, the course instructor, and I had planned to do some of this in the upcoming semester.  I hope that we will have the chance to implement our plans for non-UT Austin students at some point.  For campus-based students, however, it should not be very challenging to design some activities that require or at least strongly encourage peer learning.  I know that there was a fair amount of this happening on an informal level (e.g. students worked together on the modules; they studied for exams in small groups).  We captured information about some of this through course surveys.  But I suspect that a well-designed class activity (or series of activities) that puts students into small groups for the semester would improve the student learning and general experience even more.

Until then, though, I won't stop saying: teaching is about relationships.  Period.  The medium of instruction influences how we construct those relationships, what tools are available to us, but the process of teaching and learning is always about relationships and will always be about relationships.  Further, it is incredibly shortsighted (and uninformed) for faculty to believe that face to face relationships are inherently superior to other kinds of relationships.  This prejudice for presence is at least as old as Socrates and Plato; but it has been debunked over and over again.

P.S. So, TechCrunch asks why the university is still here.  An idiotic question, but a pretty simple answer: in part, because we haven't figured out how to support social learning online.  MOOCs are the opposite of the right answer.

Saturday, May 16, 2015

Learning Outcomes and Backward Design

 A few weeks ago, my department chair sent an email to the faculty list-serv with the news that UT Austin had once again failed to pass its SACS Accreditation.  The problem, it seems, hinged on the failure of departments/colleges to define and measure learning outcomes in a way that was acceptable to SACS..  The chair expressed her view that this was a nonsensical process, anticipating the general moaning and groaning that was sure to emerge from faculty who believe that things like learning outcomes are silly.   At the start of the academic year, our CTL had sent around a very helpful sample syllabus, with the various UT policies included as well as things like a place for the instructor to list the course's learning outcomes.  Two senior colleagues ridiculed this document as useless and stupid--after all, they have been teaching for decades and don't need such guidelines.  Sadly, I'm fairly certain that if they were asked to define the learning outcomes for their course and explain how their course was going to support students in achieving those outcomes, neither colleague would be able to do so. 

Writing learning outcomes is very difficult for faculty who were never trained to think about their teaching in such terms.  We are great at describing what content our course will cover; we are pretty good at knowing that we expect our students to master a certain amount of content or skill set by the end of the semester.  We are terrible at framing our expectations for student learning in terms of learning outcomes, with all of our learning activities in the course aligned to those learning outcomes.  We are even worse at measuring learning outcomes.  We conflate grades with learning outcomes on the regular.

Every professor will tell you that students learned in their course--they will insist on it, despite never having measured how much students knew at the start or measuring their knowledge using a standardized instrument rather than an instructor-written exam.  In fact, we generally have no real way of measuring what they did or didn't learn, and how well.  This is especially true for one-off courses, like the lower division general education courses many of us teach.  So a student of mine doesn't really know anything about Roman culture or history at the end of the semester...  This is likely to have little impact on their educational career unless they decide to be a Classics major.

To my mind, one of the great boons of digitizing our teaching is precisely that it requires faculty to finally learn about things like learning outcomes and backwards design.  These are not difficult concepts but they require some intentionality--and sometimes some assistance--to implement.  The activity of building a hybrid or online class encourages faculty to think hard about how all the pieces contribute to student learning.  The existence of the course on a digital platform means that the experience of taking or teaching a course is now preserved as an artifact that can be examined by a third party.  It is no longer an ephemeral experience in which we depend on the reports of instructors and their students to evaluate teaching efficacy.

I worry about "Big Data" intruding on academic freedom, both in terms of what we teach and how we teach.  Will we all be forced to teach in 3-5 minute blocks because someone decided that this was the average attention span?  At present, online courses are closely scrutinized and monitored by our campus administrators.  To what extent will this monitoring expand as our ability conduct this monitoring with computers grows?  Currently, we are required by law to post our course syllabus.  At what point will we be required to use the Campus LMS for all graded activities, so that student data can be captured more easily?  I suspect that the refusal of most faculty to understand that they are accountable for demonstrating student learning will make it all the easier for university administrators to start tracking students and imposing ever more restrictions on what and how we teach.

I make it a point to identify and articulate course learning outcomes for all my courses, including graduate seminars.  I also include a "map" that illustrates how the different learning activities in the course will help students reach these outcomes.  It took a bit of practice to learn to think about teaching and course design in these terms but, after two years, it's become second nature.  These are the learning outcomes for Online Rome.  The least important of them is mastery of the course content.  I am much more interested in helping students develop crucial "soft skills":

·         become active, “self-regulated” learners
·         learn and become more skilled at good time management techniques
·         learn basic skills of “reading” ancient texts/art/architecture
·         develop and practice ability to think make connections between different parts of course (i.e. think analytically)
·         develop and practice ability to evaluate competing explanations or theories
·         master basic narrative of Ancient Roman cultural history from Iron Age-2nd Century CE

Enaging Undergraduate Students in Course Design and Instruction

There is a lot of conversation on college and university campuses about getting undergraduates involved in research as soon as possible.  It is clear that the close contact with faculty as well as more experienced students is a high impact experience and is positively correlated with graduation and shorter time to degree.  At UT Austin, this emphasis on undergraduate research seems to be a main plank in our new president's platform.  It has been a central topic in the Campus Conversations, which the president sponsored over this past year in his role as the Provost.  There are many different initiatives on campus, particularly in STEM fields, which already focus on this effort, including the Freshman Research Initiative in the College of Natural Sciences.  My own home, the College of Liberal Arts, briefly had a program for freshmen and sophomores, whereby students could work with a faculty member on a project connected to the faculty's research program.  I worked with a couple of students through this program but, ultimately, found it to be an exercise in futility.  The students who were eligible had no language training and my research absolutely requires knowledge of Latin as a sine qua non.  Even the task of constructing a bibliography requires more than a knowledge of English.  I struggled to devise interested but manageable tasks for the students; and I think they left the experience wondering why anyone would want to do research in Classics!

In liberal arts, and especially in the field of Classics, the major obstacle to involving undergraduates--especially early undergraduates--is that they lack the specific skills that are necessary to do even the most basic research.  Classics training at the undergraduate level focuses on language instruction, and it typically takes a student most of their undergraduate years to even master Latin and/or Greek to a level where they can begin to make sense of an ancient text.  Very often, research is postponed to graduate training, even the PhD these days.  Given this, it is a real challenge to think about how to bring inexperienced undergraduates into the faculty research process.  In truth, it can't really happen for a scholar who produces the kind of scholarship I do.

This is not to say that I cannot engage in productive conversations about research with undergraduates; or have them learning some of the "skills of the trade" in my company.  As I learned during the production of Online Rome, though, the best place for this to happen--for me, given the kind of research I do--is around teaching.  It is my sense that the value for students in engaging in research early in their undergraduate career is not research qua research so much as it is, first, structured contact with faculty as well as, ideally, more senior graduate and undergraduate students who can act as peer mentors; second, an opportunity to see the application of knowledge (I don't think it's particularly relevant whether the application occurs in a research or a teaching environment); and third, an opportunity to see "behind the curtain" of academia, to see what it is that professors do and what it means to be a professor.  This third element is especially important for students in liberal arts who think they might want to continue their study of the field as graduate students.

When I was putting together a team of students to work with me on building Online Rome last summer, I included an advanced undergraduate student (a double major in Classics and Religion).  In retrospect, I wish I'd included more.  It was fascinating to see how, in giving her the task of creating a first draft for one of the modules, we were able to have deep and engaged conversations about the content, how best to present it, what sorts of trends it connected to, etc.  Whereas I feel like any effort I have made to involve undergraduates in my research has ultimately meant that they are doing uninteresting "grunt" work, this project allowed for genuine intellectual growth.  The key was the fact that it was project-based for the student.  She worked on it and then we discussed the work.  In the future, I hope very much to involve a larger team of undergraduates in a course-building project.  In liberal arts, and especially in Classics, it is an ideal way to connect with undergraduates on intellectual grounds, but at a level that is accessible to them (because the task is not the production of original research but, instead, how to teach complicated content to their peers); and which results in the production of something tangible and useful.

I've never been much of a fan of the "hack the syllabus" approach to undergraduate teaching, primarily because it doesn't work terribly well with the particular audiences I teach.  What does work is involving a group of students in the design and build of a course from the very start.  I can imagine a very interesting learning experience in which, for one semester, a group of students works with me to design and build a course.  The following semester, we would teach that course, with the student builders acting as peer mentors to the enrolled students.  In this way, it would be possible to put more experienced students together with less experienced students--something that happens to great benefit in graduate seminars but is nearly impossible to do in undergraduate classics courses (in part because, through an odd system of incentives, nearly all students in upper division "seminar" courses are non-majors).

At present, I have only involved undergraduates in the delivery of Intro to Ancient Rome/Online Rome as graders.  It's not a bad job and it is enormously helpful to the instructional team, especially when we are teaching very large numbers of students and need to turn around the exams quickly.  The course instructor graded 10-15 short answer sections on the exams, to establish a rubric.  The instructor then met with the graders, reviewed the rubric and scoring of points (with attention to common errors and breakdown of points for each question).  Without exception, the undergraduates did an excellent job and were very attentive and punctual.  They more than earned their stipend.  The job offered an opportunity to engage with the instructor and to see a bit how large courses are run.

I am a much bigger fan of peer mentors in the classroom.  I saw them used to excellent effect in my colleague Cynthia LaBrake's Intro to Chemistry course.  I wish that I had had the funds to create a more elaborate network of peer mentors for Online Rome. This is one aspect of the Online Rome course that could be developed.  Especially in the online class, putting current students in contact with former, successful students would go a long way towards helping the new students figure out how to take an online class and, in particular, how to do well in Online Rome.  Online bulletin boards/Rate My Professor-like sites can offer some information, but it tends to reflect narrow points of view.  It would be much more effective to have a group of former, successful students who act as mentors and graders.  If some effort were made to meet weekly with these peer mentors, it would also be an excellent way to sharpen and expand the mentors' content knowledge as well as their ability to explain complicated concepts to their peers.  There is really a great opportunity here, especially for Classics and related majors, if the funding can be found and if someone is willing to spend the time developing and running such a program.

Friday, May 15, 2015

Practice Makes Perfect: Online Course Design and Digital Affordances

Even as it has become evident that online courses will play an increasingly prominent role in higher education (and academic administrators acknowledge this reality), faculty have continued to be skeptical of their quality and ability to support crucial skills like critical thinking (see also this article).   I have always found this skepticism odd and more a reflection of the fact that most faculty have no direct knowledge of or experience with online education in its current forms.  It is especially challenging to imagine how an online course could do the same kind of work that, at least in the minds of faculty, our small seminar-style classes do.  In addition, we overlook the fact that, in truth, most of us have little sense of what goes on in the classrooms around our campuses.

When it comes to the courses of our own departmental and institutional colleagues, however, we know very little about what or how well students are actually learning.  Our default assumption is that any classroom based course is a good course with abundant, high quality student learning; but we should probably assume that such courses are the exception rather than the rule, especially on R1 public university campuses where the majority of courses are currently taught by professors in training (aka graduate students) and contingent faculty who are badly paid, carry heavy teaching loads, and lack any kind of job security.

This prejudice against online education is one that I find interesting but also troubling.  On the one hand, faculty who know nothing about it assume that it is, by nature, inferior to a classroom-based course (much like Plato and the ancients perpetuated the view that writing inferior to speech).  On the other hand, these same faculty view online courses as effectively self-teaching.  My own sense is that, lacking any first hand knowledge of the wide variability in online course designs, most faculty assume that all online courses look like the standard MOOC: a set of talking-head lectures by a content expert followed by some machine-graded quizzes and/or exams.

In reality, online courses are highly variable, far more so than classroom courses.  I suspect that, as more faculty get down in the trenches of online course design, we will see even more variability in course design.  The online platform allows for far more innovation, variation, and creativity than does the current college/university classroom, even so-called Smart Classrooms.  The trick is for the faculty designer to recognize that the online space is a wholly different kind of space with different affordances--and to leverage those affordances.  One of the most disappointing things about MOOCs is that, in most cases, they are incredibly conservative in format.  They attempt to recreate the classroom experience, but imperfectly.  Few MOOC designers approach the task of designing the learning experience as an opportunity to invent new models of teaching and learning that maximize the strengths of the digital while minimizing its weaknesses.

Instead of lamenting the absence of face to face interaction in online classes, and the challenges that this presents, we need to look at the online medium for what it CAN offer that face to face cannot--and then maximize those affordances.  This is tough work.  It requires a lot of creativity and a willingness to, in some sense, re-learn basic skills.  It requires the course designer to understand that you can have the same basic learning outcomes for a classroom course and an online course; yet the pathways to achieving those learning outcomes are likely to look very different.  This can be intimidating to successful and experienced classroom instructors.  Yet, if one tries to build an online class that is a poor imitation of a classroom course, it is not likely to succeed.

Online Rome exploited the affordances of the digital learning environment in several crucial ways.  Two of these: the ability of digital learning activities to provide immediate feedback and opportunities for repetitive practice and self-correction; and the ability to personalize student engagement with primary source material.  I'll talk about the ways that the course highlighted student engagement with primary source material in another post.  This post focuses on the ways we exploited the digital learning environment and mastery learning in the course design.

A driving theoretical principle in the design of the Online Rome course is, essentially, that practice makes perfect.  In my previous life as a serious athlete (I played fastpitch softball at a pretty high level, as a pitcher), I learned at an early age that the key to success under pressure was practice.  A lot of practice.  As a pitcher, I practiced every day.  I had a coach who critiqued the tiniest things and made me re-do pitches over and over until I got every part right.  I watched video of myself.  I got better because I worked very hard at it--even though I am 5'3 and have short "levers."  I wasn't born with a pitcher's body, but I was smart and I worked incredibly hard to maximize the talent I had.

Online Rome follows the same basic principle that hard work can make up for a lack of natural talent--and, in fact, is far more important than natural talent.  My job, as course designer, was to create activities that focus the work and provide immediate feedback so that students can recognize their misconceptions and correct them before they take hold.  This process is much easier done in the digital environment than in a classroom, especially when dealing with larger class sizes.  In a classroom, it is very difficult to know what every single student is thinking at a given moment (though student response systems like i>clickers are of great help).  It can also be difficult to clarify misconceptions, because the nature of those misconceptions will vary from student to student.  In the long run, it is much better pedagogy to train students to recognize and correct their own misconceptions using instructor-provided feedback.  In the digital environment, that feedback can be instantaneous thanks to machine grading.

2/3 of the grade on the 10 course modules is entirely about effort.  Students earn full credit if they score 90% or higher on the in-module questions, but they can repeat the module as many times as they need to.  The theory is that, by incentivizing practice, we are actually incentivizing the type of behavior that leads to learning.  Similarly, at the end of each module we included a large number of practice quiz questions (c. 35-50).  The point of these practice questions was for students to be able to check their mastery, figure out where they needed remediation, and fill in those gaps BEFORE taking a graded quiz.  The graded quiz provided motivation to do the practice quiz but, in fact, the far more important and influential learning activity was the practice quiz.

The performances on the graded quizzes at the end of each module; the essays; and the midterm exams suggests that we were right about this.  The students practice learning the content until they master it.  They are happy because this produces high grades and I am happy because it produces high quality learning, especially in the essays (where we ask them to do analysis and application).  I use various forms of digital learning activities and automated feedback in my blended classroom-based Intro to Rome class (i<clicker questions, practice quizzes, in class quizzes).  Because I teach large numbers of students, I have to teach students to use the diagnostic information that they get from these activities to self-correct.  This process is no different online than in the classroom--except that I can do it far more thoroughly in the digital learning environment of an online course.