Core Assessment Committee Summary Report for AY 2008‐2009 Membership and Responsibilities The Core Assessment Committee (CAC) convened in September 2008 with representatives from each of the College’s divisions. Members of the CAC include Marie Cantwell (Health Sciences), Catherine Carsley (co‐chair, Humanities and Arts), Deb Greenspan (co‐chair, Social Sciences), Walter Hunter (Math, Science and Advanced Technologies), Fred Kozlowski (Business and Computer Science), and Mary Beth Parkinson (Library). The CAC’s primary mission is to be “responsible for the design and implementation of the Core Assessment Plan.” The specific charges of the CAC, as agreed upon by the CAC and the Associate Vice President of Academic Affairs, Dr. Joan Brookshire, follow below: 1. Develop an overall process for assessing the new core curriculum, including, but not limited to, identifying an appropriate methodology for the assessment of core courses, describing specific assessment activities with time lines and person(s) responsible for each activity, and formulating analysis and reporting mechanisms. 2. Convene faculty groups that will, where appropriate, develop appropriate scoring rubrics. 3. Work with Institutional Research to ensure that all assessment tools, especially scoring rubrics, are appropriate to the identified assessment task. 4. Develop an annual assessment calendar. 5. Oversee the collection, analysis, and reporting of Core Assessment results. 6. Make recommendations based on assessment results. 7. Serve as individual liaisons to disciplines and programs working on Core Assessment. 8. Work with Program Assessment Committee, when needed, to ensure that program assessment plans have appropriately addressed assessment of core competencies embedded in program courses. Overview of Activities The CAC directed much of its effort toward preparing the College to effectively assess its new core curriculum. Through self‐training, research, and extensive discussion, the CAC worked to develop and refine an assessment methodology that would be minimally intrusive to faculty and students while yielding meaningful data that engender improvements in the teaching and learning processes. The details of the assessment process developed by the CAC are provided in the subsection below titled “Assessment Process.” ∗ ∗ Note that during the first full half of AY 2008‐2009, no courses had had yet been approved as Core.
2 CAC members trained themselves in assessment methods, both formally through webinars and conferences, and informally through examination of assessment methodologies employed at similar institutions. This knowledge‐base enabled CAC members to serve as liaisons to faculty within their respective divisions. In addition to providing face‐to‐face support and advice, the CAC also developed and compiled written resources for faculty members involved with Core assessment. The CAC developed a Core Assessment Instructions packet that details a step‐by‐step process for preparing for future assessment of the core curriculum. The Instructions packet is a critical aid for faculty because it contains clear and straightforward guidelines that align with the CAC’s overall vision for the assessment of the core. In addition, the CAC compiled a “toolbox” of additional resources that faculty could consult for more detailed information on specific aspects of assessment (e.g., rubric development, sampling methods). In addition, the CAC explored software packages designed to support and track ongoing core assessment. Because monitoring assessment at the institutional level is highly complex, the CAC believed that software was an essential component of successful assessment. Software packages help display and make more readily accessible – for faculty, administration, and accreditation bodies – assessment activities across the college. The CAC reviewed well‐known software vendors and ultimately endorsed WEAVEonline. A pricing proposal was submitted to Joan Brookshire in February, 2009. Finally, each ad hoc committee responsible for developing the common rubric demanded by the Core Approval Committee has been contacted by the CAC. The CAC plans to use these common rubrics as part of the assessment process. The collection and development of these rubrics is still ongoing. Progress on each of the responsibilities of the CAC is summarized in the Table entitled, CAC Progress towards Goals. Proposed Assessment Process As the result of discussion, debate, and research over the course of the 2008‐2009 academic year, the CAC developed the following process for the assessment of the core curriculum. When a course is approved for Core, the names of the individuals responsible for that course are sent to the CAC. Key points in the process are listed below. 1. The CAC disseminates to the individuals sponsoring each approved core course a Core Assessment Instructions packet (see Attachment 1). In the Core Assessment Instructions packet, faculty are led through the following steps in the assessment process to: o
Identify where the course fits in the college‐wide assessment calendar.
Identify the assessment instruments that the course will use to determine how students achieve the Core goal. Sponsors should provide a copy and description of each assessment instrument (by completing the Assessment Instrument Description sheet).
2. Multiple assessment instruments may be used to assess each Core goal. However, faculty members are encouraged to generate and select from a common list of assessment instruments in order to facilitate data comparison. The CAC has put together a brief listing of these instruments and included them in the Instructions packet.
3 3. Faculty members understand that not all sections of every course will be involved in data collection. Leon Hill from the Office of Institutional Research will collaborate with faculty in generating an appropriate sample and ensuring that the sample includes relevant sub‐groups. 4. At this point in the core process, individuals and strategies must be identified to handle the logistics of data collection. 5. Data derived from each instrument must be scored using the common rubric that was created for the core goal with which the instrument is associated. (e.g., data from assessment instruments used to measure information literacy must be scored using the common Information Literacy rubric). 6. After data are collected, relevant faculty should convene to discuss the results and modifications that might be needed to improve the teaching and learning processes and to increase student achievement of the core goal. Outstanding Issues While the CAC arrived at a consensus regarding the steps listed above, four issues remain either contended or unresolved. The issues that still remain are: •
Whether or not core goals should be assessed at the course‐level, regularly and during an ongoing course or program assessment process, or at the goal‐level, intermittently, as part of an institutional portfolio methodology.
Whether or not the faculty teams that score the assessment artifacts should be interdisciplinary or discipline‐specific.
Whether the Core Assessment Calendar, proposed as the assessment of one skill goal, one knowledge goal, and one value goal per year, is really useful to the faculty when conducted on such an extended basis. As now proposed, it will take more than four years to assess all the goals in the core.
4 Goal‐level vs. Course‐level Assessment The CAC discussed, at much length, whether core goals should be assessed through the implementation of an institutional portfolio model or through “reporting up” on goals via ongoing course‐level assessment. The issues with each model are spelled out as follows: •
Institutional Portfolio assessment model. In this model, several core goals are chosen for assessment each year. Representative samples of courses that meet each goal are identified for assessment. Artifacts for each course identified are collected, scored, and reported on at the College level. In this model, a core course that meets four core goals might be assessed for a different goal in each of four consecutive years. This model is less intrusive to faculty members but also provides less data back to individual instructors on an annual basis.
Course‐level assessment model. In this model, core goals are assessed on an ongoing basis, as part of a discipline‐driven assessment process. When other learning outcomes are assessed for the course, all core goals for the course are also assessed using common scoring rubrics. This method requires more up‐front work for faculty members, yet also yields richer and immediately usable assessment data.
Some members of the CAC held that course‐level assessment would be more useful to the faculty and would be fairly easy to report as part of an ongoing assessment process. Others thought that the institutional portfolio model would be less intrusive and easier to complete. In a our final meeting of the year, the committee held a non‐binding vote on the issue and endorsed the course‐level model by a vote of 4 to 1 with 1 abstention. Assessment by Interdisciplinary Committee vs. by Discipline‐Specific Faculty The CAC debated the issue of who will complete the actual scoring needed for core assessment. Again, the discussion centered around the Institutional Portfolio model for assessment, or some other adaptation that might serve our College needs more completely. Some versions of the Institutional Portfolio assessment model, for example, require interdisciplinary ad hoc teams to score data, college‐wide. These teams evaluate various artifacts pertaining to a core goal using a common rubric. However, this model assumes that a faculty member from, for example, Math, would be part of team who scores an artifact, for example, from a Music course. The CAC debated the advantages of this model. Other institutions (for example, The Community College of Baltimore County) have adopted a version of the institutional portfolio model that allows discipline specific‐teams to score assessment data using the common rubric established for each goal. For example, Speech faculty use the rubric established for Goal 1 to evaluate how well Speech students did on Goal 1. At the same time, English faculty will use the same rubric established for Goal 1 to evaluate how well English students did on Goal 1. The CAC emphasizes a clear benefit of the disciplinary approach: there will likely be greater faculty buy‐in on the assessment process as a whole because faculty from outside disciplines will not be evaluating discipline‐ specific artifacts. As well, if data are reviewed only at the institutional level, faculty will have great difficulty seeing its applicability to their work. Assessment is, in the end, designed to improve teaching and learning. On the other hand, the interdisciplinary assessment model has the advantage of less work for the faculty.
5 There was extensive discussion about this issue within the CAC. At the final meeting (at which all members were present), the large majority of the group supported the discipline specific approach, but the endorsement was not unanimous. Core Assessment Calendar The CAC proposed a four‐year model for core goal assessment. After an initial pilot year, each year the College would assess one skills goal, one knowledge goal, and one values goal; four years would be needed to assess the Core as a whole. A proposed calendar appears below (UPDATED FALL 2009): Category
1. 2. 3. 4. 5. 6. 7.
8. Physical and Life Sciences 9. Behavioral and Social Sciences 10. Exercise and Health Sciences 11. Civic Responsibility 12. Global Perspective and Cultural and Social Diversity
Ethical Reasoning and Behavior
The CAC had reservations about the effectiveness of such a long time‐line for assessment, and debated whether a shorter calendar would be more helpful to faculty members seeking to revise courses based on assessment feedback. Final Remarks The CAC accomplished its main goal, the proposal of a preliminary process for the assessment of the core courses; however, that process cannot be finalized until agreement on three critical issues has been reached: goal‐level vs. course‐level assessment, scoring by interdisciplinary committee or discipline‐specific committees,
6 and the necessity for an extended calendar for the assessment of core goals. More work and discussion is needed during AY 2009‐2010 to resolve these issues. Attachments
Faculty Assessment Instructions packet and forms
CAC Progress towards Goals
DRAFT WORKING DOCUMENT
April 14, 2009
Core Course Assessment Checklist Course Name and Number:
Contact Person: This course is approved for the following goals (check all that apply):
1. Communication Skills 2. Analytic Skills 3. Quantitative skills 4. Computer fluency 5. Information Literacy (required) 6. Intellectual Heritage 7. Aesthetic Sensibility and the Arts 8. Physical and Life Sciences 9. Behavioral and Social Sciences 10. Exercise and Health Sciences 11. Civic Responsibility 12. Global Perspective and Cultural and Social Diversity 13. Ethical Reasoning and Behavior
DRAFT WORKING DOCUMENT
April 14, 2009
Preparing for Future Assessment of Core Courses 1. Be ready for the Core Assessment Committee (CAC) to contact you once the course you are sponsoring has been approved as a Core course. The Committee will schedule an initial meeting to get the assessment planning process started. 2. Understand where your course fits into the Collegewide assessment calendar. The College plans to assess the 13 Core Goals, as follows (See Table 1). Category
14. Communication Skills 15. 16. 17. 18. 19. 20.
21. Physical and Life Sciences 22. Behavioral and Social Sciences 23. Exercise and Health Sciences 24. Civic Responsibility 25. Global Perspective and Cultural and Social Diversity
Ethical Reasoning and Behavior
Table 1: Calendar for Collegewide Assessment of Core Goals
3. Identify an assessment instrument for each outcome your course meets. Complete a Table of Assessment Activities for each goal your course meets. The table asks you identify which assessment instruments you have chosen to gather data about that outcome. Below are some examples of assessment instruments and the data artifacts that the instruments would yield (Table 2). Each outcome must be supported by at least one instrument that provides direct evidence of student learning. Grades and course evaluations provide only indirect, not direct, evidence. Examples of Direct Evidence of Student Learning Comprehensive exam questions and scores, accompanied by “blueprints” of what the test assesses Research paper scored using a rubric Presentation or project scored using a rubric Field supervisor evaluations and ratings Score gains between entry and exit on tests or writing samples Observations of student behavior (presentations, group work) with notes recorded systematically Portfolio of student work Summaries or analysis of or electronic discussion threads scored using a rubric Feedback from computer simulated tasks Student reflections on their values, attitudes and beliefs, if those are intended outcomes. Certification or licensure exams Table 2: Examples of Direct Evidence of Student Learning Adapted from a list provided by Dr. Linda Suskie, Middle States Commission on Higher Learning, “Examples of Evidence of Student Learning,” Montgomery County Community College, Assessment Conference, April 3, 2009
Be sure to anticipate the way in which your exam, papers or projects will “map” onto the assessment rubrics developed for each goal. CAC makes sure that you have copies of those rubrics for future reference. 4. Provide a working copy and a description of each assessment instrument to the CAC. Once you have decided which assessment instruments to use, complete a Description of Assessment Instrument for each. Attach a working copy (sample) of each instrument to each form. Note: You must complete an Assessment Instrument Description for each instrument you propose. Instruments may vary per goal or per outcome. It may be possible, for example, to propose one instrument to satisfy three outcomes of Goal 5 and a separate instrument to satisfy the fourth outcome of Goal 5. All data artifacts (data that result from the instruments selected) must be able to be scored using the common rubric for the Goal it addresses.
Table of Assessment Activities (include one table per goal) Outcomes(Sample for Goal 5)
5-1. Determine the nature and extent of information required for research. 5-2. Formulate a search strategy to access the appropriate information effectively and efficiently. 5-3. Locate and critically evaluate information from written, oral, graphic or symbolic and mass media communications. 5-4. Access and use information ethically and legally, employing the appropriate format and documentation to acknowledge sources.
Student Learning Activities
Can data artifact be scored against common rubric for Goal 5?
Description of Assessment Instrument Course Number: Name of Assessment Instrument:
(essay, test, portfolio, survey, evaluation, reflection)
Used for Which Goal(s) and Outcome(s)? Narrative Description of the Instrument: (attach a sample)