Three Strategies for GAI-Proofing Your Course

By Dr. Vlad Krotov

Generative AI has hit business educators like a freight train. In just a few months after its launch in 2022, ChatGPT had acquired 100 million users; 200 million users are predicted by 2023. Following the suit, Google has released its own conversational chatbot, Google Bard this year. Google Bard is powered by the same technology as Google’s search engine, so, unlike ChatGPT, it seems to be more aware of recent news and developments.

Shortly after the release of ChaGPT by OpenAI, several professors from top business schools announced that ChatGPT was able to pass their exams. While some business school professors still act as if ChatGPT doesn’t exist, a growing number of educators believe that Generative AI is a disruptive technology that will quickly and permanently alter the century-old rules and pedagogical approaches in business education. If this is true, then educational institutions must introduce changes and create policies to ensure that students use this new, disruptive technology in a way that does not impede their learning.

Nowadays, most students know what ChatGPT is and how to use it for completing homework assignments. How to mitigate the academic integrity issues associated with the use of GAI by students seems to be of the utmost importance to business schools, since academic integrity is important for quality of business education and is a formal requirement of all major international accreditation bodies, such as AACSB. In this article, I outline three simple strategies that business educators can use to mitigate academic integrity issues caused by GAI use. I also discuss each of these strategies’ pros and cons.

Punitive Strategy

Despite the rapid advancements in GAI, Many educators choose to teach their courses “as is”. Some specify in their syllabi that the use of Generative AI for completing assignments is prohibited and punishable under the school’s Academic Integrity Policy. AI-detection tools, such as ZeroGPT, are used to monitor student submissions for AI-generated content.

Pros

- Trusting students to make ethical choices and punishing those who do not is an old, simple, and, perhaps, wise approach for ensuring academic integrity. When students want to cheat, they will find a way to do so – by using GAI, hiring someone to do their projects, or in some other way. It is important for educational institutions to have an admissions process that screens out students who are likely to cheat in the first place. Instructors should be able to trust most students and not act as investigators and prosecutors at all times. If a cheating student is caught, the punishment should be severe enough to deter others from even considering unethical behavior.
- A minimal amount of effort on the part of faculty and the business school is required with this approach
- A business school may want to take this approach in the short term if they want to “wait and see” what happens with Generative AI in business education before making any important decisions or investments.

Cons

- AI detection tools are often ineffective at detecting AI-generated text, even though this issue is not as serious as some educators believe. False-positives are also quite common. It is possible for students to revise AI-generated text to make it unlikely that it will be flagged by such software as ZeroGPT. Furthermore, proofreading tools such as Grammarly and WordTune can produce false positives as well.
- GAI is here to stay, most likely. It may not be wise to prohibit students from using GAI tools, since these tools may soon become essential in the business world.
- In the near future, business schools will probably discover new ways to improve student learning by implementing GAI. Those business schools that do not adopt GAI for teaching and learning may soon lag behind those that do.

Flipped Classroom Strategy

It is possible to “flip” a class so that most learning and important assessments take place in a physical classroom, in front of the instructor, and with very little use of computers. For example, all exams can be administered face-to-face. Important learning exercises can be done in class as well. Major projects, while carried out outside of the class, should be presented and defended in class as well.

With this approach, the instructor can offer students help and make sure they are the ones doing the assignments. An instructor can ask individual students or student groups to demonstrate and explain progress on their project work in class every week. The grading weight devoted to attendance and participation can be increased as well, encouraging students to attend face-to-face classes. Students are free to use GAI tools outside of the classroom in any way they see fit (e.g. to prepare for a particular class session), but they should be able to demonstrate their competence face-to-face.

Pros

- Performing teaching and assessment face-to-face can be quite effective for attaining course learning objectives.
- It’s much easier for an instructor to detect cheating when most of the work is performed in front of him or her.

Cons

- While it’s possible to ask online students to take major exams or defend major term projects on-campus, this strategy is obviously not well-suited for asynchronous online courses.
- May not be appropriate for large classes, since this approach requires individual attention to every team and, sometimes, every student.

Integration Strategy

Instead of excluding GAI from the classroom, an instructor may choose to embrace the technology in a way that actually assists teaching and learning. This is probably one of the most effective yet difficult approaches. GAI is still a new technology. Many educators lack a solid understanding of how to use this technology ethically and productively.

What’s clear though, is that this approach may require a radical redesign of each course’s pedagogy. GAI can be used by students to answer basic questions at the “understanding” level of Bloom’s Taxonomy, for example. In fact, students may be instructed to ask ChatGPT or Google Bard questions in relation to the subject matter of the course and then to read and evaluate the responses as a part of an assignment. Thus, the point of the assignment is to interact with GAI and not to provide “correct answers”.

When it comes to higher-order cognitive skills, an instructor may ask very complex and context-specific questions in a format not supported by major GAI tools in order to decrease the likelihood that GAI will be used to complete these assignments.

Ideally, these assignments should be in the form where GAI still lacks capability. For example, instead of just asking to analyze a case study, the instructor can ask students to create flowcharts or UML Activity Diagrams based on the case. Alternatively, students can be asked to record videos with their analyses and post them to YouTube. Thus, even if students ask GAI for assistance, the final product will largely be their own work.

Also, while GAI tools such as ChatGPT and Google Bard are becoming increasingly knowledgeable in many topics and increasingly capable of performing very complex cognitive tasks, they often lack knowledge and understanding of very narrow and specific contexts. For example, an instructor can ask complex questions in relation to specific local individuals or organizations that may be known only to students and the instructor. For example, let’s say there’s a small, local software company where the university is located. The instructor can talk about this company in class and then asks students to come up with strategies that are suitable for this company given its unique local context

Regardless of which approach is chosen for assessing higher-order cognitive skills, the instructor can quickly run his or her questions and assignments through ChatGPT or Google Bard to see what kind of responses students are likely to get when they ask GAI for help. If GAI can answer these questions with ease, then additional context or complexity needs to be added to the question. ALternatively, the format of the assignment can be changed (e.g. from an open-ended response to a visual diagram).

Students can rephrase these specific questions as generic ones and submit them to ChatGPT. But generating a quality response will still require analyzing and accommodating the local context (and this is where most of the learning will occur) or putting the responses in a format that still entails some learning. The instructor can deduct points if the local context is not properly accounted for or the format instructions are not followed. A grading rubric can be created that evaluates students’ work based on the extent to which a solution is contextualized for a specific company or individual and the extent to which all the directions were followed. If necessary, an oral defense can be scheduled so that students can demonstrate their mastery of the material.

Pros

- Since GAI is likely to become a permanent fixture in business, this approach will make students more prepared for the era of GAI
- With this approach, the instructor can automate some of the basic tutoring tasks and focus on developing higher order cognitive skills among students via complex, contextualized, and innovative assessments.

Cons

- This approach can be time consuming, since it requires quite a bit of thinking and curriculum revision
- Many educators may lack time, expertise, or motivation for integrating GIA into their curriculum in a way that is ethical and conducive to student learning.

In conclusion, it can be said that Generative AI is very likely to become a permanent fixture in business education. Individual educators and business schools will have no choice but to adapt to this new, disruptive technology and find ways to accommodate in an ethical and productive fashion. The list of strategies for accommodating GAI in the classroom provided here is not perfect or exhaustive. What’s important though is that every educator and business school should have a strategy in relation to GAI, or they will quickly find themselves in a disadvantaged situation. Having a strategy in relation to GAI is better than having no strategy at all.

August 6, 2023August 6, 2023

What is accreditation consulting?

By Dr. Vlad Krotov

Accreditation consulting is a service provided to educational institutions, particularly colleges, universities, and schools. Accreditation firms guide educational institutions through the process of obtaining accreditation from recognized accrediting bodies or agencies. Typically, accreditation consulting firms or consultants offer expertise, guidance, and support to institutions seeking accreditation in the following areas:

- Assessment and Readiness Evaluation: Consultants evaluate the institution’s current practices, policies, and procedures to determine its readiness for the accreditation process. They identify areas of improvement and develop action plans to meet accreditation standards.
- Compliance Review: Accreditation consultants help institutions ensure compliance with the accreditation requirements and standards set by accrediting bodies.
- Documentation and Report Preparation: Consultants assist in compiling the necessary documentation and preparing reports required for the accreditation application.
- Strategic Planning: Consultants help institutions develop long-term strategic plans to align their objectives with the accreditation requirements and enhance their educational programs and services.
- Business Process Reengineering. Accreditation consultants help institutions create new processes or revamp existing ones in compliance with accreditation requirements. These processes may include tenure, promotion, assurance of learning, faculty mentoring, etc.
- Training and Workshops: Accreditation consultants may conduct training sessions and workshops for faculty and staff to familiarize them with accreditation standards and best practices.
- Mock Visits: Some consultants organize mock accreditation visits to simulate the actual review process and identify areas that need improvement.
- Continuous Improvement: Accreditation consultants work with institutions to develop a culture of continuous improvement, encouraging ongoing assessment and enhancement of educational quality.
- Data Gathering and Analysis: Accreditation consultants help institutions gather and analyze data about students, faculty, and employers. An educational institution often performs this to demonstrate compliance with specific standards and requirements.
- Communication and Liaison: Consultants may act as liaisons between the institution and the accrediting body, ensuring effective communication and addressing any queries or concerns.

Although accreditation consultants can be quite useful, institutions should not rely solely on them to get them through accreditation processes. Colleges and universities should actively engage in internal self-assessment and improvement efforts to ensure long-term compliance with accreditation standards. Accreditation is not a snapshot of a particular moment in time when a college or school excelled in meeting accreditation standards, but rather a continuous self-improvement journey. Additionally, when seeking accreditation consulting services, institutions should choose reputable firms or individuals with expertise in their specific accreditation requirements and a track record of successful outcomes.

August 2, 2023August 2, 2023

What is Triple Crown Accreditation?

By Dr. Vlad Krotov

Triple crown accreditation: AACSB, EQUIS, AMBA

Triple crown accreditation refers to a prestigious recognition awarded to business schools that have achieved accreditation from three prominent international accreditation bodies for business education. These three major accrediting organizations are:

- Association to Advance Collegiate Schools of Business (AACSB): AACSB accreditation is widely regarded as the most rigorous and prestigious accreditation for business schools. It focuses on evaluating the quality of a business school’s faculty, curriculum, teaching methods, and research output.
- European Quality Improvement System (EQUIS): EQUIS is a European-based accreditation body that assesses the overall quality and internationalization of business schools. It evaluates aspects such as governance, programs, student body, research, and engagement with the corporate world.
- Association of MBAs (AMBA): AMBA is a global accreditation body specifically focused on MBA programs. It assesses the curriculum, faculty, student diversity, and career services of MBA programs offered by business schools.

Achieving triple crown accreditation signifies that the school has met stringent international standards of excellence in business education and is recognized for delivering high-quality programs with global relevance. Triple crown accreditation is a mark of distinction and can enhance a business school’s reputation and attractiveness to prospective students and employers. It is estimated that only one percent (approximately 120) of business schools have achieved triple crown accreditation. Even so, triple crown accreditation does not imply that the school is among the top one percent of business schools worldwide.

Achieving triple crown accreditation is a challenging and lengthy process that requires a strong commitment to continuous improvement and academic excellence. As a result, only a select number of business schools around the world have earned this distinguished status. Furthermore, business schools in Europe and Asia are more likely to pursue triple crown accreditation than those in the United States.

October 19, 2022October 19, 2022

Business Accreditation and Curriculum Alignment

By Dr. Vlad Krotov & Dr. Pitzel Krotova

AACSB and ACBSP Curriculum Standards

Obtaining an international accreditation for a business school usually requires extensive revisions of existing curriculum in order to meet the requirements of curriculum-specific accreditation standards. For example, Standard 4 of the Association to Advance Collegiate Schools of Business (AACSB) requires that “the school delivers content that is current, relevant, forward-looking, globally oriented, aligned with program competency goals, and consistent with its mission, strategies, and expected outcomes” (AACSB International, 2022). Similarly, Standard 6 of the Accreditation Council for Business Schools and Programs (ACBSP) requires that “the curriculum must be comprised of appropriate business and professional content to prepare graduates for success” and that the business school “must have a systematic process to ensure continuous improvement of curriculum and program delivery” (ACBSP, 2022). In this article, we talk about the most important elements of a business curriculum and how these elements can be aligned in order to meet the accreditation requirements and build an effective, self-sustaining quality assurance system in relation to business curriculum.

Curriculum Elements

In short, curriculum describes what is taught at a business school and how it is taught (Squires, 2012). A curriculum is usually formalized using a document or a plan that spells out the following:

1. Program learning outcomes (PLOs) that graduates must master
2. Course learning outcomes (CLOs) or goals that outline smaller and specific learning objectives to be achieved within each course comprising the program
3. Alignment of program learning outcomes (PLOs) and course learning outcomes (CLOs); this alignment is usually provided with the help of a course alignment matrix (CAM) that shows how individual courses and their CLOs support PLOs
4. Appropriate assessment tools that can be used to measure CLOs and/or PLOs
5. The content or material to be taught within each course comprising the program in the form of course syllabi

There are many other elements that comprise a curriculum (see Table 1). All these elements must be properly aligned to ensure effective development of the desired competencies among students.

Curriculum Element	Description
College Mission	Defines the aim of a college, its main reason for existence
Market Conditions	Economic marketplaces often dictate which professions or competencies are in demand in the workplace
Compliance Standards	Accreditation and governing bodies often mandate competencies that a particular program needs to develop
Program Learning Outcomes (PLOs)	High-level goals (or competencies) that students are expected to attain as a result of completing a particular program of study
Course Learning Outcomes (CLOs)	Specific course-level objectives (or competencies) that students are expected to attain as a result of completing a specific course
Course Materials	Training materials used as part of a course: textbooks, books, journals and journal articles, electronic and multimedia materials, etc.
Pedagogy	Various theories, methods, or tools employed to develop competencies among students
Technology	Information and Communication Technologies (ICTs) used to deliver course content
Physical Resources	Physical facilities (e.g., classrooms, labs, specialized equipment, etc.) allocated to a course or program
Credit Hours	Amount of face-to-face or online interaction between a student and an instructor devoted to a particular course or program
Assurance of Learning (AoL)	How attainment of particular learning outcomes (or competencies) is assessed and reported at the course and program level

Table 1. Curriculum Elements (Camba & Krotov, 2015)

Curriculum Alignment

Curriculum alignment can be viewed as a triangle with the following three cornerstones: curriculum, teacher, and test (see Figure 1).

The model shows the need for the three elements to be connected or aligned. Educational goals that are targeted by the curriculum become the basis of defining the work to be done by teachers. Formal testing (or assessment) is used to evaluate the degree to which teachers further deliver the educational goals set forth by the curriculum. Thus, a well-aligned curriculum can also be viewed as a self-sufficient quality control system.

Managing Curriculum Alignment

Lewin’s process-based change management model (see Figure 2) can be used as a guiding framework for an effective curriculum alignment initiative.

Figure 2. Lewin’s Change Management Model (Kaminski, 2011)

The first stage of the curriculum alignment process is the so-called “unfreeze” stage. This stage aims to prepare for the desired changes in the curriculum by having clear and open communication with all the relevant stakeholders in relation to the desired changes in the curriculum. In this stage, people involved in delivering and managing the curriculum analyze the current curriculum and identify the changes that are necessary in order to meet the accreditation standards or achieve the desired improvements in relation to the curriculum. All the stakeholders participating in the “unfreeze” stage need to be convinced that new materials, structures, and processes must be adopted in order to achieve desired improvements. In the second stage called “change,” the stakeholders implement the intended changes to the curriculum. This phase is time-consuming, confusing, and costly. The third stage of the curriculum alignment process is the “refreeze” stage. During this stage, changes to the curriculum are stabilized. The main concern in this phase is to ensure that change becomes a permanent part of the normal process and the system does not revert to the old ways and habits.

References

AACSB International (2022). 2020 Guiding Principles and Standards for Business Accreditation. Retrieved from https://www.aacsb.edu/educators/accreditation/business-accreditation/aacsb-business-accreditation-standards

ACBSP (2022). Accreditation Standards. Retrieved from https://acbsp.org/page/accreditation-standards

Camba, P., & Krotov, V. (2015). Critical success factors in the curriculum alignment process: The case of the college of business at Abu Dhabi University. Journal of Education for Business, 90(8), 451-457.

English, F. W. (2000). Deciding What to Teach and Test: Developing, Aligning, and Auditing the Curriculum. California: Corwin Press, Inc.

Glatthorn, A. A. (1999). Curriculum alignment revisited. Journal of Curriculum and Supervision, 15(1), 26.

Kaminski, J. (2011). Theory applied to informatics-Lewin’s change theory. Canadian Journal of Nursing Informatics, 6(1).

Squires, D. (2012). Curriculum alignment research suggests that alignment can improve student achievement. Clearing House, 85(4), 129-135.

AACSB, ACBSP, Accreditation, Change Management, Curriculum, Curriculum Alignment

August 19, 2022August 19, 2022

Accreditation Reaffirmation via Skeletons of the Self-Study Teams

By Dr. James E. Mackin

Whether we’re talking about regional or professional accreditation, there is always work to be done to ensure that the institution is progressing toward the next accreditation visit (the so-called “accreditation reaffirmation”). It is all too common that once an accreditation visit has taken place and all of the recommendations emanating from the visit have been addressed, an institution will essentially relax until the next visit is imminent. Then, the scramble begins again to ensure that the institution is in compliance with the accreditation requirements. Inevitably, the institution will at the very least receive recommendations from the next accreditation visit because accreditation standards have not been addressed in between accreditation visits.

The point is that institutions should never relax when it comes to accreditation requirements. It is useful to think about accreditation as a process by which institutions can ensure that they are always doing the right things. For example, it is a good thing for students that institutions constantly assess learning outcomes, and that is why accrediting bodies require continuous assessment of learning outcomes. Therefore, even without the looming threat of accreditation, institutions should continuously address accreditation standards on an ongoing basis.

One strategy that you can use to ensure that you are always paying attention to your accreditation standards is to maintain skeletons of the self-study teams even during periods when an accreditation visit is not imminent. The skeleton self-study teams would continuously monitor the institution’s compliance with accreditation standards, and if any “relaxing” does occur, the teams can bring concomitant issues to your attention for remediation. The point is that if the institution operates as if an accreditation visit is always on the horizon, then the issues that come up at an actual accreditation visit will be relatively minor and will be straightforward to deal with.

July 10, 2022July 10, 2022

The Main Pedagogical Goals of Case-Based Teaching

By Dr. Vlad Krotov

Being put simply, a teaching case is a real or fictional story about an organization, its employees, and the issues they are facing. The value of the case study method comes from two broad areas. First, using case studies in class allows students to learn more about real companies – their successes and challenges. This has significant practical value: students can apply what they learn about real companies and managers in their workplace. From a pedagogical standpoint, case studies provide illustrations of class concepts within a real-life context. Second, using case studies allows an instructor to facilitate the development of higher-order cognitive skills among students. These higher-order skills are developed by asking students to apply class material to real-world situations. The material can be applied by finding an instance of a class concept in the case to clarify a situation that the company is facing. Alternatively, management frameworks can be used for analyzing or evaluating the current state of the company, and creating recommendations based on the more prescriptive theories that students learn in class. Each of these valuable dimensions of the case study method is expanded upon in the subsequent paragraphs.

Business is more of a professional field, rather than a hard science like Physics. In professional fields, idiosyncratic knowledge about individuals and organizations is arguably more valuable than knowledge of theoretical generalizations. If this is the case, then business professionals should benefit greatly from past stories of successful (or unsuccessful) companies or managers – just like doctors can benefit from studying prior medical cases and pondering over their patients’ medical histories. Case studies often provide fairly detailed, multi-point accounts of organizations and their employees. This knowledge can help students identify problems and create solutions for their own specific organizations, just like knowledge of prior medical cases helps doctors treat their current patients.

Moreover, these idiosyncratic accounts of people and organizations are often told with the level of detail, complexity, and ambiguity that is close to real-life business scenarios. Anyone with experience working in an organization understands that important organizational problems have many contributing factors and related issues. It is often hard to narrow down a problem to a single factor and suggest a simple solution that targets that factor only. Moreover, the way various factors contribute to an important problem is often not deterministic and highly intertwined with other issues and factors. Thus, it is hard to tell with certainty which factors contribute to the problem, in what way, and to what extent. All these issues make case studies a valuable vehicle for exposing students to the real-world complexity that is often present when dealing with various organizational issues within the real world.

As mentioned earlier, addressing these highly complex and often ambiguous organizational problems requires higher-order cognitive skills, such as the ability to apply, analyze, evaluate, and create (see Figure 1). For example, answering the question of why a particular company is experiencing a decrease in market share may require performing an analysis of the industry that the company is in, using the Five Industry Forces Framework. Similarly, evaluating the company’s current situation may require organizing a number of internal and external factors. Creating and recommending a potential solution requires the ability to understand and contextualize various prescriptive management theories (e.g. pursuing strategies that utilize internal strengths for neutralizing external threats or taking advantage of external opportunities is likely to improve organizational performance). To analyze, to evaluate, and to recommend are all higher-order cognitive skills as outlined by Bloom’s Taxonomy. Case studies provide a fruitful platform for practicing and developing these important cognitive skills among students.

Figure 1. Learning Goals under Bloom’s Taxonomy

Finally, case studies give students an opportunity to learn how to apply what they learn in class to a real-world situation. Business is largely an applied field. Basic, fundamental theories in Psychology are often applied to manage people within an organization. Fundamental theories of micro and macroeconomics are applied to an organization or industry to formulate long-term strategies for an organization. Basic mathematical models are used in Accounting and Finance. Thus, being able to apply concepts or theories within real organizations for the purpose of addressing organizational problems or pursuing opportunities is an essential skill for any business professionals. Case studies allow students to do precisely that: to practice applying abstract concepts, frameworks, and theories to complex, ambiguous organizational context for the purpose of attaining positive results for the organization.

It should be noted that these important learning goals can hardly be achieved with what seems to be a more common approach to higher education: in-class lectures delivered using PowerPoint slides followed by multiple-choice exams. Multiple choice questions largely test one’s ability to remember discrete facts and not necessarily to understand a complex organizational problem in a holistic fashion. Moreover, multiple choice questions, while capable of assessing one’s ability to apply concepts, often do so in isolation from the complexity and ambiguity of the context within which these concepts are applied. Finally, there is an inherent determinism in multiple choice questions. Only one answer is the correct one. This stands in sharp contrast to the case study approach, where there is no such thing as a “hundred percent correct answer”.

But this is not to say that multiple choice questions are an inherently inferior assessment based on a case study. Multiple choice questions serve a different purpose: to assess lower level cognitive skills, such as the ability to remember facts or understand basic concepts. That is why multiple-choice assessment is so common in lower level, undergraduate, and introductory courses. Also, with some ingenuity, one can create multiple choice questions that tap into the higher-order cognitive skills. For example, calculating the current cash position of a company and selecting the right answer from the list of several choices may require performing a complex analysis of other financial statements. Similarly, case study question may test the ability of a student to remember simple facts about the company described in the case. Thus, there is a considerable overlap between the educational goals of case-based assessment and other, more basic forms of assessment.

Krotov, V. (2002). Case-Based Assessment: A Theoretical Framework and Practical Advice. Profeducation. Available from Amazon: https://www.amazon.com/Case-Based-Assessment-Theoretical-Framework-Practical-ebook/dp/B08BYW9X79

Krotov, V., & Silva, L. (2005). Case study research: Science or a literary genre?. AMCIS 2005 Proceedings, 50.

Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H. & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. New York: David McKay Company.

Case Method, Pedagogy, Teaching

January 25, 2022January 25, 2022

Components of an Effective Assurance of Learning (AoL) System

By Dr. Pitzel Krotova

The Assurance of Learning (AoL) Requirement

A variety of terms have been used to describe Assurance of Learning (AoL): assessment, assessment of learning outcomes, outcomes assessment, and many others. AoL or assessment can be defined as a systematic and evidence-based approach for assessing the degree to which students meet the stated learning outcomes by devising changes that improve student learning.

An effective and sustainable AoL system is an important requirement of many accreditation standards, such as the Association to Advance Collegiate Schools of Business (AACSB) and the Southern Association of Colleges and Schools Commission on Colleges (SACSCOC). For example, Standard 5 (p. 41) of the 2020 Guiding Principles and Standards for Business Accreditation published by AACSB requires that each school:

… uses well-documented assurance of learning (AoL) processes that include direct and indirect measures for ensuring the quality of all degree programs that are deemed in scope for accreditation purposes. The results of the school’s AoL work leads to curricular and process improvements.

Similarly, Section 8 of the Principles of Accreditation: Foundations for Quality Enhancement published by SACSCOC requires that each educational institution:

…identifies, evaluates, and publishes goals and outcomes for student achievement appropriate to the institution’s mission, the nature of the students it serves, and the kinds of programs offered. The institution uses multiple measures to document student success.

Different Ways to Fail in AoL

Despite the importance of AoL for continuous quality improvement in relation to student learning and attaining and reaffirming accreditation, this is also the area where, based on the comments from many reviewers, many business schools struggle or simply fail. While each successful AoL system has characteristics that are unique to the institutional mission and the context where the system is implemented, it can be argued that an unsuccessful AoL system usually fails in one or more of the four areas listed below:

1. The AoL system is not designed in accordance with best practices or requirements of major accreditation agencies
2. The AoL system is not fully aligned with the institutional mission and goals
3. The AoL system does not produce any traceable changes that result in improvement in student learning
4. The AoL system is not sustainable; it is quickly abandoned after one or more cycles of assessment

Components of a Successful AoL System

In order to comply with the accreditation requirements and produce specific, traceable changes that improve student learning and help the institution achieve its mission and goals, an AoL system must rely on the following components:

- Clear, collectively developed institutional mission and goals
- A list of Institutional Learning Outcomes (ILOs)
- A list of Program Learning Outcomes (PLOs) for each program included in assessment
- A list of specific, measurable Course Learning Objectives (CLOs)
- Alignment of ILOs, PLOs, and CLOs with institutional mission and goals
- Alignment of mission, goals, and learning outcomes with appropriate accreditation standards or industry requirements
- An AoL committee or task force comprised of representatives from various academic and nonacademic units and levels of an HEI
- Templates outlining the general approach and technicalities of assessment and reporting
- A clear, collectively developed and mutually agreed upon assessment plan
- Alignment of assessment planning and reporting
- Templates for assessment maps or matrices
- A simple yet robust system for gathering, analyzing, and distributing AoL data
- Availability of Assessment Coordinators to carry out assessment for individual programs, departments, and colleges
- A variety of formative and summative measures appropriate for the stated learning outcomes
- Clear assignment of responsibilities in relation to AoL
- Clear deadlines for various AoL events and deliverables
- A formal approach to assessment data analysis
- A template for assessment reports
- Dissemination of assessment data to all the relevant stakeholders
- Decision making, recommendations, and changes driven by assessment data
- Continuous improvement in student learning and success based on AoL
- Contribution to broader societal goals through improved teaching and student learning

As one can see from the list, an effective AoL system requires many components and a holistic approach based on the collaboration of all the important stakeholders in education, such as faculty, administrators, staff, students, employers, HEIs, accreditation agencies, and society as a whole.

AoL, Assessment, Assurance of Learning

December 8, 2021December 8, 2021

Risk Management for Business Schools

By Dr. Vlad Krotov and Dr. Jacob Chacko

The 2020 Business Accreditation Standards by AACSB require a business to “maintain an ongoing risk analysis, identifying potential risks that could significantly impair its ability to fulfill the school’s mission, as well as a contingency plan for mitigating these risks.” With the recent events surrounding the COVID-19 pandemic and its impact on educational institutions around the globe, there is a growing realization among business schools and their leaders of the importance and usefulness of Risk Management in their organizations. In this article, we briefly discuss the Risk Management Process and offer simple, practical guidelines on how to identify, analyze, and mitigate risks with the help of a formal Risk Management Plan that is aligned with a broader Strategic Management Plan devised by a business school.

Simplifying Assumptions

In this article, we make a number of assumptions in relation to Risk Management (see Figure 1). We believe that these assumptions will simplify the Risk Management Process and make it more effective in mitigating the identified future risk events.

Figure 1. Risk Management Assumptions

First, we believe that Risk Management is not a “bulletproof shield” for protecting a business school against all possible risks. It is rather a tool or a method that, if used effectively, can reduce the negative impact of risk on the organization. Risk Management can also be misused and turn into a vain exercise. This usually happens when the Risk Management process is (a) based on flawed analysis that does not properly identify and analyze important risks, (b) too complex and, thus, impractical, or (c) not backed by adequate resources required for risk mitigation. Second, we also believe that Risk Management is subjective. Risk Management is much closer to art rather than science; it is based on subjective reasoning and viewpoints, requires imagination for proper risk identification, and is heavily impacted by the “unknowns.” Because of that, we are strongly against a naïve, overly quantitative approach to Risk Management. We do support a formal, structured approach to risk analysis that makes use of appropriate quantitative and qualitative factors. Third, we believe that simplicity is the most effective response to the inherent complexity and serendipity of the environment that many business schools are operating it. We believe that overly complex, highly structured plans are inherently “fragile” in the face of the uncertain, highly complex, and turbulent environment that many business schools are increasingly finding themselves in. Simple, agile plans and structures are more robust and effective during the times of turbulence and uncertainty.

Risk Management Process

Risk Management can be defined as a continuous process comprised of the following steps or phases: analysis of strategic priorities and relevant internal and external factors, identification and definition of risk bearing events, analysis of risks based on likelihood and severity of their impact, mitigating risks by devising response strategies and actions and assigning people responsible for these actions, and monitoring of risks and periodic reporting in relation to these risks to key stakeholders (see Figure 2). Each of these steps is discussed in more detail in the sections below.

Analyzing Strategic Priorities and Relevant Internal and External Factors

Risk Management starts with the analysis of the current strategic priorities. As explained in Standard 1 of the 2020 Business Accreditation Standards published by AACSB, risk management is a part of a broader strategic management process and should be carried out in a way that supports a business school in attaining its strategic goals and objectives. Many of the internal and external risks can be identified by analyzing an organization’s internal strengths and weaknesses together with external opportunities and threats (the so-called SWOT analysis).

Identifying and Defining Risks

After this analysis, the organization should be able to identify and clearly describe important risk bearing events it is facing in relation to its internal and external environments. Examples of external risks include:

- Growing competition for students among existing educational institutions
- Drops in enrollment due to demographics changes
- Deficit of resources due to worsening economic conditions

Examples of internal risks include:

- Decreases in funding due to budgeting changes at the university level
- Inadequate staffing
- Turnover in leadership

A table with clear descriptions of identified risks should be the main deliverable of the risk identification and definition phase.

Analyzing Risks Based on Likelihood and Impact

While all kinds of risks can and should be identified as a part of the Risk Management process, not all risks have the same estimated likelihood and potential impact. Thus, each risk should be carefully analyzed to determine (1) the likelihood of an event occurring and (2) severity of its impact (see Figure 3).

This categorization of risks allows one to prioritize attention and resources in relation to possible future events. Events that are very likely to occur and which can possibly have a great impact on an organization should be treated as critical events. These events require special attention and resources to prevent their negative impact on the organization. Possible future events with moderate likelihood and moderate-to-high impact should be treated as important risk events. While being treated with adequate attention and resources, as a rule, these events should require less attention and resources than critical events with high likelihood and high impact. Low likelihood events with moderate-to-high impact should require a moderate level of attention and resources. Events with moderate-to-high likelihood and low impact should be acknowledged and dealt with, but with a minimum level of resources. Finally, events with low likelihood and low impact should be discussed but probably excluded from a formal risk management plan to keep it simple.

Mitigating Risks by Devising Mitigation Actions and Assigning Responsibilities

After analyzing each possible risk event in terms of its likelihood and impact on the organization, possible actions for mitigating these risks should be devised. It is important to assign to each risk event a “risk owner”—a person responsible for taking a lead on these risk mitigation actions. More thought and extra planning should be put into critical and important events. Important organizational leaders should not be “overextended”; they should be assigned as “leads” only to critical and important risk events.

Monitoring Risks and Establishing Period Reporting to Key Stakeholders

People in charge of the specific risks should be given the formal task of monitoring the internal and external environment of a business school and carrying out mitigation actions designed to protect the organization from a possible negative impact of an event in a proactive fashion or carrying out emergency actions designed to minimize the impact of an event that has occurred already. Without a person responsible for monitoring and mitigating a potential risk event, the organization my find itself in a situation where the event is not identified or dealt with in a timely fashion. Periodic updates by people assigned to risk events should be sent to the dean. The dean can compile all of these reports in a formal Risk Management Plan update that is sent to all the key stakeholders quarterly, biannually, or annually—depending on the complexity and uncertainty of the environment that the school is operating in.

Risk Management Plan

The most important deliverable of the Risk Management process is in the form of a formal risk management plan that is updated periodically, depending on the Strategic Planning cycle length of a business school. The main elements of an effective Risk Management Plan are summarized in Table 1 below.

Strategic Goal 1 – Emphasize Faculty & Staff Development

Risk Description	Importance	Risk Owner	Mitigation Actions	Reporting Timeline	Status Updates
Inferior instructional quality in online courses	Critical	Dept. Chairs, Faculty	Comprehensive faculty training, audit of online classes	On-going	All online courses have been reviewed using a standard quality rubric
Failure to attract and retain qualified faculty	Important	Dean, Dept. Chairs	Faculty development opportunities, faculty satisfaction survey	Annual	A formal business faculty development program was established in collaboration with the Faculty Development Center
Failure to maintain appropriate portfolio of qualified faculty	Important	Dept. Chairs, Assoc. Dean	Develop and maintain a faculty resource plan	Annual	A faculty resource plan has been designed in accordance with AACSB definitions
Failure to maintain AACSB accreditation	Moderate	Dean, Assoc. Dean	Ensure adherence to AACSB standards, focus on continuous improvement	On-going	Faculty sufficiency issue has been communicated to the university’s senior leaders

Table 1. Elements of a Risk Management Plan

Note that the plan contains all the outcomes or deliverables of the steps or phases of the Risk Management Process discussed above. Periodic status updates reported by the people in charge of the risk events are appended to each of the identified risks. Another important characteristic of this Risk Management Plan summary is that it is explicitly linked to Strategic Goal 1 found in the Strategic Plan of the business school.

AACSB, Risk, Risk Management, Strategic Planning

October 21, 2021December 2, 2021

Student Learning Outcomes Assessment

By Dr. James E. Mackin

AoL, Assessment, Assurance of Learning

The assessment of student learning outcomes is probably the most fundamentally important type of assessment that an institution of higher education carries out. By performing student learning outcomes assessments, the institution is asking:

1. What have students learned?
2. How can we tell what students have learned?
3. How can we put our knowledge about what students have learned to use in providing better instruction/assessment of students?

In order to begin to address these questions, it is imperative that all of your faculty members understand how to express measurable student learning outcomes, and you will likely need to conduct numerous workshops on the topic before you can even begin to organize your student learning outcomes assessment protocols and processes. A good starting point for the discussion can be Bloom’s Taxonomy, which basically classifies and conceptualizes the verbs that are used in student learning outcomes assessment.

Student learning outcomes assessment will need to occur in four different contexts at the institution: institutional, general education, academic program, and individual courses. The assessments can be direct (e.g., some percentage of students performed at a satisfactory level on a multiple-choice question on a test) or indirect (e.g., information gathered from surveys), and they can be course-embedded (e.g., a presentation in a class) or they can be external to classes (e.g., results from implementation of a standardized test). The question is, how do we put it all together to make some sense out of what we think students are actually learning at our institution?

It is very important that all learning outcomes are embedded in the planning and assessment process. This means that a learning outcomes assessment plan should be included in each department’s strategic plan for each program that is offered by the department. In addition, the department’s strategic plan should have departmental learning goals and objectives that are broadly connected to the learning outcomes in their program-based learning outcomes assessment plans.

A general template for the learning outcomes assessment plans is shown in the template provided below.

Aol Templates Download

Normally, sections 1-3 of the template will be completed only at the beginning of the life of the relevant strategic plan or upon initial establishment of a new program, while sections 4-6 will be updated on every assessment cycle. Also, graduate programs will not use section 1, where the program learning outcomes are linked to the institutional learning outcomes. Graduate programs are more specialized than undergraduate programs and, although we need to ensure that they are delivering on their promises, these programs generally fall outside the realm of institutional learning outcomes.

There are a variety of tools that can be used to assess student learning outcomes in a program, depending on the outcome, and these tools will be expressed in section 2 of the template. Examples include:

- Course-embedded rubrics;
- Applied learning projects;
- Standardized tests;
- Capstone presentations;
- Course projects;
- Senior surveys;
- Employer surveys;
- Alumni surveys;
- Jury panel surveys;
- External examiner surveys;
- Pre- and post-test comparisons;
- Selected examination questions;
- Comprehensive exams;
- Senior theses.

In addition, student learning outcomes will be embedded in each course in an academic program, although the nature of the outcomes and the level of student achievement expected will vary from course to course. Section 3 of the template provides space for you to compile the level of learning outcome achievement expected for all courses in the program, including introductory, intermediate, and mastery. Although not all courses will address all learning outcomes, the combination of all courses in the program should give students the opportunity to achieve mastery in all of the program student learning outcomes. Ideally, lower level courses provide the opportunity for introductory achievement of learning outcomes, while intermediate and upper level courses provide opportunities for intermediate and mastery achievement of learning outcomes, respectively.

A space for the raw data that is used in assessments is not included in the template; however, that data should be stored in easily accessible locations. Again, as in the case of strategic planning, learning outcomes assessment typically involves massive amounts of data, and it would behoove the institution to invest in software that is specifically designed for outcomes assessments.

For undergraduate programs, every context of learning outcomes assessment should be connected to every other context. This is the reason that the template includes the section (section 1) where program learning outcomes are related to the institutional learning outcomes. Not all institutional learning outcomes will be addressed at all levels (introductory, intermediate, mastery) by an individual major program, but the combination of the major program, the general education program and any additional graduation requirements must address the institutional learning outcomes at the mastery level for every student. This requirement is one reason that many institutions have chosen to address their institutional learning outcomes completely through the general education program.

Because the program learning outcomes are embedded in the strategic planning process, all program learning outcomes will be carried through the review cycle shown in Figure 1.

**Figure 1. An Idealized Planning and Assessment Review Sequence.**
In this sequence, the academic department plans are reviewed by a group of department chairs/heads in a school/college and then the plans are reviewed by the relevant dean, who incorporates appropriate changes into the school/college plan. By the same token, the school/college plans are reviewed by Deans Council and then by the Provost for incorporation into the Academic Affairs Plan. The institutional oversight committee (in this case, the Budget and Planning Committee) reviews all plans on an annual basis.

A faculty subset of the institution-wide committee (the “Budget and Planning Committee” depicted in Figure1), or perhaps another committee, should then be tasked with reviewing the status of the institution with respect to the institutional student learning outcomes. The data for this review would come from the information submitted for the program student learning outcomes assessments.

If there are additional graduation requirements that lie outside the major and general education programs, the learning outcomes assessment data for those requirements will need to be collected and analyzed. It is possible to address the student learning outcomes for the general education program using the template. However, it is important to recognize that the general education program is a unique concept in the sense that, at most institutions, there is no single entity, like an academic department, that oversees general education. It is for this reason that most institutions have the equivalent of a “General Education Committee” and many have a “General Education Assessment Committee” that deals with general education for the institution as a whole. These committees consist of faculty members from across all the areas encompassed by the general education program, and they usually also include accountable administrators. As a general rule, the General Education Committee will be the originator of the program (i.e., the committee that approves all new requirements and any changes needed in the program), while the General Education Assessment Committee will be responsible for program review. Because the responsibilities of the two types of committees are intimately related – one creates and the other reviews – it is often desirable to combine the two into a single committee that handles both sets of responsibilities.

Reviewing the general education program and making changes (i.e., “closing the loop”) is an area where most institutions struggle, simply because of the wide variety of interests that are involved. The situation is simplified when courses are specifically designed for the general education program, but that scenario is rare in higher education. More often, courses serve the dual purposes of meeting academic program requirements and general education requirements. This is one reason that you will want to tie all of your programs, including the general education program, to the institutional learning outcomes. That strategy gives you the flexibility that you need to essentially serve all masters, while at the same time simplifying and constraining approaches to learning outcomes assessment.

Your General Education (Assessment) Committee can come in at the end of the chain in Figure 1 and specifically review the general education learning outcomes while the program and institutional learning outcomes are also being evaluated, probably by other committees. In principle, if all of the general education course requirements are also program requirements, it should be possible to compile the assessment information for the general education program from the information that is submitted for the academic programs. If there are courses in the general education program that are not part of any academic program, then the assessment information will need to be solicited from the relevant course instructors, and separate forms should be developed for that purpose.