ABET. Focus Report. Report of actions taken to address. program weakness and concerns. Of the INFORMATION SYSTEMS MANAGEMENT PROGRAM.

Transcripción

1 ABET COMPUTING ACCREDITATION COMMISSION Focus Report Report of actions taken to address program weakness and concerns Of the INFORMATION SYSTEMS MANAGEMENT PROGRAM Submitted by TECNOLÓGICO DE MONTERREY, CAMPUS MONTERREY June 30 th, 2009 To the Computing Accreditation Commission Computing Accreditation Commission 111 Market Place, Suite 1050 Baltimore, Maryland Ph: Fax: B.S. Information Systems Program Ave. Eugenio Garza Sada 2501 Monterrey, N.L. México, cp Information Systems Management Program Tecnológico de Monterrey Page 1

2 Contents I. Introduction... 4 A. Results from the Evaluation Visit B. Results from the first year report... 4 II. Addressing Program Weaknesses and Concerns... 5 A. Program Objectives and Learning Outcomes adjustments... 5 B. Program Objectives and Learning Outcomes assessments... 9 C. Curriculum D. Program Delivery III. Results analysis A. Program Objectives Assessment Results B. Learning Outcomes Assessment Results C. Advisory Board and Faculty meetings IV. Action Plan V. Summary Appendix A. Learning Outcomes versus ISM Program courses Appendix B. ISM Program Integrated Assessment Plan Appendix C1. TEC Alumni Survey Appendix C2. ISM Alumni Survey Appendix C3. ISM Senior Student Survey Appendix D. ISM Program Rubrics Appendix E. Official Communication: Ethics across Curricula Appendix F. ISM Program: 2009 Assessment Results Appendix G1. ISM Campus Monterrey Faculty meeting minute Appendix G2. ISM Advisory Board meeting minute Information Systems Management Program Tecnológico de Monterrey Page 2

3 List of Figures FIGURE 1. QUALITY IMPROVEMENT PROCESS (QIP)... 5 FIGURE 2. ISM CAMPUS MONTERREY FACULTY MEETING FIGURE 3. ISM ADVISORY BOARD MEETING List of Tables TABLE 1. ISM PROGRAM OBJETIVES... 6 TABLE 2. ISM LEARNING OUTCOMES... 7 TABLE 3. ISM LEARNING OUTCOMES LINKED WITH ABET IS OUTCOMES... 8 TABLE 4. PROGRAM OBJECTIVES (OP S) VS LEARNING OUTCOMES (CE S)... 9 TABLE 5. LEARNING OUTCOMES AND EVALUATION TOOLS TABLE 6. PROGRAM COURSES WITH ETHICS, CITIZENSHIP AND SUSTAINABLE DEVELOPMENT TABLE 7. PROGRAM OBJECTIVES ASSESSMENT RESULTS TABLE 8. LEARNING OUTCOMES ASSESSMENT RESULTS 2009 PART I TABLE 9. LEARNING OUTCOMES ASSESSMENT RESULTS 2009 PART II TABLE 10. RESULT ANALYSIS TABLE 11. ISM ACTION PLAN TABLE 12. ISM PROGRAM RUBRICS AND COURSES Information Systems Management Program Tecnológico de Monterrey Page 3

4 I. Introduction A. Results from the Evaluation Visit 2007 During 2007, our Information Systems Management Program (ISM Program) prepared the documentation for a self study in order to fulfill the ABET CAC requirements and satisfying the Information Systems accreditation criteria. We were visited by ABET Computer Accreditation Commission on September 2007, and we received some feedback about the information that was originally submitted. This process was very useful and extremely helpful. Outside observations made us aware of our opportunity areas to improve our academic quality and close loops on our processes. The report by the ABET commission focused on Criterion 1, which deals with program objectives, learning outcomes and assessments. Particularly, they indicate a program weakness on the Standard 1.5 which states that the results of the program s review and the actions taken must be documented. The ABET evaluation team argues that although assessment results and actions taken have been described and documentation exists for the raw data, no evidence of the systematic documentation of the evaluation of assessment results and actions taken on the basis of those results has been presented. Besides, the team pointed out some concerns about the definition of the program objectives and learning outcomes and how can they be measured. They also mentioned the missing links between assessment processes, program objectives and learning outcomes. They made suggestions about the Curriculum and Program Delivery criteria as well. B. Results from the first year report During 2008, based on ABET-CAC board suggestions, we worked on program objectives and learning outcomes definition. We also recognize the importance of systematic improvement process and defined an institutional model of Quality Improvement Process (QIP) see Figure 1, where periodically we should analyze assessment results to improve not only individual courses (One year Cycle), but program objectives and learning outcomes as well (Five years Cycle). Information Systems Management Program Tecnológico de Monterrey Page 4

5 Figure 1. Quality Improvement Process (QIP) As a result of this process, we developed a new set of program objectives, learning outcomes and assessment activities that were implemented right away. By the time that we presented the first year report, we had established the Advisory Board for the ISM Program (this board represents our constituents), had collected data and had obtained previous results. We realized that this process is quite new and its implementation takes time and effort not only from the program unit but from the entire institution. II. Addressing Program Weaknesses and Concerns A. Program Objectives and Learning Outcomes adjustments After the first year review of our program, embracing and improvement of our QIP process, analyzing results and considering your valuable feedback; we recognize the need to take one step back and rewrite our program objectives and learning outcomes. On the other hand, our institution is growing on this line too. The Academic Affairs office established new standards on program objectives and learning outcomes definition in order to fulfill our institutional vision and mission. Those must include: Information Systems Management Program Tecnológico de Monterrey Page 5

6 To be successful on his/her professional field. (Successful) To be up to date about his/her professional field and political, social and economic environment. (Up to date) To be able to work on multidisciplinary teams. (Team work) To be committed with sustainable development including economic, social and environmental issues. (Sustainable development) To take decisions based upon ethical criteria about the people and the environment. (Ethics) To be able to identify and develop new business. (Entrepreneurship) To be able to compete on international arena and to adapt within multicultural environments. (International) Under Academic Affairs office leadership, the ISM Program Evaluation Committee was integrated by ISM Program Directors from several campus like Mexico State Campus, Puebla Campus, Santa Fe Campus and Monterrey Campus. This committee rewrites the program objectives linked with our institutional vision shown on Table 1. Table 1. ISM Program Objetives OP1 OP2 OP3 OP4 ISM Program Objectives Alumni graduated from this program will be a professional that generates critical changes in processes and business. He or she will be able to perform management process optimization based on IT to reach better operational efficiency in industry, government or any national/international organizations. Alumni graduated from this program will be a professional that develops strategies based upon the use of information technologies for industry, either his/her own or not, and for both national and international. He or she contributes to the sustainable development of his/her community by using processes innovation where the major enabler is information technologies. Alumni graduated from this program will be a professional that both, individually and integrating collaborative groups, performs business process innovation supported by information technologies. Those actions contribute to improving the competitiveness and productivity of the organization under the context of the sustainable use of natural resources. Alumni graduated from this program will be a professional that respects persons and his/her environment and takes decisions based upon ethical criteria. Institutional guidelines Successful International Entrepreneurship Sustainable development Up to date Team work Sustainable development Ethics Information Systems Management Program Tecnológico de Monterrey Page 6

7 These new program objectives, describes an Information Systems Management professional from the Tecnológico de Monterrey (ITESM). Moreover, learning outcomes became our institutional standards and ABET-IS requirements as well. Table 2 shows the ISM Learning Outcomes and Table 3 shows the ISM Learning Outcomes linked to ABET-IS. Table 2. ISM Learning Outcomes CE1 CE2 CE3 CE4 CE5 CE6 CE7 CE8 CE9 CE10 CE11 CE12 CE13 ISM Learning Outcomes The student designs and models business processes that optimize the use of organizational resources such as human, financial and technological enablers. The student analyzes, designs, develops, implements and evaluates computer based information systems that fulfills the established needs and requirements of a given organization or application. The student comprehends the processes that support the administration of Information Technology services within a given organization. The student works effectively in collaborative teams to develop projects in the information technology field. The student develops the ability to analyze the local and global impact of computation in individuals, in organizations, and in society to orient with responsibility his or her information technology services. The student communicates efficiently proposals and project results in the information technology field, both using oral and written reports as well as using either English or Spanish. The student is entrepreneur, innovator and creative. The student has the ability to analyze quantitatively the information relative to the organization operation, in order to design models based upon information technologies which support decision making processes. The student has the capacity to analyze a problem and select the information technology as the major enabler to convey an appropriate solution. The student recognizes the need to commit himself in pursuing his personal and professional development. The student has the ability to apply techniques and current methodologies that are necessary for the practice of his/her profession. The student is able to identify, analyze and evaluate ethical dilemmas related with his/her person, profession and environment; respecting in all circumstances, persons and environment. The student has a global vision of society and posses the capacity to adapt him or herself to other cultures. Information Systems Management Program Tecnológico de Monterrey Page 7

8 Table 3. ISM Learning Outcomes linked with ABET IS outcomes CE1 CE2 CE3 CE4 CE5 CE6 ISM Learning Outcomes The student designs and models business processes that optimize the use of organizational resources such as human, financial and technological enablers. The student analyzes, designs, develops, implements and evaluates computer based information systems that fulfills the established needs and requirements of a given organization or application. The student comprehends the processes that support the administration of Information Technology services within a given organization. The student works effectively in collaborative teams to develop projects in the information technology field. The student develops the ability to analyze the local and global impact of computation in individuals, in organizations, and in society to orient with responsibility his or her information technology services. The student communicates efficiently proposals and project results in the information technology field, both using oral and written reports as well as using either English or Spanish. ABET IS Outcomes An ability to design, implement and evaluate a computer-based system, process, component, or program to meet desired needs. An ability to design, implement and evaluate a computer-based system, process, component, or program to meet desired needs. An understanding of processes that support the delivery and management of information systems within a specific application environment. An ability to function effectively on teams to accomplish a common goal. An ability to analyze the local and global impact of computing on individuals, organizations and society. An ability to communicate effectively with a range of audiences. CE7 The student is entrepreneur, innovator and creative. There are no equivalence CE8 CE9 CE10 CE11 CE12 CE13 The student has the ability to analyze quantitatively the information relative to the organization operation, in order to design models based upon information technologies which support decision making processes. The student has the capacity to analyze a problem and select the information technology as the major enabler to convey an appropriate solution. The student recognizes the need to commit himself in pursuing his personal and professional development. The student has the ability to apply techniques and current methodologies that are necessary for the practice of his/her profession. The student is able to identify, analyze and evaluate ethical dilemmas related with his/her person, profession and environment; respecting in all circumstances, persons and environment. The student has a global vision of society and posses the capacity to adapt him to other cultures. An ability to apply knowledge of computing and mathematics appropriate to the discipline. An ability to analyze a problem, and identify and define the computing requirements appropriate to its solution. Recognition of the need for, and an ability to engage in, continuing professional development. An ability to use current techniques, skills, and tools necessary for computing practices. An understanding of professional, ethical, legal, security, and social issues and responsibilities. There are no equivalence Information Systems Management Program Tecnológico de Monterrey Page 8

9 A direct correspondence between program objectives and learning outcomes must exist to have a coherent operation of the formative learning process. Table 4 shows the link between program objectives and learning outcomes. Also, all learning outcomes are related to courses of the entire program as shown in Appendix A. Table 4. Program Objectives (OP s) vs Learning Outcomes (CE s) OP1 OP2 OP3 OP4 CE1 CE2 CE3 CE4 CE5 CE6 CE7 CE8 CE9 CE10 CE11 CE12 CE13 The new program objectives and learning outcomes presented by the ISM Program Evaluation Committee were reviewed and accepted by the ISM National Faculty (ISM Program and Information Systems Department directors from all campus); and Information Systems Department on Campus Monterrey. This process is fully documented electronically and managed by Academic Affairs Office. B. Program Objectives and Learning Outcomes assessments Once program objectives and learning outcomes were defined, we stated the performance criteria that will focus on measurable elements included in our assessments or evaluation tools. If they are not already defined, we will develop new ones. The result is our Integrated Assessment Plan shown on Appendix B. This master plan, not only linked results with each program objective and learning outcome but define application frequencies, goals and people responsible of evaluation tools and results. Information Systems Management Program Tecnológico de Monterrey Page 9

10 Once the Integrated Assessment Plan was defined, we were ready to review and/or develop the evaluation tools needed, such as rubrics, surveys and statistics. For the program objectives assessment, we defined the following evaluation tools: TEC Alumni Survey, designed and applied, once a year, to all Tecnológico de Monterrey alumni by the Institutional Effectiveness Department. It consists of general questions related to our mission. This survey is presented in Appendix C1. ISM Alumni Survey, used to know what ISM graduates are doing after five years. Appendix C2 illustrates the questions of the survey, which are mainly related to their performance as IT professionals. For the learning outcomes assessment, we developed the following evaluation tools: ISM Senior Students Survey. This tool helps us with gathering opinions from our senior students in order to review the accomplishment of our new learning outcomes and contrast them with quantitative data obtained from other listed evaluation tools. Appendix C3 shows this survey. ISM Program Rubrics. We design rubrics to assess contents and competencies related to our courses. They provide indicators to the level of performance acquired by the student on specific learning outcomes. We organized them on five content-oriented rubrics as follow: Business Process Management (MPN Rubric) Information Systems Development (DSI Rubric) Information Technology Services (STI Rubrics) Quantitative Analysis (QA Rubrics) Information Technology (PTI Rubrics) And also, two soft skill-oriented rubrics: Team Work (TEQ Rubric) Oral & written communications (PSI Rubric) Appendix D shows rubric categories and their descriptions. The full development of these rubrics is still in progress. Information Systems Management Program Tecnológico de Monterrey Page 10

11 Quality Enhancement Plan (QEP). This plan was developed by our institution in order to accomplish our vision and mission elements related to Citizenship & Ethics criteria. The primary QEP objective is to strengthen the ethics and citizenship core competencies of Tecnológico de Monterrey students so that, they will transcend the university period and constitute a hallmark of their long term professional life. This objective is to be achieved through the institution s curricular programs which offer students learning experiences in these areas. Those competencies are: Competency 1: Reflect on, analyze and evaluate ethical dilemmas related to the person, professional behavior, and the milieu. Competency 2: Respect for people and their milieu. Competency 3: Knowing about and being sensitive towards improving social, economic and political issues. Competency 4: Acting with civic spirit and responsibility to improve the quality of life in the community, especially in needed communities. The evaluation plan for the ethics and citizenship core competencies will be carried out using both direct and indirect measurement techniques. Direct measurement techniques include two rubrics: A rubric for assessing ethic competencies that will be applied over a sample of final projects done by students completing the courses Ethics and Society; and Ethics, Profession and Citizenship. (QEP Ethics Rubrics) A rubric for assessing citizenship competencies that will be applied over a sample of final reports of experiences (REC) written by students who have completed their community service. (QEP REC & QEP - Citizenship Rubrics) Indirect measurement techniques include: National Survey of Student Engagement (NSSE) Ethics and Citizenship Evaluation Tool (QEP-ICEC Survey) Institutional Effectiveness Department is in charge of gathering, processing and sharing the data results. Information Systems Management Program Tecnológico de Monterrey Page 11

12 On the webpage detailed information of our plan is published. It includes the full QEP document English version, where rubrics and evaluation tools could be found; following the next path: Documentos QEP - Documento QEP versión inglés. CENEVAL Test. Evaluation National Center (CENEVAL) is a nationwide organization integrated by representatives from public and private universities dedicated to design and apply specialized field tests such as informatics, computer science, mechanics, electronics, and so on. We use this test as an external evaluation tool. In our case, the sections included on the test refer to Software, Hardware, Mathematics, Data Bases, Ethics and Social issues. The maximum score is 1300 and CENEVAL offers two kinds of distinctions based on the candidate s performance. A Satisfactory recognition is awarded to candidates scoring 1000 and above, whereas Outstanding recognition is given to candidates scoring 1150 or more. TOEFL Institutional Test. The Test of English as a Foreign Language is applied by our Campus Academic Affairs Office to all of our students. A score of at least 550 points is a graduation requirement for everyone. Entrepreneur Semester Fair. As a result of Entrepreneurship Development course, every semester, the students show the business plan during this fair. There are around 300 teams per semester, only 90 are selected to present their project on the fair, where is evaluated by a jury who select top ten projects. The semifinal projects are presented to a new jury who select top five projects. Final presentations are made in front of a jury integrated by local company top executives and professors. The pride is sponsored by FRISA (Coca-Cola Co.) Besides, the Office of the Registrer collects and archives all available student information such as courses taken, grades and performance that can be used as indicators of student success. This also represents an input for our evaluation process. We identify this information as Student Academic Record. Information Systems Management Program Tecnológico de Monterrey Page 12

13 Finally, the Academic Affairs Office is developing an institutional data gathering tool to help everyone responsible of handle each program evaluation. Tools, results and analysis techniques will be readily available on December Table 5 summarizes the relation between learning outcomes and evaluation tools. Table 5. Learning Outcomes and Evaluation Tools. ISM Senior Student Survey ISM Program Rubrics QEP Citizenship Rubrics QEP Ethics Rubrics QEP - REC Experience Report QEP ICEC Survey CENEVAL Test TOEFL Test Entrepreneurship Semester Fair Student Academic Record CE1 CE2 CE3 CE4 CE5 CE6 CE7 CE8 CE9 CE10 CE11 CE12 CE13 C. Curriculum The concern found here was related to the extent of how our students are exposed to the social and ethical issues in the computing and IT contexts. As part of the ISM program, we have courses related to social and ethical issues. For example, students must take an ethics course in their 3 rd semester; a course called Science, Technology and Society in their 5 th semester, and a senior course (9 th semester) in Ethics, Profession and Citizenship. At the same time, a first semester course called Introduction to the Profession, has made more emphasis in the Professional Code of Ethics proposed by ACM and has included some readings and projects related to social implications of IT in today's society. Information Systems Management Program Tecnológico de Monterrey Page 13

14 As part of the Quality Enhancement Plan (QEP), the Academic Affairs Office states a concept called Ethics across curricula (Appendix E shows the official communicate) and the ISM National Faculty worked last year on it. We established, besides the official course in Ethics and Society, a set of courses that will include contents and activities addressing the social and ethical issues related to Information Systems context. Table 6 shows the courses where those topics are discussed. Table 6. Program Courses with Ethics, Citizenship and Sustainable Development ID Course Name Sustainable Citizenship Ethics development D2001 Information Technology Law MT1001 Marketing TI1002 Introduction to the Profession of Electronics and Information Technologies TI1005 Information Systems Development Project TI1007 Information Systems Introduction TI2001 E-Business TI2002 Business Process Management TI2003 Management of Electronic and Information Technologies Projects TI3001 Information Technology Strategic Planning TI3014 Information Technologies Capstone Project TI3023 Enterprise Information Systems TI3024 Information Technology Service Management D. Program Delivery The concern found here was related to a heavy teaching load for some faculty members. As a result of the ABET Visit, our President decided to reduce the teaching workload for Program Directors so we can dedicate more time on students advising. At the same time, as our campus is moving towards a research orientation, more professors have been invited to have an active participation in our Research Chairs as well as our Outreach Continuing Education and Consultancy Chairs which includes a reduction on the teaching loads to spend more time on research or consultancy activities. Information Systems Management Program Tecnológico de Monterrey Page 14

15 III. Results analysis A. Program Objectives Assessment Results Once we applied the evaluation tools available at this point, we obtained results of the program objectives and learning outcomes assessment. Table 7 shows a summary report of ISM Program Objectives Assessment Results with traffic lights colors that show how goals were achieved. Appendix F shows the details of ISM 2009 Assessment Results. It is important to remember that this program has been updated almost every three years since Usually our academic programs are updated every five years; however, our field of knowledge is changing constantly. The main characteristics of each version are: ISM Program 2002 Version This version of the plan was built after 2005 version was release. Because, the official name of the program was changed according to an institutional strategy of reposition the information technology academic programs. It included topics such as Business intelligence and Consulting and a Capstone Project on 9 th semester. ISM Program 2005 Version This version of our program includes 3 integrated projects. The first one is on 3 rd semester where students develop a basic information system for a small nonprofit organization. The second one is on 6 th semester where students develop a Web based information system. And, the third one is on 9 th semester where student present an IT proposal to a small or medium company. This version also includes specialization tracks on Business Intelligence, Multimedia and IT Marketing. ISM Program 2008 Version This version of the plan removes specialization tracks and reinforces topics such as Business Intelligence, Business Process Management and IT Services Management. With the analysis of our program objectives assessment results, we can see that alumni with 5 years of professional experience, enrolled on the 2002 version of ISM Program, did not have business process management courses as in the 2008 version; however, more than 50% is working on this field which implies that this is a demanded skill that we already included on our program. Even when they are too young, almost 30% work as an executive developing strategies based on IT and almost 20% have his own business. But, they need to spend more time in Information Systems Management Program Tecnológico de Monterrey Page 15

16 training or have more interest in pursuing specialized certifications. However, summary results show an acceptable achievement of established goals. Table 7. Program Objectives Assessment Results 2009 CD1 CD2 CD3 CD4 Summary OP1 OP2 OP3 OP4 ET S1 S2 S1 S1 Q II.a y b G 60% 60% 80% 80% 80% 80% 30% 80% R 38% 53% NA NA 100% 94% 32% 77% ET S1 S1 S1 S1 Q G 30% 30% 70% 20% 40% 60% 90% R 29% 31% 59% 18% 53% 35% 71% ET S1 S1 Q G 40% 30% 90% R 47% 29% 100% ET S1 S1 Q G 100% 40% R 88% 35% OP# Program Objective ET Evaluation Tool S1 ISM Alumni Survey CD# Performance Criteria Q Question Number S2 Tec Survey NA Not Available G R Result B. Learning Outcomes Assessment Results Table 8 and Table 9 illustrate ISM Learning Outcomes Assessment Results. They indicate with traffic lights colors how goals were achieved. The learning outcomes assessment results show us, at this point, information about almost every evaluation tools. However, once we apply all the ISM Program Rubrics we will be able to compare students opinions versus academic results. Table 10 summarizes the results analysis for each learning outcome. Information Systems Management Program Tecnológico de Monterrey Page 16

17 Table 8. Learning Outcomes Assessment Results 2009 Part I CE1 CE2 CE3 CE4 CE5 CD1 CD2 CD3 Summary ET MPN MPN S3 MPN S3 MPN S3 Q/C TI3014 TI A & C TI B & D CD2000 G 80% 80% % % 3.5 R NA NA 4.1 NA 4.3 NA 3.3 ET DSI S3 CT1 CT2 DSI S3 DSI S3 Q/C TI A B C TI D & E TI3014 G 80% % 90% 80% % 3.5 R NA % 17% NA 3.3 NA 3.8 ET STI STI S3 STI S3 STI S3 Q/C TI3014 TI B TI A & D AD3002 G 80% 80% % % 3.5 R NA NA 3.0 NA 3.4 NA 3.5 ET TEQ TEQ TEQ TEQ S3 Q/C TI2003 TI3014 TI2007 TI G 80% 80% 80% 80% 3.5 R NA NA NA NA 4.0 ET QEP - REC S3 QEP-ICEC S3 QEP-ICEC Q/C 35 #1 9 #2 G 80% % 40% 70% 30 E 31 F & G 32 C CE6 R NA % 33% 55% ET PSI PSI S3 PSI PSI S3 TT Q/C TI3014 TI A & B TI3014 TI C & D G 80% 80% % 80% % R NA NA 4.0 NA NA % CE# Learning Outcome ES Entrepreneurship Semester Fair QEP - ET Ethics Rubrics from QEP CT1 CENEVAL Test. Software Engineering Section ET Evaluation Tool QEP - ICEC Ethics & Citizenship Evaluation Tool CT2 CENEVAL Test. Programming Section G QEP - REC Citizenship Experiences Report CT3 CENEVAL Test. Mathematica Section MPN Business Process Management Rubric R Result CT4 CENEVAL Test. DataBase Section NA Not Available S3 BTM Senior Students Survey CT5 CENEVAL Test. Hardware Section PSI Oral & Written Communications Rubric SAR Student Academic Record CT6 CENEVAL Test. Informatics & Ethics Section PTI Information Technology Rubric STI Information Technology Services CD# Performance Criteria Q/C Question/Course Number TEQ Team Work Rubrics DSI Information Systems Development Rubric QA Quantitative Analysis Rubric TT TOEFL Test Information Systems Management Program Tecnológico de Monterrey Page 17

18 Table 9. Learning Outcomes Assessment Results 2009 Part II CE7 CE8 CE9 CE10 CE11 CE12 CD1 CD2 CD3 Summary ET SAR ES S3 S3 Q/C G 20% 50% 50% 50% R 3% 0% 17% 17% ET QA CT3 S3 QA CT4 S3 QA S3 Q/C CD A TI C TI3023 G 80% 90% % 90% % 3.5 R NA 58% 2.8 NA 50% 3.4 NA 3.4 ET PTI PTI CT5 Q/C TI3014 TC2009 G 80% 80% 90% R NA NA 100% ET SAR S3 SAR S3 SAR S3 S3 Q/C G 70% 30% 30% 20% 40% 50% 40% R 10% 17% 3% 33% 16% 55% 50% ET MPN PTI PTI S3 Q/C TI3014 TI2007 TI G 80% 80% 80% 80% R NA NA NA 93% ET CT6 S3 QEP - ICEC QEP - ET QEP - ET QEP - ET QEP - ET Q/C 37 # 3 H2002 TI3014 TI2007 TI1005 G 90% % 80% 80% 70% 60% 33 B CE13 R 92% % NA NA NA NA ET SAR SAR S3 S3 S3 Q/C G 30% 5% 40% 40% 60% R 6% 2% 50% 33% 67% CE# Learning Outcome ES Entrepreneurship Semester QEP - ET Ethics Rubrics from QEP Fair CT1 CENEVAL Test. Software Engineering Section ET Evaluation Tool QEP - ICEC Ethics & Citizenship Evaluation Tool CT2 CENEVAL Test. Programming Section G QEP - REC Citizenship Experiences Report CT3 CENEVAL Test. Mathematica Section MP Business Process Management Rubric R Result N CT4 CENEVAL Test. DataBase Section NA Not Available S3 BTM Senior Students Survey CT5 CENEVAL Test. Hardware Section PSI Oral & Written Communications Rubric SAR Student Academic Record CT6 CENEVAL Test. Informatics & Ethics Section PTI Information Technology Rubric STI Information Technology Services CD# Performance Q/C Question/Course Number TEQ Team Work Rubrics Criteria DSI Information Systems Development Rubric QA Quantitative Analysis Rubric TT TOEFL Test Information Systems Management Program Tecnológico de Monterrey Page 18

19 Table 10. Result Analysis Learning Outcome CE1 CE2 CE3 CE4 CE5 CE6 CE7 CE8 CE9 CE10 CE11 CE12 CE13 Result Analysis The students opinion is good about analyze, design and modeling processes skills, however it is important to increase their ability on resources optimization. It is important to mention that, these students are enrolled on ISM 2002 plan version. Next semester, we will analyze the first generation of students enrolled on ISM Program 2008 version where resources optimization skills are reinforced. CENEVAL Test results are not good about programming skills. However, students opinions consider they have good programming skills. The students opinion says that we must reinforce PMI standard use and auditing methodologies practices. Students opinion about their team work skills is very good. Students opinion about their basic knowledge of social responsibility is good. However, they are not involved on IT community impact projects. For senior students, their oral and written communication skills in Spanish as well as in English are good. However, from the total enrolled students there still are a significant number of students under a 550 TOEFL score. This semester, there were no students participating on Entrepreneurship Semester Fair, because there were no students taking Entrepreneurial Development course. However, 17% of senior students participate on that fair when they took the course. Besides, there is not significant participation on Entrepreneurship Program (MEM) or on contests. CENEVAL Test shows poor results on Mathematics section. Students consider that they are able to analyze a problem and select the appropriate information technology for it solution. The number of senior students enrolled on our internal professional certificates is 10% and 17% obtained external professional certificates. Besides, students are not used to take courses teaching in foreign languages, but 33% of our senior students speak a third language. There is not enough interest on participate in extra academic activities. However, 50% of our senior students get Student Development Diploma. At this point, we only know that our students are able to self - learn any IT technology. Even when they know about ethics criteria, they do not know how to respect other s opinions. They international vision is good but they should participate in more international activities such as take program courses in English. Information Systems Management Program Tecnológico de Monterrey Page 19

20 C. Advisory Board and Faculty meetings These results were presented to ISM Campus Monterrey Faculty integrated by teachers from Information Systems, Computer Science and Business Administration Departments involved with ISM courses. They are on charge of design and apply ISM Program rubrics on their courses next year. Figure 2 shows ISM Campus Monterrey Faculty meeting. Figure 2. ISM Campus Monterrey Faculty meeting From left to right: Dr. Miguel Perez, Mtra. María Elena Dieck, Dr. David Alanís, Dra. Lorena Gómez, Mtra. Bertha García, Mtra. Adriana Almaguer, Mtra. Elda Quiroga (BCT Program Director), Mtra. Leticia Almaguer (ISM Program Director), Dr. Rodolfo Castello (Dean Division of Mechatronics and Information Technology), Dra. Celina Torres (Information Systems Department Director), Mtro. Román Martínez (ABET Liaison Officer at ITESM), Dr. Juan Nolazco (Computer Science Department Director). June 9 th, The conclusions of this meeting were the following: Enhance individual programming skills through integrated projects. Select some courses were we could add activities oriented to develop oral and written communication in English. At this time, only the TI3014 Capstone IT Project has this kind of activities. All of us must promote entrepreneurship and the participation on Entrepreneurship Semester Fair. We have to enhance teacher collaborative work through topic focus groups responsible of design and review course contents linked with ISM Program Learning Outcomes. Each topic focus group is going to be responsible of develop, apply and assess rubrics which achieve program learning outcomes. Each topic focus group has a representative on ISM Campus Monterrey Faculty. The minute of this meeting is shown on Appendix G1. Information Systems Management Program Tecnológico de Monterrey Page 20

21 Also, the results were presented to ISM Advisory Board during the second annual meeting. This committee is integrated by faculty and industry members. Figure 3 shows ISM Advisory Board meeting. Figure 3. ISM Advisory Board meeting From left to right: First row - Mtra. Dolores Lankenau, Mtra. Leticia Almaguer (ISM Program Director), Lic. Claudia González (Kernell IT). Second row - Dra. Celina Torres (Information Systems Department Director), Mtro. Gustavo Cervantes, Lic. Alvaro Rodríguez (Consultant), Mtro. Román Martínez (ABET Liaison Officer at ITESM), Dr. Rodolfo Castello (Dean Division of Mechatronics and Information Technology), Lic. Karina Hernández (Grupo Financiero Banorte Regional Bank). June 10th, The conclusions of this meeting were: Some courses should provide methods to link our profile to the organizational changes. Also, they need to provide organizational demands, attitude change and continuous training. It is important to communicate companies the skills and competences that ISM program develops over our students. The minute of this meeting is shown on Appendix G2. Information Systems Management Program Tecnológico de Monterrey Page 21

22 IV. Action Plan During this year we have performed important reflections about our program. The Integrated Assessment Plan has shown to be a very useful tool, it demonstrates us that there is a way that we can accomplish our ideal ISM profile. The entire institution is growing through this new process of program assessment, this is a guarantee that help will be provided for each program action plans. Table 11 shows the list of actions defined in order to reduce the gap between the goals and results of our learning outcomes. Additionally, next year we will take the following general actions: Consolidate topic focus groups. (Business Process group, Business Intelligence group, etc.) These groups will be in charge of apply rubrics, collect evidences and define action in order to improve our student skills on the line of our new learning outcomes. Table 12 illustrates the relation between ISM Program Rubrics and courses. Communicate Program Objectives and Learning Outcomes. All people related with our program must know about those, mainly our teachers and students. We are going to publish using electronic tools and printed forms this information for the academic community. We are going to design an employer oriented document that describes ISM graduates knowledge and skills. Establish a Social Center for IT innovation. This center will provide students the opportunity to have real IT projects with not profit organizations, so they can experiment and evaluate full information systems development life cycle and the social impact of computing. These actions will be leading by ISM Program Director and supported by Mechatronics and Information Technology Division Dean as well as School of Engineering Dean. Information Systems Management Program Tecnológico de Monterrey Page 22

23 Table 11. ISM Action Plan Learning Outcomes CE1 Action Plan Business Process group will apply MPN Rubrics. Results will be reviewed on December CE2 Information System group will apply DSI Rubrics. Results will be reviewed on December In order to improve CENEVAL Test on Programming section we will offer a preparation session including programming basis and algorithms topics. C3 IT Services group will apply STI Rubrics. Results will be reviewed on December C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 Teamwork group will apply TEQ Rubrics. Results will be reviewed on December Teachers involved with D2001, TI3001 and TI3023 will include topics and activities in their courses related with social responsibility and will apply QEP Citizenship Rubrics. Information System group will apply PSI Rubrics. They will define new activities in order to improve English communication skills. Results will be reviewed on December There are 42 students below 550 points on TOEFL Test. ISM Program Director will have an interview with each one in order to establish a personal English skills improvement plan. All students should take TOEFL Test during Spring semester ISM Program Director will follow up students on Entrepreneur Development course in order to encourage them to participate on Entrepreneurship Semester Fair. ISM Program Director will publish the achievements of students enrolled on MEM Program using the ISM electronic space on Blackboard. Business Intelligence group will apply QA Rubrics. Results will be reviewed on December In order to improve CENEVAL Test on Mathematics section we will offer preparation session including topics such as calculus and statistics. Information Technology group will apply PTI Rubrics. Results will be reviewed on December ISM Program Director will encourage students to select any internal or external professional certificate available for ISM Program as well as study a third language through ISM electronic space on Blackboard and personal interviews. This learning outcomes use MPN and PTI Rubrics. Business Process and Information Technology groups will apply these rubrics. Results will be reviewed on December Teachers involved with IT projects will apply QEP Ethics Rubrics on their courses, as well as Ethics course s teachers. Results will be reviewed on December ISM Program Director will encourage students to select any international interchange available for ISM Program through ISM electronic space on Blackboard. Information Systems Management Program Tecnológico de Monterrey Page 23

24 Table 12. ISM Program Rubrics and courses Learning Outcomes Rubrics Courses AD3002 Management Consulting CD2000 Quantitative Methods for Making Decisions D2001 Information Technology Law H2002 Ethics, Profession, and Citizenship TC2009 Use and Management of Operating Systems TI1005 Information Systems Development Project TI2002 Business Process Management TI2003 Management of Electronic and Information Technologies Projects TI2007 E-business for Decision Support Project TI2009 Databases for Decision Support TI3001 Information Technology Strategic Planning TI3014 Information Technologies Capstone Project TI3023 Enterprise Information Systems TI3024 Information Technology Service Management CE1 CE2 CE3 CE4 CE5 CE6 CE8 CE9 CE12 MPN DSI STI TEQ QEP Citizenship PSI QA PTI QEP Ethics Information Systems Management Program Tecnológico de Monterrey Page 24

25 V. Summary In order to address program weakness and concerns, we presented the procedures that we followed last academic period and its results. Program weakness. Your documentation requirement about program weakness has been covered as follow: Documentation of assessment data for each objective and each outcome. o Table 7, Table 8 and Table 9 show program objectives and learning outcomes summarized reports. o Appendix F shows the details of ISM 2009 Assessment Results. Documentation showing how the extent to which each objective and each outcome is met has been assessed and evaluated. o Integrated Assessment Plan is shown on Appendix B. It describes program objectives, learning outcomes, performance criteria, evaluation tool for each one as well as application frequencies, people responsible and goals. Evidence of the documentation of the evaluation of assessment results and actions taken on the basis of that evaluation. o There were two meetings where results were presented. Figure 2. ISM Campus Monterrey Faculty meeting and Figure 3. ISM Advisory Board meeting show the people involved on this process. The action plan, as a result of these meeting, is shown on Table 11. Program Concerns. The results of actions taken over program concerns have been presented as follow: Criterion I. Objectives, Outcomes and Assessments. o New ISM program objectives (OP) are shown on Table 1. Those include elements that we are established on our institutional mission. o Learning outcomes (CE) are shown on Table 2. We think that they appear now much more like learning outcomes as the previous version presented on first year report. They are linked with ABET IS outcomes requirements as well. o We defined Performance Criteria (CD) for each objective and outcome in order to split them in small measurable pieces. Information Systems Management Program Tecnológico de Monterrey Page 25

26 o Once Performance Criteria were defined, we selected evaluation tools already available; or defined new ones in order to link them with each objective and outcome. Appendix B shows the Integrated Assessment Plan. o Results showed on Table 7, Table 8 and Table 9 clearly link each objective and outcome with the evaluation tool applied. Also, Appendix F shows detailed ISM 2009 Assessment Results for each objective and outcome according with their performance criteria. o Documentation of this process is stored and available for ISM National Faculty on Academic Affairs office electronic space on Blackboard. Criterion IV. Curriculum. o Ethics across curriculum was an activity where ISM National Faculty selected program courses in order to include ethics and social responsibility topics and activities that link them with Informatics context. Table 6 shows courses selected. Criterion IV. Program Delivery. o As a first step toward work load review, our President made some changes over Program and Department Directors work load. Also, there were work load policy changes over teachers involved on our Research and Outreach Continuing Education and Consultancy Chairs. Information Systems Management Program Tecnológico de Monterrey Page 26

27 Appendix A. Learning Outcomes versus ISM Program courses. Information Systems Management Program Tecnológico de Monterrey Page 27

28 B.S. Information Systems Management 2008 Sem ID Course Name CE1 CE2 CE3 CE4 CE5 CE6 CE7 CE8 CE9 CE10 CE11 CE12 CE13 1 AD1000 Management 2 AD1001 Organizational Behavior 9 AD3002 Management Consulting 3 CD1000 Business Statistics I 4 CD1002 Business Statistics II 7 CD2000 Quantitative Methods for Making Decisions 6 CD2001 Multivariable Statistics 1 CF1000 Financial Information 3 CF1003 Administrative Accounting 2 CF1005 Cost Analysis 8 D2001 Information Technology Law 2 EC1001 Economics 8 EM1001 Entrepreneurial Development 4 FZ1000 Financial Mathematics 5 FZ2000 Financial Management 1 H1016 Foreign Language 2 H1017 Analysis and Verbal Expression Workshop 3 H1018 Ethics, Self and Society 3 H2001 Verbal Expression in the Workplace 9 H2002 Ethics, Profession, and Citizenship 4 HS2007 Humanistic Perspective 5 HS2008 Scientific and Technological Perspective 6 HS2009 Sociopolitical Perspective 1 MA1003 Mathematics I 2 MA1005 Mathematics II Information Systems Management Program Tecnológico de Monterrey Page 28

29 Sem ID Course Name CE1 CE2 CE3 CE4 CE5 CE6 CE7 CE8 CE9 CE10 CE11 CE12 CE13 4 MT1001 Marketing 1 TC1002 Programming Fundamentals 3 TC1004 Computer Organization 2 TC1005 Programming and Data Structure 6 TC1007 Computer Networks I 5 TC1009 Information Systems Development and Databases 7 TC2009 Use and Management of Operating Systems 1 TI TI1002 Introduction to the Human-Computer Interfaces and Internet Applications Introduction to the Profession of Electronics and Information Technologies 4 TI1005 Information Systems Development Project 3 TI1007 Information Systems Introduction 5 TI2001 E-Business 4 TI2002 Business Process Management 5 TI2003 Management of Electronic and Information Technologies Projects 6 TI2007 E-business for Decision Support Project 5 TI2008 Develop Environments 6 TI2009 Databases for Decision Support 6 TI3001 Information Technology Strategic Planning 7 TI3010 Knowledge Management 9 TI3014 Information Technologies Capstone Project 9 TI3022 Business Intelligence 8 TI3023 Enterprise Information Systems 8 TI3024 Information Technology Service Management Elective courses not included Information Systems Management Program Tecnológico de Monterrey Page 29

30 Appendix B. ISM Program Integrated Assessment Plan. Information Systems Management Program Tecnológico de Monterrey Page 30

31 TECNOLÓGICO DE MONTERREY The B.S. Information Systems Management Integrated Assessment Plan is based upon Program Objectives (OP s) and Learning Outcomes (CE s) and Performance Criteria (CD s) presented by ISM Program Evaluation Committee and approved by ISM Program Faculty. The ISM Program Evaluation Committee was integrated by: COORDINATOR Name: Campus: Lic. Leticia Almaguer Flores Monterrey PARTICIPANTS Name Campus Ing. Irma María García Barranco CEM Lic. Gerardo Rafael Cervera Marín CSF Ing. David Sol PUE Information Systems Management Program Tecnológico de Monterrey Page 31

32 PROGRAM OBJECTIVES (OP) OP1. Alumni graduated from this program will be a professional that generates critical changes in processes and business. He or she will be able to perform management process optimization based on IT to reach better operational efficiency in industry, government or any national/international organizations. Name Responsible: Leticia Almaguer Flores leticia.almaguer@itesm.mx Co-responsibles: Carmen Celina Torres Arcadia ctorres@itesm.mx CD1. Alumni had been involved on business process optimizations activities. Evaluation Tool ISM Alumni Survey Question number Frecuency Accomplishment Semester Responsible Q14 Q16 B. Business Process desing and modeling for resources optimization K. Business Process optimization Spring Program Director Spring Program Director >= 60% >= 60% CD2. Alumni had worked in industry, government or any national/international organizations within Information Systems field. Evaluation Tool Tec Alumni Survey Question number Frecuency Accomplishment Semester Responsible II.2.a y II.2.b 5 Full time worked with incomes Sector Distribution Private + Public Spring Program Director Spring Program Director >= 80% AVG >= 80% Information Systems Management Program Tecnológico de Monterrey Page 32