What is the primary goal of data analysis?

Summarizing and interpreting data

What is the difference between mean and median?

Mean is the average; median is the middle value.

Mean and median are used interchangeably.

Mean is the middle value; median is the average.

What does the term "outlier" mean in data analysis?

Unusual or extreme values in a dataset

The last value in a sorted dataset

The most common value in a dataset

The difference between mean and median

What is the purpose of correlation in data analysis?

Measuring the strength and direction of a linear relationship

Data Analysis and Data Analytics

Introduction to IoT Analytics IoT data and Big data - IoT Analytics Lifecycle and Techniques- IoT Data Collection, IoT Data Analysis, IoT Data Deployment, Operationalization, and Reuse- Defining IoT analytics and challenges.

Chapter 7 - Review Data and Decision Making *Glow bus due at midnight, name and student number: answer questions using content in class People have created wonderful things for centuries, and management Management can be traced as far back as 500 bc when the ancient Sumerians used written records to improve government and business activities Why is it important to lean from the past Not to repeat our mistakes Classical management approaches Scientific management Administrative Principles Bureaucratic organisation Behavioural Management Approaches Follett’s Organizations as communities The Hawthorne studies Maslow’s theory of human needs Mcgregor’s Theory x and Theory Y Argyris Personality and organisation Modern Management foundations Organises as systems Contingency thinking Quality management Quantitative and analysis and tools Evidence-based management Contributions Frederick Taylor - Father of Scientific management He noticed that workers often did their jobs with wasted motions and without a constant approach. His resulted in inefficiency and low performance He believed the problem could be fixed if workers were taught to do their jobs in the best ways and ten were helped and guided by supervisors Four guiding principles of scientific management Rules of motion, standardized work and proper working conditions Select workers with the right abilities Train workers and give them incentives Support workers by planning and smoothing the way as they do their work Frank and Lillian Gilbreth Pioneered use of motitono studies as a management tool In one famous case, the gilbreaths cut down the number of motions used by bricklayers adn tripled their productivity Contributions from scientific management Make results-based compensation a performance incentive Carefully design jobs with efficient work methods Carefully select workers with the ability to perform the job Trian workers to execute activities to the best of their abilities Train supervisors to support workers so they can perform jobs to the best of their abilities Classical Management Adiminstative principle (Henro Fayol) 1919, after a career in French industry, Henri F published “adminisration Industrielle et Generale” (General and industrial management) in which we out like his views on the management of organiztion and workers Rules and duties in management Foresight - to complete a plan of action for the future Organization - To provide and mobilize resources to implement the plan Common- to lead, select and evaluate workers to get the best work toward the plan Coordination- to fit diverse efforts together and ensure information is shared and problems solved Control- to make sure things happen according to plan and to take necessary corrective action Classical management Bureacratic organiztion (Max Weber) Max weber (Bureaucrativ organization) - late 19th century German political economist who had a major impact in the fields of management and sociology Bureaucratic Organization An ideal, intentionally rational adn very efficient form of organization Based on the principles of logic, order and legitimate authority Characteristics of BO Clear division of labour Clear hierarchy of authority Formal rules and procedure Impersonality Careers based on merit What are some disadvantages of bureaucracy Takes a long time for problems to become solved bec there are procedures and there is a chain of people in command Having the power Rules have to follow Excessive paperwork or “red tape” Slowness in handling problems Rigidity in the face of shifting needs Resistance to change Employee apathy Behavioural Management Approaches (focus on understanding the elements that affect human behaviour in organisations) Follett’s Organizations as communites Mary park follett contributed to the transition from classical thinking inot behavioural management Groups and human cooperation Groups allow individuales too combine their talents for a greater good Organizations are cooperating “communites” of managers adn workers Managers job is to help people copperate and achive an integration of goals and intrests Forward-looking managment insight: Making every emploee an owner creates a sense of collective responsibility Prescursor of employrr ownership, profit sharing and gain sharing Buniess problems invovle a varity of inter realted factors Prescursor of systems thinking Private profits realtive to public good Precursor of managerial ethics and social respinsibility Hawthorne studies Took place at western electric chicago plan, a tran led by Harvards Elton Mayo set out to learn how econmic incentives and workplace conditions affected workers output Maing objective Intial study examined how ecomoin incentives adn physical conditions affected worker output (productivity) No consistent relationship found During experientmetn they had 2 groups The expertiant groups (impoved wokring ocnditions ) The control group ( no changes to original working conidtions) No consitant relationship found, perfomance in both groups increased even after removing incentives Social setting and human relations Concluded New “social setting” led workers to do good job Good “Human relations” = higher productivity The contect - The Great Depression (1929-1940) Employee attitudes and groups processes Osme thinsf satisifed some workers but not others People resticited output to adhere to groups norms (Avoid layoffs) Lessons from he hawthrone stufirs Social and human concerns are keys to prductivity Hawthrone effect - People who are singled out for special attention perform as expected Maslow’s Theory of human needs Human needs The work of psychologist Abraham Maslow in the area if human “needs,” also has had a major impact in the behavioual apporach to management Maslow’s hierarchy of human needs Self actualization needs Higherst level: need foe self fulfillment to grow and use abilites to fullest and most creative extent Esteem needs Needs fro esteem in eyes of others need for respect, prestige, recognition; need for self esteem, personal sense of competence, mastery Social needs Need for love, affection, sense of belongingness in ones relationship either other people Safett needs Need for security, protection and stability in teh events of day to day life Physiological needs Most basic of all human needs: need for biological maintence; food, water and phydical well being Principles Defict principle: A satidifed need is not a motivator of behaviour Progress principles: A need becomes a motivator once the preceding lower-level need is satisfied Both principles cease to operate at self actulilzation level McGregor’s Theories Thepry x assumes that workers; Dislike work Lack ambition Are irresponsible Resist change Prefer to be led Theoyry y assumes that workers are Willing to work Willing to accept responsibility Capable of self control Capable of self direction Imaginative and creative According to McGregor, Managers create: Self fulfilling prophecies Implications of Theory x and y Theory x managers: Create situations where workers become dependent, passive and reluctant Theory y managers create situations where workers respond with initiative and high performance Central to notions of empowerment and self management Argyris’s theory of adult personality Classical management principles and practices inhibit worker maturation and are inconsistent with the mature adult personality Management practices should accommodate the mature personality: Increasing task responsibility Increasing task variety Using participative decision making Modern Management Foundation Quantitative analysis and Tools Analytics: the use of large data bases and mathematics to solve problems and make informed decision using systematic analysis Organization as systems System Collection of interrelated parts that function together to achieve a common purpose Subsystem A smaller component of a larger system Open systems Organisations that interact with their environment Contingency thinking Tires to maths managerial responses with problem (situation) No “one best way” to manage The “appropriate way to to manage depends on the situations Quality management Qality anc competitive advantafe are linked Total quality managment (TQM) Comprehensive approach to contiou impovment on teh entire organization ISO certification Gloval quality management standards Refine and upgrade quality to meet ISO requirments Evidednce Based Managment Making management decision on “hard facts” about what really works

Why and How Managers Plan Importance of planning The planing process Benefits of planning Planning and time management Types of PLans used by managers Long term and short term plans Strageic and tactical plans Operational plans Planning Tools and Techiqunes Forecasting Contrigency planning Scenario planning Benchmaking Use of staff planners Implementing Plans to Achive Results Goal setting Goal management Goal alignment Participation and involvement Planning Def: The process of setting objectives and determining how best to accomplish them Planning at Eaton Corporation “Making the hard decision before events force them upon you, an anticipating the future needs of the market before the demand asset itself Objectives and goals Identifity the specific results or desired outcomes that one intends to achieve Plan Def: A statement of action steps to be taken in order to accomplish the objectives (goals) Steps in the planning process: Define your objectives Determine where you stand vis-a-vis objectives Develpo premises reagrdsing future conditions Analyze alternatives and make a plan Implement the plan and evaluate results What are the benefits of planning Improves focus and flexibility Imporves action orteitation Imporves coordination and control Imporves time management Time Managment Personal time management tips Do say “no” to request that distract you form what you should be doing Dont get bogged down inn details that can be addressed later Do screen telephone calls, emails and meeting request Dont let drop in visitors, text messaging use up your time Do prioritize your important and urgent work Dont become calendar bound by letting other control your schedule Do follow priorities; do most important and urgent work first Some 77% of mangers in one survey said that digital age has increased th number of decisions they have to make 43% said there was less time available to make these decisions Types of plans used by Managers What is teh time horizon Long term vs Short term Long term Look three or more years into teh future Short term plans Typically cover one year or less However: the increasing environmental complexity and dynamism of recent years has severely tested the concept of “long-term” planning Plans are subject to frequent revisions Most executives would likely agree that these complexities adn uncertainties challenge how er actually go about planning and how far ahead we can really plan At the very least we can conclude that there is a lot less permanency to long term plans today and that tey are subject to frequent revision Managment reaeracher Eillot Jaques believes tha people vary in their capability to think with different time horizons Types of Plans used by Managers (3 of 5) Strategic plans Set broad, comprehensive and linger term action directions for teh entire organization or major division Vision Clarifies purpose of the organization and what it hopes to be on the future Typical plans Specify how the organizations resources are used to implement strategy Tactical plans in business often take the form of functional plans Functional plans Incidate how different component within the organiztion will help accompnlish the overall strategy Production plans Finacial plans Facilites Plans Logisitc plans Marketing plans Human Resource Plans Operation plans Describe short-term activities to implement strategic plans Policies: Are standing plans that communicate guidelines for decisions Ex: Policies on office romances: The media is quick to report when a top executive or public figures runs into trouble over an office affair. Are there ant policies on office romances? Employer polices on office raltioshiis vary. One survey find teh following: 24% prohibit relationships among employees in the same department 13% prohibit relationships among employees who have the smae supervisor 80% prohibit relationships between supervisors and subordinates 5% have no restrictions on office romances Procedures: Are rules that describe actions to be taken in specific situations Budgets: are single use plans that commit resources to projects or activities Zero based budgets: allocate resources as if each budget were brand new There is no guarantee that any past funding will be renwer. All propsales, old and new, must compete for available funds at teh start of each new budget cycle Forcasting Attempts to predict the future Qualitaive forecasting uses expert opinions Quantitative forecasting uses mathematical models and statiscal aanylsis of historical data dna surveys Contingency planning Identify alternative course of action to take when things go wrong Anticipate changing conditions Contain trigger points to indicate when to activate plan (or a specific course of action) Scenario planning A long term version of contingency planning Identifying alternative future scenarios Plans made for each future scenario Increases organizations flexibility and preparation for future shocks Benchmarking Use of external and internal comparisons to better evaluate current performance Adopting best practices: things people adn organization do that lead to superior performance Staff Planners Experts who assist in all steps of the planning process They help bring focus and expertise to a wide variety of planning tasks Important: Communication between staff planers landline managers is essential for teh success of teh planning process Goal Setting - Always set SMART goal The solution: Goal Aligment Between Team Leader and Team Member Jonintly plan: Set objectives, set standards, choose actions Individually acy: Perform tasks (member), provide support (leader) Jointly control: Review results, discuss implications, renew cycle x4 Collective effort and commitment Participatroy planning Includes in all planning steps that people who will be affected by the plans adn askedd to help implement them Unloacks motivational potential of goal setting Management by objective (MBO) promotes participation Participation increases understanding and acceptance of plan and commitment to success Participatory planning - Number of people involved in teh decision making process Amazon is intensely focused on what it does. It believes in creating tight single-threaded teams, also known as “2 pizza team.” Data and Decision Making What are some of the important competencies managers must have today? Delegate Marketing and technology Manager must have Technological competency Ability to understand new technologies and to use them to their best advantage Information competency Ability to locate, gather, organize and display information for decision-making and problem solving Analytical competency Ability to evaluate and analyze information to make actual decisions and solve real problems What is the difference between Data and Information Data Raw facts and observation Information Data made useful and meaningful for decision-making Important concepts Big data Exists in huge quantities and is difficult to process without sophisticated mathematical and analytical techniques Data production today Bernard Marr is an internationally best-selling author. He helps organizations improve their business performance, use data more intelligently Data mining The process of analyzing data to produce useful information for decision-makers Management Analytics The systematic evaluation and analysis of data to make informed decision Information drives management Bad Data Refers to information that can be erroneous, misleading, and without general formatting The challenge: Can er use the data that is available in the “Big Data” Needs to be valid Can not trust everything out there Being ethical Look at the trends Data is structured and unstructured Data BIg Data = Structured + Unstructured Information Drive Management decision making What are the characteristics of useful information Easy to access If its credible Accurate Characteristics of useful information: Timely High quality Complete Relevant Understandable What about bad data It's not credible Miss information If it is not structured/ organized Bias based on opinions Confusing If its updated Bad data Refers to information that can be erroneous miss What are some examples of Management information system Business intelligence -BI Information systems to extract and report data in organized ways that are useful to decision-makers Executive dashboards Visually update and display key performance metrics (or Key Performance Indicators -KPIs) and information on a real-time basis Information needs in organization External Environment Information exchanges with the external environment Gather intelligence information Provide public information Information needs within the organizations (internal Enviroement) Information exchange within the organization Facilitate decision making Facilitate problem-solving Managers as information processors Continually gather, share and receive information Now as much electronic as it is face-to-face Always on, always connected How many people telecommute at least once a week 70% of people globally work remotely at least once a week, Work at home after covid 19 our forecast Our best estimate it that 25-30% of the workforce will be working form home multiple days a week by the end of 2021 As of 2023, 12.7% of full time employees work from home, while 28.2% work a hybrid model Managers as problem solvers Problem-solving The process of identifying a discrepancy between actual and desired performance and taking action to resolve it Ishikawa Fishbone diagram To identify the cause of problems Decision A choice among possible alternative courses of action Performance threat Something is wrong or has the potential to go wrong Performance opportunity The situation offers the chance for a better future if the right steps are taken Problem-solving approaches or style - from textbook Problem avoiders Inactive in information gathering and solving problems Problem seekers Proactive in anticipation of problems and opportunities and taking appropriate action to gain an advantage Problem solvers Reactive in gathering information and solving problem Managers - can approach problems in a systematic or intuitive manner Systematic thinking approaches problem in rational, step-by-step and analytical fashion Intuitive thinking approaches problems in a flexible and spontaneous fashion Multidimensional thinking- applies both intuitive and systematic thinking Managers face structured and unstructured problems Structure problems Are ones that are familiar, straight forward, and clear with respect to information needs Program decisions apply solutions that are readily available from past experiences to solve structured problems Know how to solve them Familiar Know what we are dealing with Unstructured problems Are ones that are full of ambiguities and information deficiencies Nonprogrammed decisions apply a specific solution to meet the demands of a unique problem Commonly faced by higher-level management Crisis decision making A crisis involves an unexpected problem that can lead to disaster if not resolved quickly and appropriately Ruled for crisis management Figure out what is going on Remember that speed matters Remember that slow counts, too Respect the danger of the unfamiliar Value the skeptic Be ready to “fight fire with fire” Managers make decisions with various amounts of information Certain environment Offers complete information on possible action alternatives and their consequences Risk environment Lacks complete information but offers probabilities of the likely outcomes for possible action alternatives Uncertain environment Lacks so much information that it is difficult to assign probabilities to the likely outcomes of alternative Ex: Certain and uncertain environments: The worldwide Governance Indicators for over 200 countries, comparing distinct environments (Canada-Brazil) Step 1-Identify and define the problem Focuses on information gathering information processing and deliberation Decision objectives should be established What are some common mistakes in definding problems? Common mistakes in defining problems Defining the problem too broadly or too narrowly Focusing on symptoms instead of causes Choosing the wrong problem to deal with Step 2- Generate and Evaluate Alternative Courses of Action Potential solutions are formulated and more information is gathered, data are analyzed, the advantages and disadvantages of alternative solutions are identified Common mistakes: Abandoning the search for alternatives too quickly Step 3- Decide on a preferred course of Action Two different approaches Behavioural model leads to satisficing decisions Classical model les to optimising decisions Behavioural Model Rationality is bounded because: There are limits our thinks capacity Available information (incomplete) Time constraints Step 4-Implement the decision Involves taking action to make sure the solution decided upon becomes a reality Managers need to have the willingness and ability to implement action plans Problems: Lack of participation error should be avoided Step 5 - Evaluate Results Involves comparing actual and desired results The positive and negative consequences of the chosen course of action should be examined If actual results fall short desire results, the manager returns to earlier steps in the decision-making process At all steps, check ethical reasoning Ask these spotlight questions Utility Does teh decision satisfy all constituents or stakeholders Rights Does the description respect the rights and duties of everyone? Justice Is the decision consistent with the canons of justice Caring Is the decision consistent with my responsibilities to care? Issues in decision-making How do errors happen? Heuristics: are strategies for simplifying decision-making Availability Bias: Bases a decision on recent information or events Representativeness bias: Bases a decision on similarity to other situations Anchoring and Adjustment Bias: Bases a decision on incremental adjustment from a prior decision point Framing error: Tring to solve a problem in the context perceived, positive or negative Confirmation Error: Focusing on information that confirms a decision already made Escalating commitment: Continuing a course of action even though it is not working Creative Decision making Creativity is the generation of a novel idea or unique approach that solves a problem or crafts an opportunity Big C: Creativity occurs when extraordinary things are done by exceptional people Little C: Creativity occurs when average people come up with unique ways to deal with daily events and situations The three types of situational creativity drivers Chapter review What are objectives and goals? The specific results or desired outcomes What are the 5 characteristics of great (SMART) goals? Forecasting - Attempts Qualitative forecasting uses options Quantitative forecasting uses mathematical models and statistical analysis of historical data and surveys Scenarios-Oracle’s crystal ball combines qualitative and quantitative methods

The following terms will be encountered in the lesson:  Research data- is any information that has been collected, observed, generated, or created to validate a research study.  Data analysis- a process that involves examining, and molding collected data for interpretation to discover relevant information, draw or propose conclusions, and support decision-making to solve a research problem.  Data Interpretation- is the process of making sense of numerical data that has been collected, analyzed, and presented.  A Conceptual framework is an analytical tool that is used to get a comprehensive understanding of a phenomenon. It can be in different fields of work and is most commonly used to visually explain the key concepts or variables and the relationships between them that need to be studied.

What is a rubric? A tool comprising a set of criteria (with possible levels of performance quality on the criteria) developed to assess learners’ work, from written to oral to visual. It is used tomeasureperformance,suchastheprocess of doing something (e.g.,playing a musical instrument, making a speech) or products of the learners’ work (e.g., concept map, laboratory report, bookshelf) (Brookhart, 2013). BENEFITS OF USING RUBRICS Support authentic assessment Reflects how well learners are able to apply knowledge inthe real-world context. Communicate expectations Gives learners an idea of what is expected of them. It is especially useful when the rubrics are communicated to the learners before they are assessed. Improve performance Explicit criteria and performance level descriptions allow learners to understand the desired performance. Learners are able to assess themselves by referring to the specific criteria and performance-level descriptions. Provide informative feedback Instructors are able to provide constructive feedback to learners on their weaknesses and strengths. Promote thinking andlearning 4 Provide informative feedback Instructors are able to provide constructive feedback to learners on their weaknesses and strengths. Learners are able to review and revise their work,thus reflecting on their learning experiences. Ensure fairness Learner performance assessed fairly given its objectivity. It helps avoid disputes between learners and instructors about the scores/grades achieved. TYPES OF RUBRIC ANALYTIC It consists of individual criterion with corresponding descriptor of performance. HOLISTIC It consists of performance descriptors that are placed together to refeclet to overalll performance. ANATOMY OF ANANALYTIC RUBRIC Rating scales with corresponding scores or weights The row represents the criteria for the desired performance, while the column represents the evaluation score. Under the rating scale (corresponding weights orscorescanbeassigned),theperformance descriptors are explicitly stated ANATOMYOF AHOLISTICRUBRIC Descriptions: It comprises the rating scale (corresponding weights or scores can be assigned) in the row while the combined desired performance descriptors are placed in the column. Description of the task The purpose of the assignment is to assess learner’s cognitive and analytic skills in applying knowledge gained and constructed throughout the course Diffusion of Innovation,bywatching the Surrogates movieand writing ananalytical review of the movie in the context of innovation diffusion.Iwant to provide learners with informative feedback on their cognitive and analytic skills such as the following: applying the concepts of innovation diffusion,making judgmentson the scenes related to innovation diffusion identified from the movie,selecting and critiquing theories of innovation diffusion and making connections between the theories,aswell asarguingand proposing necessary solutions to the problemss hown in the movie. ESTABLISHING ALTERNATIVEASSESSMENTINHIGHEREDUCATION VALIDITYAND RELIABILITYOF RUBRICS. Validity Measuring what is supossedto be measured. Reability Yielding consists results. Instruments that are used in the alternative assessment must be aligned to the learning outcomes and measure well what it intends to measure (valid) and produce consistent scores (reliable). The valid instrument will manifest the true ability (latent trait) of learners and permit appropriate inferences to be made about a specific group of people for specific purposes. TYPES OF VALIDITY FACE VALIDITY Simple form of validity thatapplies a superficial and subjective assessment whether the instrument measures what it is supposed to measure. CONTENT VALIDITY Refers to the extent to which the items on a measure assess the same content or how wellthe content material was sampled inthe measure. CONSTRUCT VALIDITY Refers to the extent to which the test may be said to measure a theoretical construct or trait. CONCURRENT VALIDITY Refers to the extent to which scores onanewmeasure are related to scores from a criterion measure administered at the same time. PREDICTIVE VALIDITY Refers to the uses of the scores from the new measure to predict performance on a criterion measure administered ata later time. STEPS TO CONSIDER WHEN ESTABLISHING CONTENT VALIDITY Calculate the level of expert agreeement for the content validity, get expert to verfy. Interview the expert ,make meta contentdata análisis from literatura. STEPS TO CONSIDER WHEN ESTABLISHING CONSTRUCCT VALIDITY Administer the instrument for alll learners, revise any item necccesay, run an apropriates statistical analiysis, administerthe instrument to learners as a pilot test . CONSTRUCTMAP Morepreciseconceptthan construct. Ranges from one extreme to another(fromhightolow,small tolarge,positivetonegative,or strongtoweak). Identifiesthepositionofthe respondentsinthisrange. Representativenessofsampling (questions and ability of respondents). EXAMPLEO FACONSTRUCTMAP:AFFECTIVE LEVELOF AFFECTIVE VARIABLES EXAMPLESOFITEMSIN MEASURINGTEAM WORKING SKILLS 5. Characterisation Learnersvolunteerstodothe groupworks. 4. Organisation Learners are willing to help others,althoughitisnottheir scopeoftask. 3. Valuing Learners respect other team members’opinionwhendoing thediscussion. 2. Responding Learnergivescooperationwhen neededingroupworks. 1. Receiving Learneracceptsthediversityof races and nationalities among groupmembers. EXAMPLEOFACONSTRUCTMAP:PSYCHOMOTOR LEVELOF PSYCHOMOTOR VARIABLES EXAMPLESOFITEMSIN MEASURING DIGITAL SKILLS 7.Origination Learnerscanmodifytheirowndevicesto performbetter. 6.Adaptation Learnerscansolveandtroubleshootthe problemwhileusingthecomputer. 5.ComplexOvertResponse Learnerscanusethecomputercompetently. 4.Mechanism Learners can use the computer independently,butstillmakeminorerrors. 3.GuidedResponses Learnerscanusethecomputer,butstill needguidance. 2.Set Learnersarereadytousethecomputer. 1.Perception Learnerscanobservehowtousecomputer. EXAMPLEOFACONSTRUCTMAP:COGNITIVE LEVELOF COGNITIV E VARIABLES EXAMPLESOFITEMS IN MEASURING THINKINGSKILLS 6. Creating Learners are able to suggest anewmodelorframeworkof learningdigitalcommunity. 5. Evaluating Learners are able to judge the impactofthescenariotowards educationperspective. 4. Analysing Learnerscandifferentiate the factsusingafew theories. 3. Applying Learnerscansolveproblems usingthefactsgiven. 2. Understanding Learnersareabletoexplainthe factsusingtheirownwords. 1. Remembering Learnersonlymemorisethe. Direction of Increasing “X” Learners Learners with high “X” Learners with mid range “X” Learners with low “X” Responses to Item Item response indicate highest level of X Item response indicate higher level of X Item response indicate lower level of X The construct map shows the lower ability students are in line with the lower level of items. This shows that when educators plan to develop an instrument, it Item response indicate lowest level of X Direction of Decreasing “X” is crucial to create an item difficulty thatrepresents learners’ ability. Learners’ ability Learners who engage in level characterisation Learners who engage in level organisation Learners who engage in level valuing Learners who engage in level responding Learners who engage in level receiving Direction of Decreasing“X” MEASURINGCONSTRUCTVALIDITY Unlike content validity, this construct validity can be analysed using statistical analysis. Use Exploratory FactorAnalysis [EFA], Confirmatory FactorAnalysis [CFA] or Unidimensionality to confirm all items are measuring the right construct and the raw variance explained for the latent variables is sufficient. Gap initem map also can show accuracy in construct validity. RELIABILITY The degree to which test scores are consistent over repeated administrations of the same/ equivalent test and therefore considered dependable and repeatable for an individual learner.A test thatproduces highly consistent and stable results (i.e. relative free from random error) is said to be highly reliable. TYPESOFRELIABILITY Test-retest demonstrates the stability of a measure over time 01 Internal consistency most of the items within a rating scale of a concept show consistency of scoring. Inter-rater the extent to which two or more independent raters are consistent in observing, recording and scoring data (should be 70% or higher agreement) 04 Intra-rater relies on one rater to rate an object or event twice (70% or higher of agreement) FACTORSAFFECTING VALIDITYANDHOWTO INCREASEVALIDITY? FACTORS AFFECTING VALIDITY HOWTO INCREASE VALIDITY? 1. Inaccuracy of items in measuringtheoutcomes 1. Vetting session to get reviewsfromtheexpert. 2. Pooritemsdevelopment 2. Followtheformatandtips indevelopinggooditems. 3. Unclearinstructions 3. Do pilot testing to measuretheusabilityof thetest. 4. Interveningevents 4. Controltheinternalthreats validityfactors. 5. Itemsdifficultyisnot suitableforthelearners 5. Create a construct map toensurethereisanitem thatrepresentslearners ability. FACTORS AFFECTING RELIABILIT Y HOWTOINCREASERELIABILITY? 1. TestLength 1. Thetestlengthshouldbeappropriate withtestdifficulty. 2. Test retest interval 2. Suggesteddurationisbetween3 weeksto2months. 3. Variability of scores 3. Doconstructmaptoensuretheitems aresuitablewithlearners’ability. 4. Guessing 4. Penalisetheguessinganswers.You alsocandetecteitherthelearnersare guessing or not using the statistical analysis named guessing analysis andpersonfitanalysis. 5. Inconsistency score from different raters 5. Appointtheratertomarkcertain questionsforalllearners(Thisalways happen when you have more than onesectionandhavemorethanone lecturer). CONCLUSION Coming back to the issue of validity and reliability in assessment, there is a need for educators to put an effort to ensurethattheitemsintheformofquestionsorinstructions arenotonlyclearbutalsoabletomeasurewhatitisintended tomeasurebasedontherelatedlearningoutcomes. Establishingvalidityandreliabilityofinstrumentscan provide educators with some indications of the quality of the measuring tools being used. Valid and reliable instruments enabletheeducatorstocontinuouslyusethemeasuringtools withoutreservation. Reliablenot valid Precisenot Accurate Reliableand valid Preciseand Accurate NotReliable butvalid NotPrecisebut Accurate NotReliable butNotvalid NotPrecisebut NotAccurate 94

We found you were making a quiz on the subject of "What is a rubric? A tool comprising a set of criteria (with possible levels of performance quality on the criteria) developed to assess learners’ work, from written to oral to visual. It is used tomeasureperformance,suchastheprocess of doing something (e.g.,playing a musical instrument, making a speech) or products of the learners’ work (e.g., concept map, laboratory report, bookshelf) (Brookhart, 2013). BENEFITS OF USING RUBRICS Support authentic assessment Reflects how well learners are able to apply knowledge inthe real-world context. Communicate expectations Gives learners an idea of what is expected of them. It is especially useful when the rubrics are communicated to the learners before they are assessed. Improve performance Explicit criteria and performance level descriptions allow learners to understand the desired performance. Learners are able to assess themselves by referring to the specific criteria and performance-level descriptions. Provide informative feedback Instructors are able to provide constructive feedback to learners on their weaknesses and strengths. Promote thinking andlearning 4 Provide informative feedback Instructors are able to provide constructive feedback to learners on their weaknesses and strengths. Learners are able to review and revise their work,thus reflecting on their learning experiences. Ensure fairness Learner performance assessed fairly given its objectivity. It helps avoid disputes between learners and instructors about the scores/grades achieved. TYPES OF RUBRIC ANALYTIC It consists of individual criterion with corresponding descriptor of performance. HOLISTIC It consists of performance descriptors that are placed together to refeclet to overalll performance. ANATOMY OF ANANALYTIC RUBRIC Rating scales with corresponding scores or weights The row represents the criteria for the desired performance, while the column represents the evaluation score. Under the rating scale (corresponding weights orscorescanbeassigned),theperformance descriptors are explicitly stated ANATOMYOF AHOLISTICRUBRIC Descriptions: It comprises the rating scale (corresponding weights or scores can be assigned) in the row while the combined desired performance descriptors are placed in the column. Description of the task The purpose of the assignment is to assess learner’s cognitive and analytic skills in applying knowledge gained and constructed throughout the course Diffusion of Innovation,bywatching the Surrogates movieand writing ananalytical review of the movie in the context of innovation diffusion.Iwant to provide learners with informative feedback on their cognitive and analytic skills such as the following: applying the concepts of innovation diffusion,making judgmentson the scenes related to innovation diffusion identified from the movie,selecting and critiquing theories of innovation diffusion and making connections between the theories,aswell asarguingand proposing necessary solutions to the problemss hown in the movie. ESTABLISHING ALTERNATIVEASSESSMENTINHIGHEREDUCATION VALIDITYAND RELIABILITYOF RUBRICS. Validity Measuring what is supossedto be measured. Reability Yielding consists results. Instruments that are used in the alternative assessment must be aligned to the learning outcomes and measure well what it intends to measure (valid) and produce consistent scores (reliable). The valid instrument will manifest the true ability (latent trait) of learners and permit appropriate inferences to be made about a specific group of people for specific purposes. TYPES OF VALIDITY FACE VALIDITY Simple form of validity thatapplies a superficial and subjective assessment whether the instrument measures what it is supposed to measure. CONTENT VALIDITY Refers to the extent to which the items on a measure assess the same content or how wellthe content material was sampled inthe measure. CONSTRUCT VALIDITY Refers to the extent to which the test may be said to measure a theoretical construct or trait. CONCURRENT VALIDITY Refers to the extent to which scores onanewmeasure are related to scores from a criterion measure administered at the same time. PREDICTIVE VALIDITY Refers to the uses of the scores from the new measure to predict performance on a criterion measure administered ata later time. STEPS TO CONSIDER WHEN ESTABLISHING CONTENT VALIDITY Calculate the level of expert agreeement for the content validity, get expert to verfy. Interview the expert ,make meta contentdata análisis from literatura. STEPS TO CONSIDER WHEN ESTABLISHING CONSTRUCCT VALIDITY Administer the instrument for alll learners, revise any item necccesay, run an apropriates statistical analiysis, administerthe instrument to learners as a pilot test . CONSTRUCTMAP Morepreciseconceptthan construct. Ranges from one extreme to another(fromhightolow,small tolarge,positivetonegative,or strongtoweak). Identifiesthepositionofthe respondentsinthisrange. Representativenessofsampling (questions and ability of respondents). EXAMPLEO FACONSTRUCTMAP:AFFECTIVE LEVELOF AFFECTIVE VARIABLES EXAMPLESOFITEMSIN MEASURINGTEAM WORKING SKILLS 5. Characterisation Learnersvolunteerstodothe groupworks. 4. Organisation Learners are willing to help others,althoughitisnottheir scopeoftask. 3. Valuing Learners respect other team members’opinionwhendoing thediscussion. 2. Responding Learnergivescooperationwhen neededingroupworks. 1. Receiving Learneracceptsthediversityof races and nationalities among groupmembers. EXAMPLEOFACONSTRUCTMAP:PSYCHOMOTOR LEVELOF PSYCHOMOTOR VARIABLES EXAMPLESOFITEMSIN MEASURING DIGITAL SKILLS 7.Origination Learnerscanmodifytheirowndevicesto performbetter. 6.Adaptation Learnerscansolveandtroubleshootthe problemwhileusingthecomputer. 5.ComplexOvertResponse Learnerscanusethecomputercompetently. 4.Mechanism Learners can use the computer independently,butstillmakeminorerrors. 3.GuidedResponses Learnerscanusethecomputer,butstill needguidance. 2.Set Learnersarereadytousethecomputer. 1.Perception Learnerscanobservehowtousecomputer. EXAMPLEOFACONSTRUCTMAP:COGNITIVE LEVELOF COGNITIV E VARIABLES EXAMPLESOFITEMS IN MEASURING THINKINGSKILLS 6. Creating Learners are able to suggest anewmodelorframeworkof learningdigitalcommunity. 5. Evaluating Learners are able to judge the impactofthescenariotowards educationperspective. 4. Analysing Learnerscandifferentiate the factsusingafew theories. 3. Applying Learnerscansolveproblems usingthefactsgiven. 2. Understanding Learnersareabletoexplainthe factsusingtheirownwords. 1. Remembering Learnersonlymemorisethe. Direction of Increasing “X” Learners Learners with high “X” Learners with mid range “X” Learners with low “X” Responses to Item Item response indicate highest level of X Item response indicate higher level of X Item response indicate lower level of X The construct map shows the lower ability students are in line with the lower level of items. This shows that when educators plan to develop an instrument, it Item response indicate lowest level of X Direction of Decreasing “X” is crucial to create an item difficulty thatrepresents learners’ ability. Learners’ ability Learners who engage in level characterisation Learners who engage in level organisation Learners who engage in level valuing Learners who engage in level responding Learners who engage in level receiving Direction of Decreasing“X” MEASURINGCONSTRUCTVALIDITY Unlike content validity, this construct validity can be analysed using statistical analysis. Use Exploratory FactorAnalysis [EFA], Confirmatory FactorAnalysis [CFA] or Unidimensionality to confirm all items are measuring the right construct and the raw variance explained for the latent variables is sufficient. Gap initem map also can show accuracy in construct validity. RELIABILITY The degree to which test scores are consistent over repeated administrations of the same/ equivalent test and therefore considered dependable and repeatable for an individual learner.A test thatproduces highly consistent and stable results (i.e. relative free from random error) is said to be highly reliable. TYPESOFRELIABILITY Test-retest demonstrates the stability of a measure over time 01 Internal consistency most of the items within a rating scale of a concept show consistency of scoring. Inter-rater the extent to which two or more independent raters are consistent in observing, recording and scoring data (should be 70% or higher agreement) 04 Intra-rater relies on one rater to rate an object or event twice (70% or higher of agreement) FACTORSAFFECTING VALIDITYANDHOWTO INCREASEVALIDITY? FACTORS AFFECTING VALIDITY HOWTO INCREASE VALIDITY? 1. Inaccuracy of items in measuringtheoutcomes 1. Vetting session to get reviewsfromtheexpert. 2. Pooritemsdevelopment 2. Followtheformatandtips indevelopinggooditems. 3. Unclearinstructions 3. Do pilot testing to measuretheusabilityof thetest. 4. Interveningevents 4. Controltheinternalthreats validityfactors. 5. Itemsdifficultyisnot suitableforthelearners 5. Create a construct map toensurethereisanitem thatrepresentslearners ability. FACTORS AFFECTING RELIABILIT Y HOWTOINCREASERELIABILITY? 1. TestLength 1. Thetestlengthshouldbeappropriate withtestdifficulty. 2. Test retest interval 2. Suggesteddurationisbetween3 weeksto2months. 3. Variability of scores 3. Doconstructmaptoensuretheitems aresuitablewithlearners’ability. 4. Guessing 4. Penalisetheguessinganswers.You alsocandetecteitherthelearnersare guessing or not using the statistical analysis named guessing analysis andpersonfitanalysis. 5. Inconsistency score from different raters 5. Appointtheratertomarkcertain questionsforalllearners(Thisalways happen when you have more than onesectionandhavemorethanone lecturer). CONCLUSION Coming back to the issue of validity and reliability in assessment, there is a need for educators to put an effort to ensurethattheitemsintheformofquestionsorinstructions arenotonlyclearbutalsoabletomeasurewhatitisintended tomeasurebasedontherelatedlearningoutcomes. Establishingvalidityandreliabilityofinstrumentscan provide educators with some indications of the quality of the measuring tools being used. Valid and reliable instruments enabletheeducatorstocontinuouslyusethemeasuringtools withoutreservation. Reliablenot valid Precisenot Accurate Reliableand valid Preciseand Accurate NotReliable butvalid NotPrecisebut Accurate NotReliable butNotvalid NotPrecisebut NotAccurate 94 ". Would you like to continue making it or start afresh?

Data Analysis and Financial Literacy

Related quizzes

Related quizzes