|
Context/Needs Assessment |
Inputs/ Processes |
Intermediate Goals/Outputs |
Outcomes |
|
Over the past several years, Norfolk State University and other institutions nationwide have noted challenges to improving student success and retention.
Data collected by ACT show that up to one-fourth of all students at four-year colleges do not return for their second year of school. According to the National Center for Education Statistics, dropout rates are particularly high for African American and Hispanic students (ACT, 2005). The freshman to sophomore year retention rate for NSU has fluctuated between 63 and 71% over the past five years. |
Create an interactive assurance of learning networked environment that not only encourages intellectual engagement and active participation from all students during lectures and discussions, but also allows instructors the capability to monitor student progress, conduct real-time assessment, and provide immediate feedback and assistance to assure student learning.
Evaluated by the instructional redesign of three management information systems courses which will follow these constructivist theoretical recommendations: • student controls learning process • learning is embedded in complex, problem-based, real-world tasks • learning environment is open and flexible • assessment is continuous and embedded in learning tasks |
· Increase in the intellectual engagement of students by instructors. · Increase in collaborative learning among students. · Increase in real-time assessment and in immediate feedback/assistance to students. These objectives will be operationalized by the increase in student participation during class time and measured by the increase in the quantity and quality of interactions between students and faculty. An instrument will be developed which tracks the number of interactions, as well as the quality of the interactions. A rubric will be attached to the instrument to insure consistency of the rating of the quality of the interactions. The use of HP Photosmart digital camera to document sample evidence of such interactions is planned. Another way these objectives will be operationalized is by students' increased understanding of the course learning outcomes and measured by adopting a Pre-Test/Post-Test Value-Added Approach. Assessment tests will be created for the three redesigned courses and will be given to the students during the first and last week of classes. Once the pre-test/post-test data are collected, statistical analyses are to be conducted to determine if a positive and significant difference exists between pre- and post-tests. These results will be compared with those given to the control group to assess the effectiveness of the proposed course redesign.
To assess the effectiveness of real-time assessment of student learning and immediate feedback/assistance, a questionnaire will be designed to measure the impact of this objective from the students' perspective. The results of this survey will be compared against the control group results.
|
Our students will benefit by gaining access to wireless networked, collaborative learning environments very similar to the modern corporate environment models used in a global society. These enhanced learning environments made possible by HP Technology for Teaching Grant Initiative will increase the probability of student success and retention.
Evaluated by improvements in students success and retention as measured by surveys, post-tests and retention rates. |
|
Courses |
Baseline (Fall 2008) |
Instructional Delivery Modifications (I.D.M.) (Spring 2009) |
I.D.M. with Pen-based Technologies (Fall 2009) |
|
MIS 284 (so.) |
Variables11 |
Variables12 |
Variables13 |
|
MIS 390 (jr.) |
Variables21 |
Variables22 |
Variables23 |
|
MIS 410 (sr.) |
Variables31 |
Variables32 |
Variables33 |
The impact of this project on student learning has been overwhelming. The research team has tracked student progress by using a value-added approach: pre-tests and post-tests on the main student learning outcomes per course were given at the beginning and ending of each semester. The results were analyzed using t-tests for the sample difference.
T-Test Analysis of the Difference between Pre-test vs. Post-test Results in MIS 284 during Fall 2008_Baseline
|
t-Test: Two-Sample Assuming Equal Variances |
||
|
|
|
|
|
|
Pre |
Post |
|
Mean |
41.36842 |
61 |
|
Variance |
113.8012 |
56.2 |
|
Observations |
19 |
11 |
|
Pooled Variance |
93.22932 |
|
|
Hypothesized Mean Difference |
0 |
|
|
df |
28 |
|
|
t Stat |
-5.3665 |
|
|
P(T<=t) one-tail |
5.1E-06 |
|
|
t Critical one-tail |
1.701131 |
|
|
P(T<=t) two-tail |
1.02E-05 |
|
|
t Critical two-tail |
2.048407 |
|
The difference was statistically significant at better than alpha = .05
T-Test Analysis of the Difference between Pre-test vs. Post-test Results in MIS 410 during Fall 2008_Baseline
|
t-Test: Two-Sample Assuming Equal Variances |
||
|
|
|
|
|
|
Pre |
Post |
|
Mean |
48 |
65 |
|
Variance |
36.44444 |
35.77778 |
|
Observations |
10 |
10 |
|
Pooled Variance |
36.11111 |
|
|
Hypothesized Mean Difference |
0 |
|
|
df |
18 |
|
|
t Stat |
-6.32577 |
|
|
P(T<=t) one-tail |
2.91E-06 |
|
|
t Critical one-tail |
1.734064 |
|
|
P(T<=t) two-tail |
5.82E-06 |
|
|
t Critical two-tail |
2.100922 |
|
The difference was statistically significant at better than alpha = .05
T-Test Analysis of the Difference between Pre-test vs. Post-test Results in MIS 284 during Spring 2009 Course_Modified
|
t-Test: Two-Sample Assuming Equal Variances |
||
|
|
|
|
|
|
Pre |
Post |
|
Mean |
23.77143 |
81.43333 |
|
Variance |
282.0639 |
243.8402 |
|
Observations |
35 |
30 |
|
Pooled Variance |
264.4689 |
|
|
Hypothesized Mean Difference |
0 |
|
|
df |
63 |
|
|
t Stat |
-14.2508 |
|
|
P(T<=t) one-tail |
1.11E-21 |
|
|
t Critical one-tail |
1.669402 |
|
|
P(T<=t) two-tail |
2.23E-21 |
|
|
t Critical two-tail |
1.998341 |
|
The difference was statistically significant at better than alpha = .05
T-Test Analysis of the Difference between Pre-test vs. Post-test Results in MIS 410 during Spring 2009 Course_Modified
|
t-Test: Two-Sample Assuming Equal Variances |
||
|
|
|
|
|
|
Pre |
Post |
|
Mean |
50 |
79.6 |
|
Variance |
44 |
29.15556 |
|
Observations |
7 |
10 |
|
Pooled Variance |
35.09333 |
|
|
Hypothesized Mean Difference |
0 |
|
|
df |
15 |
|
|
t Stat |
-10.1392 |
|
|
P(T<=t) one-tail |
2.08E-08 |
|
|
t Critical one-tail |
1.75305 |
|
|
P(T<=t) two-tail |
4.16E-08 |
|
|
t Critical two-tail |
2.13145 |
|
The difference was statistically significant at better than alpha = .05
The researchers also wanted to test whether there was improvement in the post-test results for both courses from one semester to the other. This would demonstrate that there was academic growth directly attributed to the course modifications introduced in the Spring 2009 Course_Modifications treatment group vis-à-vis the Fall 2008_Baseline control group. For this analysis we used the t-test with unequal variances since we were comparing two different groups, and not a pre- vs. post-test for the same group.
T-Test Analysis of the Difference between Fall 2008_Baseline Post Test vs. Spring 2009 Course_Modified Post Test Results in MIS 284
|
t-Test: Two-Sample Assuming Unequal Variances |
||
|
|
|
|
|
|
Post_08 |
Post_09 |
|
Mean |
61 |
81.43333 |
|
Variance |
56.2 |
243.8402 |
|
Observations |
11 |
30 |
|
Hypothesized Mean Difference |
0 |
|
|
df |
36 |
|
|
t Stat |
-5.6162 |
|
|
P(T<=t) one-tail |
1.13E-06 |
|
|
t Critical one-tail |
1.688298 |
|
|
P(T<=t) two-tail |
2.26E-06 |
|
|
t Critical two-tail |
2.028094 |
|
The difference was statistically significant at better than alpha = .05
T-Test Analysis of the Difference between Fall 2008_Baseline Post Test vs. Spring 2009 Course_Modified Post Test Results in MIS 410
|
t-Test: Two-Sample Assuming Unequal Variances |
||
|
|
|
|
|
|
Post_08 |
Post_09 |
|
Mean |
65 |
79.6 |
|
Variance |
35.77778 |
29.15556 |
|
Observations |
10 |
10 |
|
Hypothesized Mean Difference |
0 |
|
|
df |
18 |
|
|
t Stat |
-5.72953 |
|
|
P(T<=t) one-tail |
9.85E-06 |
|
|
t Critical one-tail |
1.734064 |
|
|
P(T<=t) two-tail |
1.97E-05 |
|
|
t Critical two-tail |
2.100922 |
|
The difference was statistically significant at better than alpha = .05
Our team has yet to include the pen-based technology treatment. We would do so this coming Fall 2009 semester. The team wanted to proceed in three stages (one per semester) in order to identify the baseline (Fall 2008) variables that would serve as control; introduce course modifications (based on Constructivist theory during the Spring 2009 semester) needed to take advantage of the technology treatment in the future (third and last stage to take place in the Fall 2009 semester.)
The basis of the instructional delivery modifications incorporated into the three courses offered in the Spring 2009 semester are grounded on Constructivist tenets such as: a) students control learning process, b) learning is embedded in complex, problem-based, real-world tasks, c) learning environment is open and flexible, and d) assessment is continuous and embedded in learning tasks.
The rubric used to assess the quality of interaction is as follows:
|
1 |
2 |
3 |
4 |
5 |
|
Student inquires about definition(s). [Recalls, recognizes, lists, identifies, labels, names, etc.] |
Student recalls definition(s) AND contributes additional insight(s). [Extends, discovers, distinguishes, etc.] |
Student recalls definition(s) and contributes additional insight(s) PLUS compares and contrasts points of view offered so far. [Connects, deduces, explains, etc.] |
Student recalls definition(s) and contributes additional insight(s) plus compares & contrasts points of view offered so far WHILE attempting to synthesize. [Integrate, combine, derive, generalize, formulate, modify, recommend, etc.] |
Student recalls definition(s) and contributes additional insight(s) plus compares & contrasts points of view offered so far WHILE successfully synthesizing them. [Integrating, combining, deriving, generalizing, formulating, modifying, recommending, etc.] |
The instrument used to gather the observational data (quality, quantity, and type of interaction per observed session) is as follows:
|
Date: |
Class: |
Type: |
Rating: |
Comments: |
|
July 18, 2008
|
MIS 410-01
|
SI (student-to-instructor)
|
4
|
Student 1 tried to generalize on the necessary conditions for an entity to be classified as "weak." The instructor had to correct a couple of his/her observations.
|
|
|
|
SS (student-to-student)
|
2
|
Student 2 properly identified definition and offered an alternate one s/he learned in another class.
|
|
|
|
... |
... |
... |
|
|
|
... |
... |
... |
We can report that the instructional delivery changes implemented in the past Spring 2009 Course_Modified semester have had a strong positive impact on student learning. The quantity and quality of student interactions have gone way up; though there is always room for improvement which we are hoping to produce and directly attribute to tablet PC-based course modifications during the Fall 2009 semester.
|
Fall 2008_Baseline |
|
|
||||
|
Count of Rating |
Course |
Interaction |
|
|
|
|
|
|
MIS 284 |
MIS 284 Total |
MIS 410 |
|
MIS 410 Total |
Grand Total |
|
Date |
SI |
|
SI |
SS |
|
|
|
8/27/2008 |
|
|
1 |
1 |
2 |
2 |
|
9/3/2008 |
2 |
2 |
|
|
|
2 |
|
9/10/2008 |
|
|
1 |
|
1 |
1 |
|
9/17/2008 |
1 |
1 |
|
|
|
1 |
|
9/29/2008 |
|
|
2 |
1 |
3 |
3 |
|
10/20/2008 |
|
|
2 |
|
2 |
2 |
|
11/3/2008 |
|
|
4 |
|
4 |
4 |
|
11/17/2008 |
6 |
6 |
|
|
|
6 |
|
11/19/2008 |
1 |
1 |
|
|
|
1 |
|
Grand Total |
10 |
10 |
10 |
2 |
12 |
22 |
Notice that the number of student-to-instructor (SI) interactions are rather low in our baseline sample for the Fall 2008 semester. (There were a couple of student-to-student (SS) interactions in the MIS 410 course.) The highest occurrence in our baseline sample is six (6) SI interactions observed during the 11/17/08 session of MIS 284.
|
Spring 2009 Course_Modified |
|
|
|
||
|
|
|
|
|
|
|
|
Count of Rating |
Course |
Interaction |
|
|
|
|
|
MIS 284 |
MIS 284 Total |
MIS 410 |
MIS 410 Total |
Grand Total |
|
Date |
SI |
|
SI |
|
|
|
1/14/2009 |
11 |
11 |
21 |
21 |
32 |
|
1/21/2009 |
9 |
9 |
8 |
8 |
17 |
|
1/23/2009 |
7 |
7 |
16 |
16 |
23 |
|
1/26/2009 |
13 |
13 |
8 |
8 |
21 |
|
1/28/2009 |
9 |
9 |
11 |
11 |
20 |
|
2/4/2009 |
6 |
6 |
12 |
12 |
18 |
|
2/6/2009 |
3 |
3 |
13 |
13 |
16 |
|
3/4/2009 |
4 |
4 |
4 |
4 |
8 |
|
3/18/2009 |
8 |
8 |
|
|
8 |
|
3/25/2009 |
6 |
6 |
8 |
8 |
14 |
|
3/27/2009 |
4 |
4 |
|
|
4 |
|
3/30/2009 |
13 |
13 |
|
|
13 |
|
4/1/2009 |
14 |
14 |
16 |
16 |
30 |
|
4/3/2009 |
|
|
15 |
15 |
15 |
|
4/8/2009 |
3 |
3 |
16 |
16 |
19 |
|
4/15/2009 |
|
|
8 |
8 |
8 |
|
4/20/2009 |
|
|
4 |
4 |
4 |
|
Grand Total |
110 |
110 |
160 |
160 |
270 |
When comparing the Spring 2009 Course_Modified results with the Fall 2008_Baseline ones, we clearly can see a huge increase in the quantity of interactions between students and instructors (SI). The course modifications done within the constructivist framework have resulted in an increase in the quantity of interactions between students and instructors. The highest level of interactions observed for MIS 284 occurred during the 4/1/09 session (=13 SI interactions), and for MIS 410 occurred during the 1/14/09 session (=21 SI interactions).
Another area where we would like to report a change between the Spring 2009 Course_Modified results with the Fall 2008_Baseline is in the area of quality of interactions. It should be noted that the quality of interactions may seem as going down to the 1s and 2s rankings in the Spring 2009 Course_Modified treatment groups as compared to the Fall 2008_Baseline courses as a percentage of the total number of observations. But if you look at the actual number of observations that were ranked in the 2s and 3s, these numbers are much larger for the Spring 2009 Course_Modified treatment group. We hypothesize that the estimate of the population proportion of 1s, 2s, and 3s rankings using the sample proportion is too dependent on small sample sizes (such as the one for the Fall 2008_Baseline control group). Nonetheless, quality of interactions is an area where the researchers would like to see an increase in the upcoming Fall 2009_Pen-Based semester.
The quality of interaction was assessed using the rubric presented above. The results of the type of interaction for the Fall 2008_Baseline are presented graphically:
The results of the quality of interactions for the Spring 2009 Course_Modified group are presented graphically.
We would like to see a corresponding increase in the quantity of student-to-student (SS) interactions. Perhaps, this will be observed during the Fall 2009_Pen-Based semester when pen-based technology course modifications are introduced.
We have not yet introduced IT-modified instructional delivery. Our experimental design calls for a three-stage process: in stage one (Fall 2008) we created the baselines and benchmarks needed as control measures; in stage two (Spring 2009) we introduced course modifications based on constructivist theory; in stage three (Fall 2009) we will introduce course modifications based on tablet PC technologies. The team felt that in this manner we would be able to isolate the effects due to constructivist theory vs. constructivist theory AND tablet PC-based course modifications. Any difference between the last two treatments would be attributed uniquely to tablet PC-based modifications.
Bonk, C. J., & Cunningham, D. J. (1998). Searching for learner-centered, constructivist, and sociocultural components of collaborative educational learning tools. In C. J. Bonk, & K. S. King (Eds.), Electronic collaborators: Learner-centered technologies for literacy, apprenticeship, and discourse (pp. 25-50). Mahwah, NJ: Lawrence Erlbaum Associates.
Johnson, D. W., & Johnson, R. T. (1975). Learning together and more: Cooperation, competition, and individualization. Englewood Cliffs, NJ: Prentice Hall.
Jonassen, D. H. (1992). Objectivism versus constructivism: Do we need a new philosophical paradigm? Educational Technology Research and Development, vol. 39(3), pp. 5-14.
Kuh, G. D., Kinzie, J., Schuh, J. H., & Whitt, E. J. (2005). Assessing conditions to enhance educational effectiveness: The inventory for student engagement and success. San Francisco: Jossey-Bass.
Kuh, George D. (2001). Assessing What Really Matters to Student Learning: Inside the National Survey of Student Engagement. Change, v33, n3, pp. 10-17.
Oliver, R., Omari, A., & Herrington, J. (1998). Exploring student interactions in collaborative World Wide Web computer-based learning environments. Journal of Educational Multimedia and Hypermedia, vol. 7(2/3) pp. 263-287.
Slavin, R. E. (1991). Synthesis of research on cooperative learning. Educational Leadership, vol. 48(5), pp. 71-82.
return to top | previous page | next page
Content ©2009. All rights reserved.
Date last modified: June 15, 2009.
Created with SoftChalk LessonBuilder
