AMCIS 2018 – Video Tutorials Improve Students’ Ability to Recover from Errors

Slides

Abstract

This experiment compared three forms of tutorials: text, video, and segmented videos. It asked participants to follow along with a step-by-step Excel conditional formatting tutorial, and then to transfer this knowledge to a new problem. Participants assigned to the text tutorial were much slower than those assigned the video tutorials. This performance difference was mostly due to the difficulty text participants had in detecting and recovering from errors. Error detection and recovery may explain performance differences found in earlier studies comparing text and video tutorials. Participants reported no difference in cognitive load between the three instructional formats. But, those with low pre-existing Excel skills reported higher cognitive load and made more errors on the knowledge transfer task. This study also found that self-reported Excel competency is only weakly correlated with actual performance on an Excel assessment.

Full paper

Link on main AIS website.

Online PDF Pre-print Option

How Do Academic Disciplines Use PowerPoint?

I’m happy to say that my article reviewing PPT files has been published. I used C# to automate the analysis of a 30,000 PowerPoint files from a large academic publisher. It was a fun exercise in “big” data, involving a lot of files and a lot of data cleaning.

How Do Academic Disciplines Use PowerPoint?
http://link.springer.com/article/10.1007/s10755-016-9381-8?

 

This project analyzed PowerPoint files created by an academic publisher to supplement textbooks. An automated analysis of 30,263 files revealed clear differences by disciplines. Single-paradigm “hard” disciplines used less complex writing but had more words than multi-paradigm “soft” disciplines. The “hard” disciplines also used a greater number of small graphics and fewer large ones. Disciplines identified by students as being more effective users of PowerPoint used larger images and more complex sentences than disciplines identified as being less effective in this regard. This investigation suggests that PowerPoint best practices are not universal and that we need to account for disciplinary differences when creating presentation guidelines.

http://link.springer.com/article/10.1007/s10755-016-9381-8?

Plotting Eye Tracking Data in Space-Time Cubes

My latest poster presentation will be at IEEE VIS 2017, Phoenix Arizona, October 1-6 (2017).

Abstract— It is challenging to visualize the time component of eye-tracking data.  Scanpaths can show where a single user looked, and in what order, but multiple users’ scanpaths can easily overwhelm viewers.  This paper’s approach shows larger trends without hiding short duration fixations.  Each user’s fixations are plotted in a separate space-time cube, where fixation x- & y-coordinates are plotted normally, but the z-axis is used to represent time.  The fixations are joined by a line, which is color-coded when it intersects areas of interest (AOIs).  The resulting cubes, one per user, are then placed into a 3-dimensional space side-by-side.  The result can be viewed close up to see an individual user’s gaze, or zoomed out to see larger patterns.  When viewed from above, the result looks similar to Sparklines.  This design is demonstrated on the eye movements of users watching training videos.  It is able to show patterns not visible through other techniques.

Eye Tracking Data v.2

I’ve been working on a better way to visualize data generated by my eye tracking camera. The typical approach smashes data into a single image. However, I’m really interested in seeing how eye gaze moves over time, meaning that aggregating loses the critical aspects.

The gif above shows the tool I’m currently developing.  It uses three.js to place the recorded video and image slide into a 3d space.  I then display *every* person’s gaze as a point, and update it in real-time. This also allows me to include pupil dilation, which is a key marker of cognitive load.

 

It still needs work, but is a pretty cool way of seeing how people reacted during the one-on-one experimental sessions

How Student Performance in First-Year Composition Predicts Retention and Overall Student Success


My co-authored chapter in “Retention, Persistence, and Writing Programs” titled “How Student Performance in First-Year Composition Predicts Retention and Overall Student Success” has now been published.  This was a cooperative project with Bruce Feinstein and Matthew Bridgewater.

The biggest factor we discovered was the relationship between success in writing programs and graduation.  Instead of being an isolated issue safely  hidden in GE, writing courses predict graduation just as well as courses in a student’s major.

 

The big chart for us was the following.  It shows the grade earned in a single course, separated into major classes and general education writing courses.

Viz of Eye Tracking Data

I’m working on a new way to visualize eye-tracking data. Last year, I had a number of students watch a video while my camera recorded their eyes. After exporting the raw data, I wrote a custom script with Three.JS to plot the fixations in a 3d world.

The gif to the right shows my current version. Each person’s eye track is shown as a line. As time progresses, the lines shift/move with their eye fixations.

The major benefit of this approach is that we can zoom out and see aggregate behavior fairly clearly. While we can get a similar look with the bar chart I posted earlier in the year, this way does a better job of allowing you to compare the time dimension.

If you have a Google Cardboard, the current version will allow you to see the visualization in a 3d VR world.

Textbooks for Responsible Data Analysis in Excel

My most recent paper, titled “Textbooks for Responsible Data Analysis in Excel” has been put online at the Journal of Education for Business.

Abstract: With 27 million users, Excel (Microsoft Corporation, Seattle, WA) is the most common business data analysis software. However, audits show that almost all complex spreadsheets have errors. The author examined textbooks to understand why responsible data analysis is taught. A purposeful sample of 10 textbooks was coded, and then compared against spreadsheet development best practices. The results show a wide range of approaches, and reveal that none of the 10 books fully cover the methodologies needed to create well-rounded Excel data analysts. There is a need to re-evaluate the teaching approaches being used in office application courses.