Experiment

Outline

This experiment is composed of three modeling tasks, performed first with one tool (Gentleman or MPS), followed by the other tool. The model manipulated in those tasks represents Traffic light (TL) configurations. Before starting the tasks, participants are given hands-on tutorial, to help them familiarized themselves with the editor. At the end of the tasks, participants are invited to complete a survey. It serves to identify their profile and characterize their experience with each tool.

Tasks: (1) Design a model, (2) Evolve a model and (3) Evaluate a model.

Tutorial: Using an editor (concepts and projections) to manipulate mindmaps, define an instance, perform some editing operations (eg: select, copy-pasting, autocompletion, undo/redo).

Protocol (PDF)

Preliminary analysis

The experiment was conducted on two groups, differentiated by order of the tool with which they performed the tasks. To verify the influence of the order, we performed a preliminary analysis. It compares the means and variance between each group, using a confidence interval of 95%.

Metrics

  • NB_Selection Number of selection (a mouse click or the end of a pressed navigation key).
  • NB_Blank Number of blank selection (subset of NB_Selection).
  • NB_Miss Number of missed selection (subset of NB_Selection).
  • NB_CxtChange Number of tab or window changed in the editor
  • T_Idle Idle time. It begins after 2 seconds of inactivity.
  • NB_Idle Number of idle occurrences
  • T_QA Question and answer (QA) time. It begins after 3 seconds of interaction with the supervisor.
  • NB_QA Number of QA occurrences.
  • T_Total Total time taken to complete the task.
  • RT_Success Success rate.
  • NB_Block Number of occurrences when the participant was blocked in his task, and requested some assistance.
  • NB_Typos Number of typos (including wrong inputs).
  • NB_DesignErr Number of design errors. Occurs when the requirements of a task are not satisfied.
  • NB_Bug Number of internal errors.
  • Detection Detection rate. Ratio: total number of errors / number of errors found.
  • RT_Recover Recovery rate. Ratio: total number of errors / number of errors corrected.
  • T_Recover Total time taken to recover from all the errors following their detection.

Task 1: Design a model

Create a TL model with the following three lights: Go (color) as GREEN (name), Hurry as YELLOW and Stop as RED.

  • Stop behaviours
    • [temporal: after 15 s] --> Go
  • Hurry behaviours
    • [temporal: after 5 s] --> Stop
  • Go behaviours
    • [temporal: after 30 s] --> Hurry

Collected data

  • Gentleman CSV
    Data collected (records) during task 1 for Gentleman.
    Each record is identified with a timestamp that correspond to a participant.
  • MPS CSV
    Data collected (records) during task 1 for MPS.
    Each record is identified with a timestamp that correspond to a participant.
  • Statistical analysis PDF
    Statistical analysis (report) produced for task 1 with SPSS version 25
    • Descriptive analysis
    • ANOVA test
    • Eta-squared test

Task 2: Evolve a model

Add the two following modes to the model created in task 1.

Mode Normal

  • Lights: same as those created in task 1
  • Initial light (start): Stop
  • Behaviours
    • [manual: On signal] --> Police

Mode Police

  • Lights: ON as YELLOW and OFF as BLACK
    • ON behaviours
      • [temporal: after 2 s] --> OFF
    • OFF behaviours
      • [temporal: after 2 s] --> OFF
  • Initial light (start): ON
  • Behaviours
    • [manual: On signal] --> Normal

Collected data

  • Gentleman CSV
    Data collected (records) during task 2 for Gentleman.
    Each record is identified with a timestamp that correspond to a participant.
  • MPS CSV
    Data collected (records) during task 2 for MPS.
    Each record is identified with a timestamp that correspond to a participant.
  • Statistical analysis PDF
    Statistical analysis (report) produced for task 2 with SPSS version 25
    • Descriptive analysis
    • ANOVA test
    • Eta-squared test

Task 3: Evaluate a model

Evaluate an existing TL model using the following requirements

  • Mode: Automatic
  • Lights: GL as GREEN, YL as YELLOW and RL as RED
  • Initial light: GL
  • Allowed behaviours:
    • GL --> YL
    • YL --> RL
    • RL --> GL
  • No manual behaviours

1. Identify the errors found in the model.
2. Apply the changes to respect the requirements.

Collected data

  • Gentleman CSV
    Data collected (records) during task 3 for Gentleman.
    Each record is identified with a timestamp that correspond to a participant.
  • MPS CSV
    Data collected (records) during task 3 for MPS.
    Each record is identified with a timestamp that correspond to a participant.
  • Statistical analysis PDF
    Statistical analysis (report) produced for task 3 with SPSS version 25
    • Descriptive analysis
    • ANOVA test
    • Eta-squared test