Experiment
Outline
This experiment is composed of three modeling tasks, performed first with one tool (Gentleman or MPS), followed by the other tool. The model manipulated in those tasks represents Traffic light (TL) configurations. Before starting the tasks, participants are given hands-on tutorial, to help them familiarized themselves with the editor. At the end of the tasks, participants are invited to complete a survey. It serves to identify their profile and characterize their experience with each tool.
Tasks: (1) Design a model, (2) Evolve a model and (3) Evaluate a model.
Tutorial: Using an editor (concepts and projections) to manipulate mindmaps, define an instance, perform some editing operations (eg: select, copy-pasting, autocompletion, undo/redo).
Preliminary analysis
The experiment was conducted on two groups, differentiated by order of the tool with which they performed the tasks. To verify the influence of the order, we performed a preliminary analysis. It compares the means and variance between each group, using a confidence interval of 95%.
Metrics
Task 1: Design a model
Create a TL model with the following three lights:
Go (color
) as GREEN (name
),
Hurry as YELLOW and
Stop as RED.
Collected data
Task 2: Evolve a model
Add the two following modes to the model created in task 1.
Mode Normal
Mode Police
Collected data
Task 3: Evaluate a model
Evaluate an existing TL model using the following requirements
1. Identify the errors found in the model.
2. Apply the changes to respect the requirements.
Collected data
Feedback survey
With the survey, we've identified the participants background and collected their subjective evaluations of Gentleman and MPS.
Questions – Modeling experience
Those questions (extracted from the survey) were answered for each tool and used as metrics.