Page tree
Skip to end of metadata
Go to start of metadata

The question type programming interaction is a special input interaction. If participants enter a program code, this code will be evaluated either by defined test cases or a template with a corresponding return value. The automatic assessment is done using an interface for the server application   VPL . Supported programming languages include C, C++, Haskell, Java, Mathlab/Octave and Phython.


 This question type is a paid function and only available with a licence for the BASIC or PRO version of the ONYX Testsuite.

Content

Question

FunctionDescription

Question

Questions come with a WYSIWYG editor. This allows you to design your questions to your liking and include media and formulas.

Score

By default, the maximum score for the question will be displayed. To change this setting, switch to the Scoring tab.

Answers

Answering a programming interaction is done by entering a program code. To do this, the participant has access to a code editor with syntax highlighting. The author must choose the desired programming language and can enable an automatic assessment.

FunctionDescription

Auto Assessment

By enabling this option, the programming interaction will be automatically assessed. You can choose whether the assessment should be done using Test cases or the Return value of a template. If automatic assessment is not enabled, the question will be marked for the manual assessment, and the participant will receive a corresponding notification during the test or in the results view.

Programming language

Select the expected programming language for entering and evaluating the question. Supported programming languages include C, C++, Haskell, Java, Mathlab/Octave and Phython.

Scoring

If automatic assessment is enabled, you can choose from two scoring options:

  • Test cases:  The scoring will be based on the number of correct test cases. The participant receives percentages of the maximum score per correct test case.
  • Return value of template: The scoring will be performed using a template, with the program code returning a percentage value as console output ("Grade: = >> 100").

Test cases

If the option Test cases has been selected for scoring, a corresponding input field will be displayed. Test cases must be entered in VPL syntax. Each test case must be described as follows:

  • Case = case name
    Defines the start of a (new) test case. The case name helps to interpret the results report and understand which test cases have been successful and which have failed.
  • Input = text (optional)
    Defines the input for the program code. The input can span several lines and ends with the next VPL expression (e. g. output).
  • Output = text
    Defines the expected output for the program code. The output can span several lines and ends with the next VPL expression (e. g. output).

Any number of test cases can be defined. The scoring is done according to the number of correct test cases. For the program code written by the participant, is checked whether the defined output will be reached using the corresponding input. The number of successful test cases in relation to the total number of test cases results in the score, calculated as a percentage of the maximum score defined by the author. 

The VPL documentation gives an overview of the VPL syntax and shows more examples in the section Creating automated tests.

Template

Regardless of the selected scoring option, the author may define a program code as a template in which the program code entered by the participant will be included. The placeholder {LEARNERRESPONSE} defines the exact point at which the participant input will be included. If no template has been defined, the program code of the participant will be executed on its own.

The template can be used in addition to the scoring. If the scoring option Return value of template has been configured, the program code must return a percentage value as a console output. The output must have the following format:

  • Grade: =>> 100
    The scoring (grade) is indicated as a percentage. The score will be calculated as a percentage of the maximum score defined by the author.

Solution

The hint at the solution will be displayed to the participant as a sample solution, for example, while running a test for which solutions have been enabled or in the results view after completing a test.

More information

Code editor

For entering the program code, both the participant and the author have access to a special code editor based on the ACE editor (https://ace.c9.io/). In particular, this enables an improved overview due to syntax highlighting, specific indentation and outdentation, and program code-specific operator support. The editor is preconfigured according to the selected programming language.

Results report (VPL output)

The results data includes the full VPL output as a text file. This file will be stored in the data archive for each test run and participant. In the future, testers may look forward to a direct output display in the results views.