Once you have collected Submissions, you may want to Review them to streamline the process of deciding which to Accept/Reject. Reviews on Submissions can be collected when Reviewers fill in their responses on a Review Form (designed by you). Custom Review Forms can be built by adding Review Criteria and Criteria weights to calculate an overall Review Score. Review Scores provide a simple metric to compare Submissions with, helping to eliminate bias.

In this article you will learn how to:

1. Create a Review Form
2. Adjust Review Criteria Options
3. Calculate Review Scores
4. Modify your Review Form

Create a Review Form

You can decide how you would like Submissions to be rated, by adding custom Review Criteria of your choice. The label of each Review Criterium will be the question asked to the Reviewer, and their ratings and comments will be used to provide an overall evaluation of each Submission they have been assigned. This article provides more information on Assigning Reviews. Review Forms must be added to the Call Template under the Review Phases Column.

To follow the steps in this article you will need to have a Pre-existing Call. Let's begin by adding a Review Phase that will hold your Review Form:

You are now entering the Review Form Editor. This controls the Form that your Reviewers will fill in to leave their Reviews on Submissions. You can add Review Criteria either to a specific Submission Field, a Section or to the Submission Form as a whole (hence the different positioning of the Add Review Criterium buttons shown below). This allows you to capture very specific and detailed comments/ratings from your Reviewers, exactly where Submitters fill in their responses.

Let's take a look at a few basic Review Criteria configurations:

This will attach a Review Criterium specifically to the Submission Form Field it has been added to. In our example above, we can see that we will be adding this Review Criterium to the Subject Area Form Field.

The Yes/No Review Criterium is handy for questions that only require a Yes or No response from Reviewers. In the example below we can see that the Subject Line will be the question asked to Reviewers for this Criterium. If you would like this question to be optional for Reviewers, tick the Optional checkbox. On the right you have the additional options to delete this added Review Criterium, or re-position it higher or lower on your Review Form. Weights will be explained below.

This will allow you to collect any final Review comments at the end of your Form.

For Non-numerical Review Criteria that require text inputs, you can require a minimum or maximum amount of words/characters for this Field.

*Note: Your Review Form is not fully saved yet.

You have now successfully created a Review Form and added it to your Call Template. Find out more about the different Review Criteria you can add to your Form and the Review Phases in these respective articles.

How are Review Scores Calculated?

Overall Review Scores are calculated as a weighted average of the Numerical Review Criteria multiplied by their weights. Numerical Criteria refer to the Review Criteria that include numerical values such as Sliders, Single-Option Fields (Dropdown and Yes/No) and Multi-option Fields (Check-boxes). For each Field that can have a numerical value selected, we calculate the score based on the rating system added to this Field and the value selected by the Reviewer. Each Field’s value is mapped into a number between 0-1 based on the Field’s configuration. This will be explained in more detail below.

Numerical Review Criteria

Numerical Review Criteria include all the Review Fields that use numbers to provide a rating. Let's take a closer look at the Numerical Field options available and how they are calculated:

Sliders

Sliders values are projected into a (min)-(max) function to be converted to a value between 0 and 1. In the example below we can see that with a Slider that has the values 4 to 18, if a Reviewer selects 11 the score of this Field would be calculated as (11-4)/(18-4). This number will be multiplied by the weight of 3 to contribute to the overall Review Score.

Single-option Based Numerical Fields

These include Autocomplete, Radiogroup, Dropdown and Yes/No Fields. When calculating the value of a Single-option Based Field, the selected Field options are projected into a (min)-(max) function. In the example below with the values of Option A (score 4), Option B (score 6), and Option C (score 13), if Option B is selected the score for this Review Field will be calculated as (6-min(4,6,13))/(max(4,6,13)-min(4,6,13)). The calculated number will then be multiplied by the weight of 4 to contribute to the overall Review Score.

Multi-option Based Numerical Fields: Checkboxes

These include Checkbox Fields. When calculating the value of a Multi-option Based Numerical Field, the score is calculated using the sum of all options as the maximum. In the example below with the Checkbox options: Option A (score 3), Option B (score 9), and Option C (score 15), if Options A and B are selected the score for this Review Field will be calculated as (3+9-min(3,9,15))/(SUM(3,9,15)-min(3,9,15)). The calculated number will then be multiplied by the weight of 2 to contribute to the overall Review Score.

Weights

Weights are used to decide how much each review question should be taken into account when calculating the overall review score.

If certain Review Criteria are more important that others for example, these should be given bigger weights. Review Criteria combined with their weights help calculate an overall Review Score that provide an easy metric to compare Submissions with.

*Tip: As a best practice when translating a paper-based Review Form into a Slayte Review Form, it works best to assign to each Criterium the weight of each Criterium’s maximum value. This synchronizes the values on the slider/dropdown with their impact to the overall score. In the example below we can see that this slider has a scale between 0 to 10, so the weight of 10 has been added to synchronize this Review Criterium with the overall score.

Negative Weights

Review Criteria can also have negative weights, which count as 'penalty scores.' The overall Review Score however will always remain between 0 and 1. Setting a negative weight on a Review Criterium will make it so that the maximum selected value for that field will be the biggest penalty, i.e. equal to the associated weight.

Reviewers cannot see the weights of Review Criteria and can also not see which criteria have negative weights. A weight of 0 means that the field does not contribute to the overall Review Score (not commonly used).

Non-numerical Review Criteria

Non-numerical Criteria such as Review Comment and Static Text (that only require textual inputs) will not be included in the overall Review Score. Comments left by Reviewers on a Submission may help to provide textual reasoning that help evaluate Submissions, however these will not be included when calculating the overall Review Score.

For Submissions that will be reviewed by multiple Reviewers, the overall Review Score will be calculated as an average between the combined Review Scores provided by each Reviewer.

Optional Fields

If a Numeric Review Criterium is optional, then it only contributes to the score if a value is selected by the Reviewer. In the example below, we can see that Slider A has the Rating Scale (0 -10) with a weight of 4 and Slider B has a Rating Scale (0-10) with a weight of 6, but Slider B is optional.

In Scenario A, a Reviewer chooses the Slider value of 4 for Slider A and 6 for Slider B. This Review Score will be calculated as follows:

Review 1 A:4 + B:6 => ((4-0)/(10-0)*4+(6-0)/(10-0)*6)/(4+6)= 52%

In Scenario B a Reviewer chooses the value 4 for Slider A, and does not choose a value for Slider B (as it is optional) this Review Score will be calculated as:

Review 2 A:4 + no value selected for B => ((4-0)/(10-0)*4)/(4)= 40%