Challenge Problem: Scoring and Configuration
How the scoring is done in challenge problem?
During the contest, the master judge returns the score for only a certain percentage (20% is the current number) of the total uploaded test files. However, the test case judge runs for all the test files and computes the scores for all test files. The master judge saves them internally. It does not show this score to the user. Also, if there is any error (WA, TLE, RTE) in the solution against any test file, including the files for which the score is not shown, the judge should return the error to the user.
After the contest is over, the total score averaged over all the test files (including the remaining 80%), is shown to the user after doing a quick rejudge for the problem (which will only copy the pre competed score and not judge all the solutions once again).
What is the change?
We have simplified the process for this 80%-20% scoring in the challenge problem. Now we just need to pass the test file number from the "auxiliary field" to the master judge. The master judge will return score for only those files. Scores for other files will be computed but not shown during the contest.
Previously, we used to do a hard rejudge of all the AC submissions on challenge problem due to which many of the solutions turned out to be TLE due to high load and resource contention on the judge. Now onwards, we will be saving the score of all the testcases (during the contest) in a temporary table and will swap the scores from this temporary table (after the contest). Thus saving us time and the TLE issue mentioned above will be addressed.
Once the contest is over, just by clicking Quick Rejudge button, the scores will get updated.
How to configure a Challenge problem from now on?
There are two phases now:
The settings mentioned below need to be configured before/during the contest:
After the contest is over, the test files' numbers mentioned in Auxiliary data passed to masterjudge should be removed. This should be done before moving the problems to practice section. This is done so that in the practice section, all the users get the score on the entire test data.
After this, as mentioned above we will need to a "Quick rejudge" for all the solutions which will copy the precomputed scores on all test files, to the main score stream and on completion of this process, the users will be able to see the final scores.
PS: Number of submissions allowed in a challenge problem in CodeChef's Long Challenge is 200