Testing
Want to test problems created for CodeChef contests? We need a bunch of hardcore problem solvers to solve and scour our new problems for errors. Our fearless Problem Testers certify the quality of problems before the rest of the programming world can even have a look at them! We currently need Problem Testers for our Long Contests and our Cook-Off's. If you think you have what it takes, read the rules and apply to become a Problem Tester.
Roles & Responsibilities
  • For every problem set you test, you must provide:
    • Solutions to all the problems with appropriate comments for understanding.
    • A list of suggestions to improve the set of Test Cases.
    • Verify and suggest appropriate Time Limits for test cases such that solutions in various programming languages should pass.
    • Suggest modifications to the constraints on the input.
    • Your opinion about the actual difficulty levels of the problems, which will be verified and approved by the CodeChef Admin.
    • Make sure that problems of right difficulty levels and topics for a set are used. Suggest changes to the problems or replacement of a problem to have appropriate difficulty levels and varied topics.
  • No cheating! You are not allowed to participate in a contest that includes any of the problems you test. Also, you are not allowed to share any information about the problems or answers with anyone.
  • We're serious. Before you can test any problem, we require that you sign a Non-Disclosure Agreement saying that you won't cheat.
  • Tester will be the key person responsible for the contest. Nothing goes on a live contest without the setter consulting the tester.
  • Not all changes are to be done by the tester. He has to identify the correction and get it done by the setter. In case of a setter failing to do so and time not permitting, the tester will have to escalate the same to the CodeChef admin immediately and may make the correction himself.
  • For the contest that you test, you must be available during the contest to resolve any issue pertaining to any of the problems of that contest. You will have to help to reply / moderate / approve the comments on the problems. For short contests, you have to be online during the whole duration of the contest. For Long contest, you have to login every day to approve/ reply comments.
  • For the contest that you test, you must also answer the questions or any queries pertaining to the problems that you have tested after the end of the contest on the problem discussion forum. Most of the questions are asked till 1-2 weeks after the contest ends.
  • The tester will have to verify the editorials for the correctness, before the editorialist posts it.
  • Another key part of the testing is, testing the tiebreaker problem in long contests. The new scoring mechanism is implemented as under:
    • For 20% of the test cases you upload the correct judging program. For the other 80% of the test cases you upload the same judging program modified as follows: instead of outputting the correct score, it outputs a score of 0.0 (but it still checks for correctness, so that if the result is Wrong answer, then it produces Wrong answer as the verdict for that test case, but if the result is Correct answer it produces a score of 0 for that test case).
    • After the contest is finished, the judge program for these 80% test cases is replaced with the correct judge program (the one producing the correct score) and all the solutions needs to be re-judged.
  • The tester has to make sure that the total number of test files for a problem should be kept to minimum (preferably not exceeding 10 for the hardest problem and not exceeding the number of subtasks for the easy ones). If needed, more test data can be added to the existing test files. (The reason being the judge get overloaded if there are more test files, it overloads the judge.)
  • Under the exceptional situation of your tested problem having identified to have weak test data after the contest, you have to add additional test data to improve the problem in the practice section.
Frequency and Schedule
If you want to test for any of our contests, please let us know well in advance as per the following schedule:
LONG CONTEST
90 days before the contest
COOK-OFF
60 days before the contest
LUNCHTIME
60 days before the contest
Compensation
We pay our Testers as per the following structure:
LONG CONTEST
USD 500 for the whole set
COOK-OFF
USD 200 for the whole set
LUNCHTIME
USD 200 for the whole set