[PYTHON] [Atcoder] [C ++] I made a test automation tool that can be used during the contest

** In case of WA (wrong answer) ** ** In case of TLE (calculation time over) ** In the current setting, if ** 2 seconds ** is exceeded until the result is output, it is treated as TLE. Even if you run in an infinite loop, it will be cut off in about 3 seconds and the next test will be done. ** For CE (compile error) ** In the first place, it doesn't matter that it doesn't compile, so it ends without testing.

2.3 Automatic authentication

If you run this test during the contest, you will not be able to get the question page without logging in. By setting the user name / password etc. with the -i option, automatic authentication and test execution during the contest are enabled.

>python autojudge.py abc143 abc143_d -i
Atcoder Username:tks_fj * Atcoder account user name
Atcoder Password:******** Atcoder account password
Src Directory(Ex. ./aaa/abc140/abc140.cpp => input "./aaa"):../../02_contest
Judging abc143/abc143_d...
testcase 1: AC
testcase 2: AC
testcase 3: AC
result: AC

** This configuration information is stored in the local configuration file (setting.conf), so you only need to do this once. ** ** (This work can also be done by directly rewriting the configuration file.)

2.4 Added test case

You can also add new test cases interactively In order to support input and output of multiple lines, "quit" + Enter is used to exit.

(input:20 7 Output:Add 6)
>python autojudge.py abc143 abc143_a -a
type test input(exit by "quit")
20 7
quit
type test output(exit by "quit")
6
quit

If you test it again, it will be added to testcase4.

>python autojudge.py abc143 abc143_a
Judging abc143/abc143_a...
testcase 1: AC
testcase 2: AC
testcase 3: AC
testcase 4: AC
result: AC

** For initial settings, refer to autoJudge: Initial Settings or the next chapter. ** **

3. Design / implementation

I will also introduce the design and implementation inside, which also serves as a memorandum.

3.1 Directory structure

The following artifacts are managed in a repository called ** autoJudge **.

• ** autojudge.py (main body of the program) **
• ** setting.conf (setting file) **
• ** testcase / (test case, generated at runtime) **

It is assumed that ** autoJudge ** will be cloned in the directory where the source code (.cpp) to be tested is stored. It is recommended to operate with the following directory structure and naming convention. (But it works even if it is not ↓ (see 4. Supplementary -p option))

.
├── autoJudge * The deliverable created this time
│   ├── setting.conf * Configuration file
│   ├── autojudge.py * Program body
│ ├── testcase * List of test cases(Generated during test)
│   │   ├── abc142@abc142_a.txt
│   │   └── abc143@abc143_a.txt 
│   └── design.pu * Class diagram
├── abc142 * Contest name
│   ├── abc142_a.cpp * Problem name.cpp
│   ├── abc142_b.cpp
│   ├── abc142_c.cpp
│   ├── abc142_d.cpp
│   ├── abc142_e.cpp
│   └── abc142_f.cpp
├── abc143
:

3.2 Configuration file (setting.conf)

This is a configuration file used for authentication and searching for the source location.

setting.conf

username:tks_fj
password:*******
srcpath:../

• Enter the user name / password in username / password (required when logging in)
• Describe the relative path from autoJudge.py to "contest name / problem name" in srcpath

It can be set with the -i option introduced in ** 2.3 Automatic authentication **, but of course it is OK to rewrite directly.

3.3 Program body (autojudge.py)

3.3.1 Class

The program consists of the following two classes.

• ** ManageTestCases **: Get and manage test cases
• ** ExecuteTestCases **: Execute tests based on test cases

3.3.2 ManageTestCases: Test Management Class

• ** RegisterUser (): Editing the configuration file (setting.conf) **
• Update the configuration file by receiving the user name, password and source code path from the standard input.

autojudge.py

    def RegisterUser(self):
        """user settings(First time)"""

        print("Atcoder Username:", end="")
        username = input().rstrip('\r\n')
        print("Atcoder Password:", end="")
        password = input().rstrip('\r\n')
        print("Src Directory(Ex. ./aaa/abc140/abc140.cpp => input \"./aaa\"):", end="")
        srcpath = input().rstrip('\r\n')

        with open(CONF_FILE, "w") as f:
            f.write("username:" + username + "\n")
            f.write("password:" + password + "\n")
            f.write("srcpath:" + srcpath + "\n")

• ** GetTestCases (): Get test cases **
• Generate the URI of the question page from the contest name / question name at the first execution, get the test case and record it in the file
• Read the file and get the test case for the second and subsequent executions.
• Returns test cases and contest name / question name information (testinfo).

autojudge.py

    def GetTestCases(self, test_name, islogin = False):
        """Get test cases from the specified question name and return a list"""

        self.__UpdateConf()
        file_name = self.contest + "@" + test_name + ".txt"
        testinfo = [{"contest":self.contest, "testname":test_name}]
        #To reduce the server load, do not scrape the acquisition of the same information.
        if file_name in os.listdir(TESTCASES_PATH):
            testcases = self.__ReadFile(file_name)
        else:
            testcases = self.__ScrapePage(test_name, islogin)
            self.__WriteFile(file_name, testcases)
        return testinfo + testcases

• ** AddTestCases (): Add test cases **
• Interactively add test cases (input, output) for the specified question.
• Accept multi-line test cases (quit with "quit" + Enter)

autojudge.py

    def AddTestCases(self, test_name):
        """Add your own test case to the acquired test case"""

        self.__UpdateConf()
        testcase = {}
        print("type test input(exit by \"quit\")")
        testcase["input"] = ""
        while(1):
            line = input()
            if (line == "quit"):
                break;
            testcase["input"] += line + "\n"
        print("type test output(exit by \"quit\")")
        testcase["output"] = ""
        while(1):
            line = input()
            if (line == "quit"):
                break;
            testcase["output"] += line + "\n"
        file_name = self.contest + "@" + test_name + ".txt"
        if file_name in os.listdir(TESTCASES_PATH):
            testcases = self.__ReadFile(file_name)
        testcases.append(testcase)
        self.__WriteFile(file_name, testcases)

3.3.3 ExecuteTestCases: Test execution class

• ** Execute (): Execute test case **
• If the build (\ _ \ _ Build) passes, it will be executed (\ _ \ _ Run), and finally the result will be output (\ _ \ _ Result) (AC, WA, TLE).
• If the relative path of the source is not set, it will be automatically generated from the configuration file.
• __Run Detects TLE by starting a child process inside Run and measuring the execution time.

autojudge.py

    def Execute(self, srcpath = ""):
        """Run the test"""

        print(YELLOW + "Judging " + self.testinfo["contest"] + "/" + self.testinfo["testname"] + "..." + COLORRESET)
        if (srcpath == ""):
            srcpath = self.__GetPath()
        self.__Build(srcpath)
        if (self.result["build"] == 0):
            self.__Run()
        self.__Result()

3.4 Test case (testcase /)

• A text file is generated under the test case when the test is first executed. (contest name @ problem name.txt)
• The format is as follows

abc143@abc143_a.txt

[test case 0]
---input---
12 4
---output---
4
---fin---
[test case 1]
---input---
20 15
---output---
0
---fin---

3.5 Result display

• Display the results and overall results for each test case.
• AC is displayed in green, and WA, TLE, and CE are displayed in yellow.
• WA (Error in result) outputs prediction and result.

4. Supplement

4.1 Regarding the python environment

The following modules are required. You can put it in immediately with pip, pip3, conda, etc.

pip3 install requests
pip3 install bs4, lxml

4.2 Option list

• Add test case with -a
• Initial setting with -i (update setting file)
• Specify the source code path directly with -p

>python autojudge.py --help
usage: autojudge.py [-h] [-p PATH] [-a] [-i] contest_name question

positional arguments:
  contest_name          set contest name(ex. abc143)
  question              set question name(ex. abc143_a)

optional arguments:
  -h, --help            show this help message and exit
  -p PATH, --path PATH  set path of source code
  -a, --addtest         add testcase
  -i, --init            set configuration

4.3 Regarding path specification (-p)

When the directory structure of the source code is not "3.1 Directory structure" (For example, in the following cases)

.
├── 02_contest
│   ├── abc143
│   │   ├── abc143_a.cpp
│   │   ├── abc143_b.cpp
│   │   └── abc143_c.cpp
├── 04_autotest
│   ├── autoJudge
│   │   ├── README.md
│   │   ├── autoJudge.py
│   │   ├── autojudge.pu
│   │   ├── setting.conf
│   │   └── testcase

You can specify the source code location directly with the -p option.

>python autojudge.py abc143 abc143_a -p ../../02_contest/abc143/abc143_a.cpp
Judging abc143/abc143_a...
testcase 1: AC
testcase 2: AC
testcase 3: AC
testcase 4: AC
result: AC

4.4 Known issues

• It may be TLE at the first execution (includes scraping time ??)
• ** Not compatible with the problem that the error is below a certain level (AC) **
• Is the TLE judgment accuracy subtle?
• PermissionError is output during TLE (Win10 environment only)
• Not compatible with run-time error (RE)
• Sometimes bugs

5. Postscript

• plantUML was good (used to generate class diagrams)
• BeutifulSoup was also good (used for HTML parsing)
• argperse is also slightly bent (used for optional analysis)
• There were many pioneers
  • https://github.com/kyuridenamida/atcoder-tools
  • https://github.com/nodchip/OnlineJudgeHelper
• I want to turn blue quickly