IPynb scoring system made with TA of Introduction to Programming (Python)

This article is the 9th day article of DSL Advent Calendar 2019. Continuing from the article on the 8th day, we will talk about the lecture "Introduction to Programming" at our university.

I am doing TA in that lecture, but in short, there is homework, such as students being given weekly assignments and finishing by the next week's lecture. The TA is tasked with scoring the assignment every week. At the beginning, the content of the task is simple with only a simple program, but as the number of times increases, the content and comparison with the output become more complicated, so I made a scoring system that runs on Python.

"** Let Python do the boring things **".

Task

Questions can be answered with jupyter notebook. It is distributed in .ipynb format, and there are cells for questions and answers in it, so you can write the code there and check if the output is correct. Basically, it tells you to edit only the contents of the code cell and check the result, so you can always score by just checking the output of a specific cell.

In order to have an image below, I will put an image that easily reproduces the task.

I made the problem very simple, but it looks like this.

About scoring

Scoring criteria

First of all, about the scoring criteria. It will change depending on each task and problem, but there are three general criteria below.

1. The program does not cause an error
2. The output result is correct
3. The specified grammar or syntax is used

The first and second criteria remain the same, but the third "specified grammar and syntax" is the for and ʻif` that you learned that week. (The structure of the program is not specified every time)

Scoring method

Answers to the assignments and scoring criteria will be distributed to TA in advance. About 20 people have answered ipynb for each TA, open it on jupyter notebook, check each source code, output result and re-execution, and write the scoring result in the evaluation column. Currently, the maximum score is 4, and you can check about 4 for each, but I couldn't do this work, so I made a script. (Because the system of a certain II is not a lab but a notebook, it was hard to open a lot of tabs, wait for communication, and re-evaluate the cell)

Scoring system

Here's what to do

1. List ipynb for everyone
2. parse ipynb
3. Identify the answer cell
4. Compare the source code and output with the correct answer with the regular expression re and the evaluation ʻexec ()`
5. If the score is perfect, write the score in ipynb

I will explain each of them with the source code.

List ipynb for everyone

Use the Python standard module pathlib.

import pathlib

# path to notebook(.ipynb)
p_nb = pathlib.Path("../path/to/notebook/")

parse ipynb

Since the content of ipynb is json, you can parse it with json.load (). I also use the glob method, which handles pathlib well when looping through all of ipynb.

import json

# notebooks(.ipynb)
for _nb in p_nb.glob("*.ipynb"):
    # load json (*.ipynb)
    with open(_nb, "r")as f:
        nb = json.load(f)
        cells = nb["cells"]

I will omit the detailed explanation of the structure of ipynb, but since the cell information is as {"cell ": [cell1, cell2, cell3], ...}, nb ["cells "] You can get the list with .

Please note that the source code after this is indented one step lower, but will be omitted.

Identify the answer cell

The contents of the cell are roughly as follows.

--cell_type:'markdown','code', etc. --source: Markdown text, source code, etc. (line-by-line list) --outputs: Source code output results (including errors)

Other information is included, but the above three are used in this system.

Now, specify the cell for the answer. ** This time, the source code is written to be comparable to the ipynb given above as an example. ** ** Since it is used up to 4 times in one scoring, it is functionalized.

def identify_cell(sentence, cells, cell_type='markdown'):
    """Return the cell numbers that match the conditions from all cells"""
    for i,_cell in enumerate(cells):
        if _cell['cell_type'] == cell_type \
           and _cell['source'][0] == sentence:
            _cell_num = i
            break
    return _cell_num


# identify cell
_cn = identify_cell(sentence="##Question 1\n",
                    cells=cells)
ans_cell = cells[_cn + 1]

ʻIdentify_cell ()` is checking The cell_type is markdown, and the first line of source is "## Q1 \ n". The cell number that matches the above conditions is returned by return.

From here, it is good to check the contents of the cell while using pprint () etc.

It should be noted that it is the source code and its output result that you want to check the answer, so the next cell, _cn + 1th cell, is stored in the variable as the answer cell.

Compare source code and output with correct answer

Output the first question, "Hello World!". Let's check the correctness of the problem. This time, you only have to check if the output result is correct, so it is as follows.

try:
    result = ans_cell['outputs'][0]["text"][0]
    if result == "Hello World!":
        score += 1
except:
    pass

By the way, try-except is used because it corresponds to the case where the task is unanswered. (There is no content in outputs) It was okay to deal with it with if, but when I saw many answers, errors were thrown in various places, so I compromised with this.

In the current source code that is actually in operation, it works just by changing the path and this part every week. Answer patterns and notes are listed separately at the bottom of the article.

If the score is perfect, write the score in ipynb

Originally, you can write something other than the perfect score, but just in case you check the answer ipynb and if you have a wrong problem, I will make a separate comment, so I am using the current specifications.

if score == 2:  #Perfect score=In case of 2 points
    # identify cell
    _cn = identify_cell(sentence="##Evaluation\n",
                        cells=cells)

    #Overwrite score cell
    cells[_cn + 1] = {'cell_type': 'code',
                      'execution_count': None,
                      'metadata': {},
                      'outputs': [],
                      'source': [str(2)]}

    #Dump to ipynb for answer(Overwrite)
    json.dump(nb, _nb.open("w"))

This was provided by a person who is also doing TA in our laboratory (@y_k).

As for the contents of the program, only when the score is perfect, the cell in which the evaluation result is entered is overwritten with the perfect score and the ipynb is saved. However, if you overwrite the wrong one, it will be troublesome, so in the production environment, we also have a script to back up the original ipynb separately.

Notes

Here are some parts to keep in mind when checking the answer. In addition to the ones listed, if you are a beginner in programming, you will throw various changing balls, so it is rather difficult.

The output pattern is different between the output with print () and the output with only variables that can be used with ipython.

Even this alone is quite difficult because the contents of "outputs" are different and the program changes significantly.

When you have to evaluate a program, you can't easily get the output result by ʻexec (script)`.

This is also quite difficult, and when there are some evaluation criteria that need to be re-evaluated, I use ʻexec (script)(script is the source code of str) to check the result, butprint ()` in script is not displayed on jupyter notebook, and it is quite difficult to get the output. I wrote a program that overwrites the standard output, but I may not recommend it very much (because print () at normal time is also not displayed). I will post the source code for the time being.

import sys
import io
import contextlib

@contextlib.contextmanager
def stdoutIO(stdout=None):
    old = sys.stdout
    if stdout is None:
        stdout = io.StringIO()
    sys.stdout = stdout
    yield stdout
    sys.stdout = old


#Run
try:
    with stdoutIO() as s:
        exec(script)
    
    #Print below()Get the output of
    output = s.getvalue().split()
except:
    #In case of script Error except
    pass

Full-width half-width problem

If there are symbols or numbers in the answer that are full-width, they will not match the correct answer and will be treated incorrectly. Therefore, it will be a little easier if you convert the output result to half-width characters in advance.

def trans_hankaku(sentence: str) -> str:
    """Function to convert full-width characters to half-width characters"""
    return sentence.translate(
        str.maketrans(
            {chr(0xFF01 + i): chr(0x21 + i) for i in range(94)}
        ))

It may be easier to define a function that converts to half-width characters as described above.

Summary

This time, I was able to practice "let Python do the boring things"! (I have never read) I hope that today's article will help someone as well as yesterday's DSL Advent Calendar Day 8 article.

The source code and ipynb used this time are here. https://github.com/liseos-x140/scoring_ipynb

Afterword

The structure of ipynb generated may differ depending on the version and type of jupyter, so I would be grateful if you could use it as a reference only. Also, the source code is still in the development stage, so I hope to update it again as soon as it is organized to some extent.

reference

https://www.oreilly.co.jp/books/9784873117782/