Runner Host Library

Python Host

The Scorers

This module provides all the types of Scorer classes.

A Scorer is specialized to evaluate one aspect of the submission data. Each homework assignment provides the script to create and glue one or more scorers toegether, to give the final score of a submission.

You may refer to HwScore to see how scores from multiple scorers are composed up. Also, you may refer to Python Judging to check the usage of all Python scorers.

class pyhost.scorer.BlackBoxScorerMaker(schema, csvdata, input_class_weight=0.6, boundary_value_weight=0.4)

A factory to create both InputClassScorer and BoundaryValueScorer. You may refer to Input Judging for examples.

Parameters:

• schema (CsvSchema) – The schema for this scorer to parse csv data.
• csvdata (object) – Iterable object over str, each representing a row in the csv data. Usually a file object.
• input_class_weight (float) – The weight for InputClassScorer.
• boundary_value_weight (float) – The weight for BoundaryValueScorer.

boundary(description)

Get the decorator to a BoundaryValueScorer validator.

class_(description)

Get the decorator to a InputClassScorer validator.

get_scorers(weight=1.0)

Get the list of InputClassScorer and BoundaryValueScorer which can be appended to scorer list and fed into SafeRunner.run.

Parameters:weight (float) – The total weight of two scorers. Returns:[(Scorer, weight), ...].

class pyhost.scorer.BoundaryValueScorer(schema, csvdata, check_classes=None)

A InputDataScorer called BoundaryValue Scorer. This class distinguishes from InputClassScorer only on the name.

class pyhost.scorer.CodeStyleScorer(filelist, skipfile=None, errcost=10.0)

The scorer to give a score according to the coding style.

The most well-known coding style in Python world is probably pep8. This scorer will evaluate the student code with pep8.py, and will decrease errcost from the score for each warning. The minimum score will be 0.

Parameters:

• filelist – Iterable names of files that should be checked.
• skipfile – A callable() object to filter the filelist. We only leave the names where skipfile(name) returns False.
• errcost – The cost of each coding style error.

static FromHandinDir(ignore_files=None)

Create a new CodeStyleScorer for all files under the submission working directory unless they belong to ignore_files.

Parameters:ignore_files – A collection of file names to be ignored.

class pyhost.scorer.CoverageScorer(suite, filelist, stmt_weight=0.5, branch_weight=0.5)

The scorer to give score according to the coverage rate.

The coverage rate is a very important score when it comes to unit testing. We may provide a module, tell the students to write unit testing code, and give them score according to the coverage rate of their testing code on our module.

This scorer measures two types of coverage rate: the statement coverage rate and the branch coverage rate. Each type of coverage rate will result in a score, and the final score will be composed up according to stmt_weight and branch_weight.

Parameters:

• suite – A TestSuite or a lazy loader to generate the TestSuite object.
• filelist – Iterable names of files that should be tested by the students. The coverage rate will be measured on these files.
• stmt_weight (float) – The weight of statement coverage rate.
• branch_weight (float) – The weight of branch coverage rate.

static FromHandinDir(files_to_cover, test_pattern='test_.*\.py$', stmt_weight=0.5, branch_weight=0.5)

Create a new CoverageScorer.

The files for the students to test is provided in files_to_cover, and the file name patterns of unit testing code is defined in test_pattern. The stmt_weight and the branch_weight may also be specified.

Parameters:

• files_to_cover (list of str) – List of files to measure the coverage rate on.
• test_pattern (str) – The regex pattern of testing code file names.
• stmt_weight (float) – The weight of statement coverage rate.
• branch_weight (float) – The weight of branch coverage rate.

class pyhost.scorer.InputClassScorer(schema, csvdata, check_classes=None)

A InputDataScorer called InputClass Scorer. This class distinguishes from BoundaryValueScorer only on the name.

class pyhost.scorer.InputDataScorer(name, schema, csvdata, check_classes=None)

The base class for input data scorers.

Input data scorers are mainly used in BlackBox testing homework. The students may provide some structured data in CSV format, and we want to check whether the data covers all the equivalent classes and boundary values.

To achieve that goal, I introduced InputDataScorer. It can takes a set of methods as condition validators, where each methods only returns True on a certain class of input data or boundary value.

Then the student submitted data will be checked by all the condition validators one row after another. The scorer will count the number of validators who have ever reported True, and give the score according to that portion.

Parameters:

• name (GetTestString) – The name of this scorer, should be set by derived classes.
• schema (CsvSchema) – The schema for this scorer to parse csv data.
• csvdata (object) – Iterable object over str, each representing a row in the csv data. Usually a file object.
• check_classes (list of callable() objects) – The initial list of input data validators.

empty()

Whether or not this scorer has no input data validator?

getDescription(check_class)

Get the description for given validator.

The description attribute of the given validator, or __name__ if description doesn’t exist, or the string representation of the given validator if __name__ doesn’t exist either.

Parameters:check_class – The callable() data validator. Returns:The description of given validator.

rule(description)

Make a decorator to callable() objects which will add description attribute to the given method, and will add that method to check_classes.

Usage:

@scorer.rule('a >= 1 and b <= 2')
def a_must_not_less_than_1_and_b_must_not_greater_than_2(a, b):
    return a >= 1 and b <= 2

Note

If the check method raises any exception, this rule will be regarded as not matched. This design purpose is to ease the rules like matching a string that can be converted into int.

Returns:The decorator.

check_classes = None

The input data validators.

csvdata = None

Rows of csv data.

schema = None

Store the CsvSchema.

class pyhost.scorer.ObjSchemaScorer(schema)

In unit testing homework, we want the students to organize their testing code in the exact way we tell them. So we need to validate the structure of objects. ObjSchemaScorer can parse object structure and give the score.

Parameters:schema (RootSchema) – A collection of object structure validators.

class pyhost.scorer.Scorer(name)

The base class for all scorer types.

Parameters:name (GetTextString) – The translated name of this scorer.

do_run()

Derived classes should implement this to do actual evaluations.

run()

Run the evaluation and give the score.

If a ScorerFailure is raised, the score and explanation carried by the error will be copied into this scorer, as the final result.

Other exceptions are not catched by this method. We may just leave those errors not processed, so that the runner host will exit with a non-zero exit code, which will force the submission to be rejected.

brief = None

The brief explanation of the score, a translated string. (GetTextString)

detail = None

Detailed explanation of the score, a list of translated strings. You may initialize the list yourself in do_run(). (list of GetTextString)

name = None

Store the translated name of this scorer. This name will be displayed to the students in detailed submission report.

score = None

The final score given by this scorer.

Derived class should set this attribute in do_run() to give the score (where the value should fall in the range [0, 100]).

time = None

The time consumed by this scorer. This attribute may keep None if the scorer fails to run.

class pyhost.scorer.UnitTestScorer(suite)

The scorer to give score according to a unit testing.

Suppose the student has submitted some code, and we want to evaluate the functionality of the code. We can provide our testing suite in the homework evaluation script, expecting the student code to pass these tests.

Then we may feed the testing suite into UnitTestScorer. This scorer will run all the testing cases, and give the score according to the amount of passing cases.

Parameters:suite – A TestSuite or a lazy loader to generate the TestSuite object.

static FromHandinDir(test_pattern='test_.*\.py$')

Create a new UnitTestScorer on the given files. The unittest.suite.TestSuite objects will be discovered and loaded from the files matching the given pattern when do_run() is called.

Parameters:test_pattern (str) – The regex pattern of file names.

static FromNames(names)

Create a new UnitTestScorer on the given names. The testing cases specified by names will not be loaded until do_run() is called.

Parameters:names (list of str) – A sequence of object names that can be resolved into a unittest.suite.TestSuite.

static FromTestCase(testcase)

Create a new UnitTestScorer on the testing case class. This method makes a lazy loader so that the testing case object will not be constructed until do_run() is called.

Parameters:testcase – A class derived from unittest.TestCase.

suite = None

Store the testing suite. If it is a callable() object but not a #: TestSuite, execute it to get the real testing suite. Keep the suite lazy can prevent the scorer from exploits.

pyhost.scorer.load_suite(suite)

If the given testing suite is a callable() object and is not a TestSuite, execute this method as if is a lazy loader to generate the testing suite.

Parameters:suite – A testing suite or a lazy loader.

Object Schema

This module provides the utilities to define the rules of object structures, and to validate the student submissions using these rules.

class pyhost.objschema.BaseSchema(parent, pattern)

The base class for all kinds of schemas.

A schema is a set of rules that defines the structure of particular objects. It takes a regular expression as pattern to match the objects, and check the object according the defined rules.

The basic rule of any schema is the existence of object. The object may be marked as REQUIRE, ALLOW or DENY. If not defined, default rule is ALLOW.

Parameters:

• parent (object) – The parent object whose members will be validated by this schema.
• pattern (str or SRE_Pattern) – The name pattern to filter members in the parent. The matched elements will be validated by this schema.

allow()

Mark this schema as ALLOW, and return this schema itself.

check(collector)

Check all the rules from this schema to its children recursively, and put the results in collector.

check_children(collector)

If this schema exists, check all children.

check_require(collector)

Check whether this schema is marked as REQUIRE but the object does not exist. All the children schema will also be notified.

check_self(collector)

Check all the rules of this schema.

Derived classes should update exist value, indicating whether the object of this schema exists.

deny()

Mark this schema as DENY, and return this schema itself.

get_description()

Get a one line description of this schema.

print_tree(pad=0)

Recursively print the schema tree.

require()

Mark this schema as REQUIRE, and return this schema itself.

children = None

Store the schemas who takes the items in objects as parents (or, the children of this schema).

exist = None

Whether any object matched by this schema exists?

exist_rule = None

The rule of existence for objects matched by this schema.

objects = None

Store the objects matched by this schema.

parent = None

The parent object whose members will be validated by this schema.

pattern = None

The name pattern to filter members in the parent.

class pyhost.objschema.ClassSchema(parent, pattern)

The object structure rules for the classes belong to a parent module.

Parameters:

• parent – The parent ModuleSchema
• pattern – The name pattern to match member classes.

method(pattern)

Add a MethodSchema to this ClassSchema.

Parameters:pattern (str or SRE_Pattern) – The name pattern to match the methods. Returns:This ClassSchema instance itself.

class pyhost.objschema.MethodSchema(parent, pattern)

The object structure rules for the methods belong to a parent class.

Parameters:

• parent – The parent ClassSchema
• pattern – The name pattern to match member methods.

class pyhost.objschema.ModuleSchema(parent, pattern)

The object structure rules for a module.

Parameters:

• parent – The parent RootSchema.
• pattern – The module name.

class_(pattern)

Add a ClassSchema to this ModuleSchema.

Parameters:pattern (str or SRE_Pattern) – The name pattern to match the classes. Returns:This ModuleSchema instance itself.

class pyhost.objschema.RootSchema(rootpath)

The root schema that manages a set of file schema instances.

Parameters:rootpath (str) – The root path of the files matched by this schema.

module(pattern)

Add a ModuleSchema to this RootSchema.

Parameters:pattern (str) – The full name of module. Returns:This RootSchema instance itself.

class pyhost.objschema.SchemaExistRule

The existence rule of objects.

ALLOW = 1

The related object is allowed to exist.

DENY = 2

The related object is not allowed to exist.

REQUIRE = 0

The related object is required to exist.

class pyhost.objschema.SchemaResultCollector

The collector of the results of schema. Detailed messages of the schema should be LazyString instances.

addError(msg)

Add an error check point to this result collector.

addSuccess()

Add a success check point to this result collector.

clear()

Clear all the statistics of this result set.

Submission Failure

exception pyhost.errors.ScorerFailure(brief, score=0.0, detail=None)

Stop the execution of a Scorer and set the score, the brief explanation and the detail explanation.

Parameters:

• brief (GetTextString) – The brief explanation.
• score (float) – The final score given by this error.
• detail (list of GetTextString) – Detailed explanation of the score, a list of translated strings.

brief = None

The brief explanation of the score, a translated string. (GetTextString)

detail = None

Detailed explanation of the score, a list of translated string. (list of GetTextString)

score = None

The final score given by this error.

Other Utilities

class pyhost.utility.Pep8DetailReport(options)

The helper class to gather pep8 coding style evaluation results.

build_report()

Generate the detail explanation.

Returns:A list of GetTextString.

class pyhost.utility.UnitTestScorerDetailResult

The helper class to gather unit testing results.

details = None

The translated detail explanation is gathered in this list.

pyhost.utility.format_exeception()

Format the exception traceback.

Returns:The traceback string.