This part of the documentation describes the common library. All of the components, including the runner, the website, and even the submitted program, can import this module.
Homework assignments in Railgun are defined by xml files. All the definition xml files along with the resources should be placed in a single directory. You may refer to Homework for more details about how to define a homework assignment.
When the Railgun system is booted, it will scan all the directories under config.HOMEWORK_DIR to load the assignments. Each homework assignment is stored in a Homework. Also, the loaded Homework instances are gathered in a global HwSet instance.
All the classes to represent the homework assignments are defined in this module. The relationships between these classes are:
In addition, the JSON objects carrying the scores for a submission used in the communication between the runner host and the website are also defined in this module. The relationships between these classes are:
You may head over to read more about these classes.
Manage a set of file rules.
A file rule is a 2-element tuple (pattern, action), where pattern determines what files to be ruled, and action determines what operations these files will take.
pattern is a regular expression. It should contain patterns to the relative paths of files.
action may be one of the following operations: ACCEPT, LOCK, HIDE and DENY.
Since the file rules are mainly used to determine what files in a homework pack can be revealed to students, and what files submitted from the students can be received.
You may refer to Archive Packing for more details about these four actions.
Indicate that the matching files are revealed to students in the homework attachment, and students can overwrite these files in their submissions.
Indicate that the matching files are revealed to students in the homework attachment, but the students cannot overwrite these files in their submissions.
Indicate that the matching files are NOT revealed to students, NOR should they been overwritten.
Indicate that the matching files are denied, and the submission containing these files will be REJECTED immediately.
Get the action to take on given file.
| Parameters: |
|
|---|---|
| Returns: | One action out of (ACCEPT, LOCK, HIDE, DENY). |
Append a file rule (action, pattern) to the end of rule list.
The rules are matched in list order. If a given file is matched by some rule in the list, all the remaining rules after it will not be checked any longer.
| Parameters: |
|
|---|
Prepend a file rule (action, pattern) to the front of rule list.
| Parameters: |
|
|---|
Return a new iterator on given file iterator, where files not taking operations from allowed_actions should be removed.
Because this method return the new file name iterator, not a new list, you must store the list somewhere if you want to use the result for more than one times, for example:
rules = FileRules()
accepted_files = rules.filter(
dirtree('.'),
(FileRules.ACCEPT, FileRules.LOCK),
)
| Parameters: |
|
|---|---|
| Returns: | The new file name iterator. |
Load file rules from an xml node object.
This method does not care about the tag name of the given node. It only parses the children of given node. Here’s an example of a file rule node:
<rules>
<accept>.*\.py$</accept>
<lock>.*\.conf$</lock>
<hide>.*\.pyc$</hide>
<deny>.*\.exe$</deny>
<rules>
| Parameters: | xmlnode (xml.etree.ElementTree.Element) – The xml node object that contains file rules. |
|---|
Object to store the homework info (name, description & solution).
The description and the solution should be written in Markdown syntax. There are some extensions to the original Markdown standard. Refer to Writing Description for more details.
| Parameters: |
|---|
Generate html description and solution from Markdown source code.
After this method is called, formatted_desc and formatted_solve will be set.
| Parameters: | hwslug (str) – The slug of owner homework. Necessary when formatting markdown sources. |
|---|
The description of the homework (Markdown source code).
The formatted description of this homework. Homework will call format_markdown to set this field, so you may use it for free.
The formatted solution of this homework. Homework will call format_markdown to set this field, so you may use it for free.
The language name of this info object, should be compatible with request.accept_languages.
The name of the homework.
The solution of this homework (Markdown source code).
Store the settings for a particular scorer in code.xml.
This class is a component in HwCode. Different programming languages may carry different scorers, and may reveal different level of program output to the students.
| Parameters: | detail – Whether this scorer should show detail reports? |
|---|
Get scorer settings from given xml node object.
An xml node to configure a particular scorer may be like this:
<scorer name="CodeStyleScorer" detail="true" />
| Parameters: | xmlnode (xml.etree.ElementTree.Element) – Xml node object holding the settings. |
|---|
Whether or not to display the detail of this scorer? If None, Railgun will use HwCode.reportRuntime as value.
Store the definition of a particular programming language in a homework assignment.
The Railgun system is designed to allow the students to choose their preferred programming language the finish their homework. Because of these, each homework should contain seperated code files for each individual language.
HwCode is the class to store language definitions and settings. The settings may include compiler parameters, runner parameters and verbosity of output messages.
All the configurations are loaded from code.xml. Refer to File Structure to see how code.xml is organized.
| Parameters: |
|---|
Get the settings of a particular scorer.
| Parameters: | typeName (str) – The scorer type name. It may be the full name of the scorer, or a partial name that matches the last part of the type name. For example, if we call get_scorer with:
We will both get the common.scorers.BaseScorer. |
|---|---|
| Returns: | The scorer setting object if found, or None otherwise. |
| Return type: | HwScorerSetting or None |
Load the definitions as a code package under path.
This method will create a HwCode object that loads settings from [path]/code.xml, of which the programming language will be set to os.path.split(path)[1].
| Parameters: | path (str) – The root directory of the code package. |
|---|---|
| Returns: | A HwCode object. |
An xml node object that stores the compiler parameters. Let the runner host to parse the parameters.
The file match rules of this code package.
Whether or not students can download an attachment for this programming language?
The programming language of this code package.
The root path of code package, should be [hw.path]/code/[lang].
Whether the compiler messages should be revealed to students? Default value is True.
Whether the runtime messages should be revealed to students? Default value is False.
An xml node object that stores the runner parameters. Let the runner host to parse the parameters.
A dict (“Scorer’s Type Name” -> HwScorerSetting). Store the specialized settings for various scorers.
Store the definition of a homework assignment.
The constructor of this class will only create an empty Homework object. You may use Homework.load(path) to construct the instance according to external homework definition files.
Get the HwCode object whose programming language is lang.
| Returns: | The requested HwCode object. |
|---|---|
| Raises: | KeyError if given language is not found. |
Get the programming language names of all HwCode objects.
| Returns: | list of all programming language names. |
|---|
Get the final deadline of this homework.
| Returns: | A tuple of (deadline datetime, score scale) if the final deadline exists, or None if the homework has already expired. |
|---|
Get the locale names of all HwInfo objects.
| Returns: | list of all locale names. |
|---|
Get the next deadline of this homework.
| Returns: | A tuple of (deadline datetime, score scale) if the next deadline exists, or None if the homework has already expired. |
|---|
List all runtime files for given programming language.
This method is typically called when you want to get the list of files that will be copied to runtime directory when executing a submission in the given programming language.
Note
Some files in the result of this method should not be revealed to students in a downloadable attachment! Use with caution!
| Parameters: | lang (str) – The programming language name. |
|---|---|
| Returns: | An iterable object of file names. |
Load the definitions of homework under path.
This method will load the settings from [path]/hw.xml, and create all HwInfo and HwCode objects according to the sub-directories.
Also, the descriptions and solutions in every locale will be formatted from Markdown source code to html display text. You may get the html from the formatted_desc and formatted_solve attribute of HwInfo.
| Parameters: | path (str) – The root directory of homework. |
|---|---|
| Returns: | A Homework object. |
| Raises: | ValueError if the homework definition is wrong. |
Pack the attachment for given programming language into filename.
All the code and resource files will be packed into a new zip archive file. This behaviour is controlled by FileRules in the Homework and the HwCode objects. You may refer to Archive Packing for more details.
| Parameters: |
|---|
List of (deadline datetime, scale factor).
The file matching rules that affects all programming languages.
The root directory of this homework assignment.
The url slug of this homework assignment.
Unique id of this homework. The submissions and final scores are associated with homework according to uuid.
Collection of Homework definition objects.
Given the root directory of all homework definitions, HwSet will discover and load all homework assignments under that directory.
| Parameters: | hwdir (str) – The root directory of all homework definitions. |
|---|
Reload the homeworks under root directory.
You may manually call this method to reload the definitions. HwSet will not monitor the file changes.
The root directory of all homework definitions.
A serializable partial score object.
The final score of a submission is a combinition of all scorers. The scores from every scorer are multiplied with a given weight, and sumed up to form the final score. A HwPartialScore is the score of one scorer.
You may refer to HwPartialScore for more details about the JSON format.
| Parameters: |
|
|---|
Make a HwPartialScore object from a plain object.
| Raises: | TypeError or KeyError if the given plain object does not represent a valid HwScore object. |
|---|
Convert this partial score object to a plain object that can be serialized in JSON message.
| Returns: | A plain object that is composed only with dict, list, str, bool and numeric types. |
|---|
The brief output message of this scorer. (railgun.common.lazy_i18n.GetTextString)
The detailed output messages of this scorer. (class:list of railgun.common.lazy_i18n.GetTextString)
Get the running time of this scorer.
| Returns: | A datetime.timedelta object, or None if timing is not performed. |
|---|
The human readable name of this scorer. (railgun.common.lazy_i18n.GetTextString)
The final score, range in [0, 100].
The running time of this scorer (in seconds).
The type name of this scorer.
The weight of this scorer.
A serializable final score object. You may refer to HwScore for more details about the JSON format.
| Parameters: |
|
|---|
Add a HwPartialScore object.
Make a HwScore object from a plain object.
| Raises: | TypeError or KeyError if the given plain object does not represent a valid HwScore object. |
|---|
Convert this partial score object to a plain object that can be serialized in JSON message.
| Returns: | A plain object that is composed only with dict, list, str, bool and numeric types. |
|---|
The compiler error message. (railgun.common.lazy_i18n.GetTextString or basestring)
The HwPartialScore objects. (list of HwPartialScore)
A brief comment on the submission. (railgun.common.lazy_i18n.GetTextString)
Wrap the AES encryption algorithm so that you can use secret keys in any size to encrypt a plain text in any length, and decrypt it backwards.
The technical parameters of this AES cipher is:
| Parameter | Description |
|---|---|
| BlockSize | 16 |
| IVSize | Equal to BlockSize. |
| KeySize | 32 |
Since AES cipher is a block cipher, the size of plain text must be a multiple of BlockSize bytes. AESCipher would pad the input text to a multiple of KeySize bytes by following method:
AES cipher relies on not only the key to decrypt a cipher text, but also the initial vector to do the decryption. So we must prepend IV to the front of cipher text. Thus the final encrypted text should be:
`initial vector` (`IVSize` bytes) + `ciphertext` (padded)
Any implementation of AES cipher in the runner hosts must be compatible with this method.
| Parameters: | key (str) – Encryption and decryption key for AES algorithm. If len(key) != 32, it will be padded or truncated. |
|---|
Utilities to load objects from csv file.
Suppose you are given csv data like this:
You may derive an object schema from CsvSchema, giving the names and types of the columns:
class MyObjectSchema(CsvSchema):
name = CsvString()
stdno = CsvString(name='student-number')
year = CsvInteger()
registered = CsvBoolean()
Then you may get the objects by:
with open('data.csv', 'rb') as f:
for obj in CsvSchema.LoadCSV(MyObjectSchema, f):
print obj
Note
The first row in csv file must be the column names! However, they may not be at the same order as defined in schema. CsvSchema uses this row to detect the order.
Define a column in csv file.
This is the base class for all types of columns. You may inherit this class to provide your own field type, for example:
import json
class CsvJsonField(object):
def fromString(self, value):
return json.loads(value)
def toString(self, value):
return json.dumps(value)
| Parameters: |
|---|
Utilities for conversions between plain datetime, localized datetime and UTC datetime. These three types are all datetime.datetime objects, while the only difference is their tzinfo property. tzinfo is the timezone information attached to datetime.datetime.
| Type | Description |
|---|---|
| plain | tzinfo is None, which means no timezone is attached. Only (year, month, day, hour, min, sec, ms) tuple of such datetime objects are meaningful. |
| UTC | tzinfo is pytz.UTC, which means the time tuple represents a datetime in UTC timezone. |
| localized | tzinfo neither None, nor pytz.UTC, representing a datetime in given timezone. |
Be careful that some timezone, such as Asia/Shanghai, is not on the exact boundary of hours (Asia/Shanghai is +08:06). Others may carry DST data. So you SHOULD ONLY use the methods provided in this module to do datetime conversions.
Attach timezone to a plain datetime object.
This method only attaches the timezone data to dt, with no adjust performed on time data. This method will ensure that the tzinfo attached to plain datetime object is on the exact boundary of hours.
| Parameters: |
|
|---|---|
| Returns: | a localized datetime object. |
Convert a UTC datetime object to localized one.
This method will adjust the time data, to make sure the conversion from UTC to localized timezone is correct.
| Parameters: |
|
|---|---|
| Returns: | a localized datetime object. |
Strip timezone from a localized or UTC datetime object.
This method only strips the timezone data from dt, with no adjust performed on time data.
| Parameters: | dt (datetime.datetime) – the localized or UTC datetime object. |
|---|---|
| Returns: | a plain datetime object. |
Convert a localized datetime object to UTC one.
This method will adjust the time data, to make sure the conversion from localized timezone to UTC is correct.
| Parameters: | dt (datetime.datetime) – a localized datetime object. |
|---|---|
| Returns: | a UTC datetime object. |
The unique interface for archive file extractors.
Railgun system can extract various types of archive files. This class provides a unique interface to create an extractor on given archive file. The format of archive will be recognized according to file extension, For examples:
>>> Extractor.open('a.zip')
<ZipExtractor instance>
>>> Extractor.open('a.rar')
<RarExtractor instance>
Also, the Extractor objects implements context manager, for example:
with Extractor.open('a.zip') as f:
for fname, fobj in f:
print 'the content of %s is:' % fname
print fobj.read()
Count all files in the archive.
| Parameters: | maxcount (int) – maximum files to count. If exceeds this limit, the method will be interrupted, and maxcount + 1 will be returned. |
|---|---|
| Returns: | the number of files in this archive. |
Get iterable (fname, fobj) from the archive.
You may simply iterate over a Extractor object, which is same as calling to this method.
| Returns: | list of tuple (fname, fobj), where fname is a str, and fobj is a file-like object. |
|---|
Check whether this archive contains only one top-level directory?
Some students may compress a whole directory. We want to detect such situations.
Note
OS X may add a hidden directory named __MACOSX to zip archives. This method will ignore such directory.
| Returns: | True if the archive file indeed contains only one top-level directory, while False otherwise. |
|---|
Get an iterable object over all relative paths of entities under directory parent.
| Returns: | iterable object over all relative paths. |
|---|---|
| Raises: | Exception from the system libraries. |
Read the file contents of path.
| Returns: | File content as string, or None if any error occurred. |
|---|
Create a new zipfile.ZipFile object.
| Parameters: | filename (str) – the file system path of created zip file. |
|---|---|
| Returns: | zipfile.ZipFile object. |
Pack all entities in files under base_path into target zipfile.
| Parameters: |
|
|---|---|
| Returns: | target |
Remove the first directory of given path and return modified str.
If ‘/’ is not found in path, assume that it does not contain a directory. For examples:
>>> remove_firstdir('/entity')
'entity'
>>> remove_firstdir('top/entity')
'entity'
>>> remove_firstdir('entity')
'entity'
| Parameters: | path (str) – The input path string. |
|---|---|
| Returns: | Path of which the first directory is removed. |
Utilities for serializable lazy translated strings.
As mentioned in I18n Everywhere, we want our Railgun system to support translations, even for the strings already stored in databases.
We know that a simple solution to the translations in a website system is to use the gettext package. Suppose we’ve created a message category including the translation from “My name is %(name)s.” to “我的姓名是 %(name)s.”, then we may write:
from gettext import gettext
>>> type(gettext('My name is %(name)s.'))
str
>>> print gettext('My name is %(name)s.') % {'name': 'Alice'}
'我的姓名是 Alice.'
The main functionality of gettext is to lookup a translation table, find the suitable translated text for input string, and then return it. However, gettext can only be configured to use a global locale.
To support multi-users in a website environment, we may use the gettext and lazy_gettext method from flask.ext.babel:
from flask.ext.babel import gettext, lazy_gettext
>>> gettext('My name is %(name)s.', name='Alice')
'我的姓名是 Alice.'
>>> type(gettext('My name is %(name)s.', name='Alice'))
:class:`str`
>>> type(lazy_gettext('My name is %(name)s.', name='Alice'))
:class:`speaklater._LazyString`
The gettext from flask.ext.babel will use the locale settings from current request, but it will translate the string at once, so you may not create a global translated text with gettext.
On the other side, the lazy_gettext will store all the arguments and create a speaklater._LazyString instance, and will only do the translation until it is being output. So you can make a global translated text with lazy_gettext.
However, we still have problems. speaklater._LazyString could not be serialized into a JSON message, nor could it be stored into database.
This is why I create my own GetTextString and lazy_gettext() method. However, these utilities do have some limitations:
Note
When to use flask.ext.babel.lazy_gettext() and when to use railgun.common.lazy_i18n.lazy_gettext()? Well, if you want to store the lazy string into database, or if you want to transfer the string over website api, you have to use lazy_gettext() provided by this module. However, if you just want to use it for website display, both can work.
Make a serializable lazy gettext object.
| Parameters: |
|
|---|
alias of GetTextString
Convert a basestring or a GetTextString object to a JSON serializable plain object. You may refer to GetTextString for more details about the JSON format.
| Parameters: | s – The string object to be converted. |
|---|---|
| Returns: | A plain object that is composed only with dict, list, str, bool and numeric types. |
| Raises: | TypeError if s is not converible. |
Convert a plain object to basestring or GetTextString. You may refer to GetTextString for more details about the JSON format.
| Parameters: | s – The plain object. |
|---|---|
| Returns: | A basestring or a GetTextString object. |
| Raises: | KeyError if s is not converible. |
Indicate that the timeout was reached when executing a process.
Execute a command, read the output and return it back.
| Parameters: | |
|---|---|
| Returns: | (stdout, stderr, exit code) |
| Return type: | tuple |
| Raises: | OSError on missing command or any other OS errors. |
| Raises: | ProcessTimeout if a timeout was reached. |
Find object according to name.
This method will try to import all the intermediate modules to find the requested object.
Examples:
>>> find_object('os')
<module 'os' from '?'>
>>> find_object('os.path')
<module 'posixpath' from '?'>
>>> find_object('os.remove')
<function posix.remove>
>>> find_object('os.path.split')
<function posixpath.split>
>>> find_object('os.path.split.__name__')
'split'
| Parameters: | name (str) – The name of the object, should contain full path from the global scope. |
|---|---|
| Returns: | The object instance. |
| Raises: | ImportError if object with name cannot be found. |
It’s a common requirement to create a temporary directory for some work, and delete it after the work has done. TempDir provides such utility to manage a temporary directory.
All the temporary directories are created under config.TEMPORARY_DIR. You may create a temp directory like this:
with TempDir() as d:
fpath = d.fullpath('subfile.txt')
with open(fpath, 'wb') as f:
f.write('hello, world!')
| Parameters: | name (str) – The name of this temporary directory. If not given, it will generate a randomized name by uuid.uuid4().get_hex(). |
|---|
Change the Unix file system mode of this directory.
| Parameters: |
|---|
Change the owner uid and gid of this directory.
| Parameters: |
|---|
Delete the temporary directory recursively.
If the directory does not exist, this method will do nothing.
Copy all files from source directory into this.
| Parameters: |
|---|
Extract files into this directory.
| Parameters: |
|
|---|
Get the fullpath for child entity.
| Parameters: | subpath (str) – Relative path for the child entity. |
|---|
Create the temporary directory.
If the directory already exists, this method will do nothing. You may call chown to ensure the mode of this directory.
| Parameters: | mode (int) – Unix file system mode for this temporary directory. |
|---|
Hold the name of this temporary directory.
Hold the path of this temporary directory.
A useful class to find all the urls from given text, and replace them with a callback function.
For example, if you want to replace all file:/// urls to http:// ones, where all the files are located under /var/www/share/, and all the http urls should start with http://localhost/files/, then we may write:
def ReplaceUrl(url):
if url.startswith('file:///var/www/share/'):
return 'http://localhost/files/%s' % url[22:]
return url
matcher = UrlMatcher(schemas=['file'])
payload = "Here is the movie: " \
"file:///var/www/share/favourites/Harry-Potter.mov"
print matcher.replace(payload, ReplaceUrl)
Note that we only consider the following characters as components of urls:
A-Z, a-z, 0-9, and any one of "-_.~!*';:@&=+$,/?#"
| Parameters: | schemas (list) – Interested url schemas. |
|---|
Reformat the given path to Unix style.
The given path will be modified according to following rules:
There’s some special cases:
Examples:
>>> reform_path('1\\2')
'1/2'
>>> reform_path('////1////2')
'/1/2'
>>> reform_path('/1/2/3/../4/../..')
'/1'
>>> reform_path('/1/..')
'/'
>>> reform_path('../')
Traceback (most recent call last):
File "a.py", line 63, in <module>
reform_path('../')
...
ValueError: .. out of root
Traceback (most recent call last):
File "a.py", line 63, in <module>
reform_path('../')
...
ValueError: .. out of root
| Parameters: | path (str) – Original path string. |
|---|---|
| Returns: | The translated new path. |
| Raises: | ValueError if ”..” could not find any parent to consume. |