pyratemp

1.1   Features

My template-engine has everything I think a template-engine needs:

• "category 3" (see my thoughts about Template Engines)

• template-defined functions/macros

• inclusion of other templates

• very powerful expressions (due to Python)

• pseudo-sandbox: the Python-expressions are evaluated inside a "pseudo-sandbox" which prevents that "bad things" are done by accident; and even a "real" sandbox could be added

• clear template syntax ("There should be one -- and preferably only one -- obvious way to do it." [2])

• non-XML, so it can be used for any kind of documents

• integrated special-character-escaping (e.g. HTML, LaTeX)

• good error-handling and good error-messages, incl. the exact position of an error in the template-file

• completely uses Unicode

• easy to use

• well documented

• fast and lightweight

• modular: pyratemp consists of several parts, which can even be used separately or replaced by an alternative implementation

• small code base, which is well documented and should be easy to read and understand

  (about 500 lines-of-code + 500 lines of docstrings and comments)

• extensible: additional functions can be easily added to the templates, and due to its modularity, whole parts can even be replaced

3.1   Expressions

pyratemp uses (restricted) embedded Python-expressions.

An expression is everything which evaluates to a value, e.g. variables, arithmetics, comparisons, boolean expressions, function/method calls, list comprehensions etc. [4]. And these Python expressions can be directly used in the templates -- this makes pyratemp very powerful.

Examples:

• variable-access: var, mylist[i], mydict["key"], myclass.attr
• function/method call: myfunc(...), "foobar".upper()
• comparison: i < 0  or  j > 1024
• arithmetics: 1+2

For details, please read the Python-documentation.

Note that accessing undefined variables in the template is considered to be an error. I chose this behavior in contrast to many other template-engines (which often ignore undefined variables), since ignoring undefined variables would silently hide some errors, which would be a bad idea. For ignoring or using a default-value for undefined variables, see default() below.

The following Python-built-in values/functions are available by default in the template:

True
False
None

abs()
chr()
cmp()
divmod()
hash()
hex()
len()
max()
min()
oct()
ord()
pow()
range()
round()
sorted()
sum()
unichr()
zip()

bool()
complex()
dict()
enumerate()
float()
int()
list()
long()
reversed()
str()
tuple()
unicode()
xrange()

Additionally, the functions default() and exists() are defined as follows:

default("expr", default=None):

default() tries to evaluate the expression expr. If the evaluation succeeds and the result is not None, its value is returned; otherwise the default-value is returned instead. Note that expr has to be quoted.

Since it is considered an error if the template tries to evaluate an undefined variable, this can be used to use default-values for optional variables, e.g. Name: @!default("myvar", "No name given.")!@.

exists("varname"):

This tests if a variable (or any other object) with the name varname exists (in the current locals-namespace). Note that the name of the variable has to be quoted, and that this only works for single variable names, e.g. exists("mylist"). If you want to test more complicated expressions, use the default()-function.

It's especially useful in if-conditions to check if some (optional) variable exists, and then to branch accordingly.

Please look into the pyratemp-docstring for more examples of default() and exists().

More/user-defined functions can be added to the template as described in User-Interface.

3.2   Substitution

The template can contain placeholders, which get replaced by a string. pyratemp has two different placeholders:

• @!EXPR!@ escaped substitution: special characters are escaped
• $!EXPR!$ unescaped/raw substitution

Normally, you should always use @!...!@, since this escapes special characters. Currently, the following formats are supported:

• HTML (default): & < > " ' are escaped to &amp;, &lt;, &gt;, &quot;, &#39.
• LATEX: currently only some special characters are replaced, so use with care!
• NONE: no characters are replaced

These placeholders can contain any Expression (incl. macros), which is evaluated when the template is rendered.

3.3   Comments

Comments can also be used in a template, and they are especially useful for temporarily disabling parts of the template:

• #!...!# single-line comment with start- and end-tag
• #!... single-line-comment until end-of-line, incl. newline

Comments can contain anything, but comments may not appear inside expressions, substitutions or block-tags.

The second version also comments out the newline, and so can be used at the end of a line to remove that newline in the result of the template (like a backslash in Python-strings).

3.4   Blocks

The control-structures, macros etc. have a special syntax, which consists of a start-tag (which is named according to the block), optional additional tags, and an end-tag (all on their own line):

<!--(...)-->
   ..
   ..
[<!--(...)-->]
   ..
   ..
<!--(end)-->

All tags which belong to the same block must have the same indent! Nesting blocks is also possible, but the tags of nested blocks must have a different indent than the tags of the enclosing blocks.

Note that you should either use spaces or tabs for indentation. Since pyratemp distinguishes between spaces and tabs, if you mix spaces and tabs, two indentations might look the same (e.g. 8 spaces or 1 tab) but still be different, which might lead to unexpected errors.

Often, it's good to indent the content of a block to improve readability, but that's not necessary -- it's also possible to use a block where the start- and end-tag are indented e.g. 4 spaces and the content of the block isn't indented at all.

There's also a single-line version of a block. Here, the start- and end-tag must be in the same line, and nesting is not supported [5]:

...<!--(...)-->...[<!--(...)-->...]<!--(end)-->...

<!--(if EXPR)-->
...
<!--(elif EXPR)-->
...
<!--(else)-->
...
<!--(end)-->

...<!--(if EXPR)-->...<!--(elif EXPR)-->...<!--(else)-->...<!--(end)-->...

<!--(for VARS in EXPR)-->
...
<!--(else)-->
...
<!--(end)-->

...<!--(for VARS in EXPR)-->...<!--(else)-->...<!--(end)-->...

<!--(macro MACRONAME)-->
...
<!--(end)-->

...<!--(macro MACRONAME)-->...<!--(end)-->...

MACRONAME

MACRONAME(KEYWORD_ARGs)

<!--(macro header)-->
  <!--(set_escape)-->
    html
  <!--(end)-->
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
          "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
  <title>A simple example: @!title!@</title>
</head>
<body>
<!--(end)-->

<!--(macro myfunc)-->
<li><!--(if exists("link"))--><a href="@!link!@">@!default("item",link)!@</a><!--(else)-->@!item!@<!--(end)--></li>
<!--(end)-->


@!header!@
<ul>
@!myfunc(item="Macros are fun!")!@
@!myfunc(item="Simple is better", link="http://www.simple-is-better.org")!@
@!myfunc(link="http://www.wikipedia.org")!@
</ul>

<!--(raw)-->
...
<!--(end)-->

...<!--(raw)-->...<!--(end)-->...

<!--(include)-->
  FILENAME
<!--(end)-->

...<!--(include)-->FILENAME<!--(end)-->...

<!--(set_escape)-->
  FORMAT
<!--(end)-->

...<!--(set_escape)-->FORMAT<!--(end)-->...

4.1   User-Interface

The user-interface of the template-engine consists of a single class: pyratemp.Template. This loads a template, checks its syntax, parses it, and can render it with your data.

class Template(string|filename|parsetree, [data], [encoding], [escape]):

Load (and parse) a template. The template can either be directly given as string, or loaded from a file, or an already parsed tree can be used.

The optional parameter data defines the data which should be filled into the template by default (=values used if this data is not given when calling/rendering the template), encoding tells the charset-encoding of the loaded template (default: UTF-8), and escape defines which special-characters should be escaped in substitutions by default (currently supported: NONE, HTML, and partially LATEX). The escaping can also be set directly in the template (see set_escape), and setting the escape-format in the template overrides the one set here.

Note that you have to use keyword-parameters here!

To render the template, simply call the template with your data as parameters. This returns the result in Unicode, and you should encode it depending on your needs. Of course, the same template can be rendered several times with different data.

Note that data and the parameters when calling the template can contain nearly anything: single variables, lists, other dictionaries, nested structures, functions and even classes. So be careful what you pass to the template. If you e.g. pass the Python-built-in-function open to the template, your template will be able to open arbitrary files!

Example:

>>> import pyratemp
>>> t = pyratemp.Template(filename="test.html", {"number": 1}, escape=pyratemp.HTML)
>>> result1 = t(person="Monty")
>>> result2 = t(person="Adams", number=42)
>>> print result1.encode("utf-8")
>>> print result2.encode("ascii", 'xmlcharrefreplace')

4.2   Loader

There are two sources to "load" a template from: either directly from a string, or from a file. Since templates-from-files can include other templates (by using <!--(include)-->)), this "template-loading" is encapsulated into its own classes.

class StringLoader([encoding]):

The "string-template-loader" simply returns the given template-string, decoded to Unicode. Note that including other templates is not possible when loading a template directly from a string.

To load the string, load(string) is used.

class FileLoader(allowed_path, [encoding]):

When loading templates from files, including other templates is supported. But for simplicity and security, only files which are in the same directory as the template-file itself can be included. This directory must be given as allowed_path. To load a template, load(filename) is used, where filename is the basename of the template-file (without path). The loaded template, decoded to Unicode, is returned.

4.3   Parser

It's better, cleaner and even faster to parse the template once and afterwards separately render it (maybe multiple times).

The parser analyzes the template-string, checks the syntax (and throws exceptions with detailed error-descriptions if there is an error), and generates a parse-tree. Since indentation is used for nesting in the template, the template can be completely parsed by using regexps, which makes parsing really fast and simple. Most of the parser-code is used to check for (syntax-)errors and to create error-messages.

4.4   Evaluation

pyratemp uses Python-expressions [7] in its templates.

But since it is a really bad idea to directly embed unrestricted code into a template [8], pyratemp uses a "pseudo-sandbox" for evaluating the Python-expressions. This restricts the embedded expressions, so that the template-designer only has the necessary functionality, and that he cannot do "bad things" by accident. But note that this may not be a real sandbox! Although I currently don't know any way to break out of this sandbox, and I think that it shouldn't be possible to break out (without passing in an unsafe function [9]), I'm not absolutely sure about that.

So, if you want to use pyratemp for "untrusted" templates, you should make sure that it's not possible to break out. There are different possible ways:

• Approve that it's not possible to break out of the integrated "pseudo-sandbox".

• Add a really sandboxed expression-evaluator, and use it instead of the TemplateEval class. This could even be done incrementally, by first writing a simple evaluator which only supports string-substitution, and then adding arithmetics and other functionality as needed.

  But since such a sandboxed evaluator would increase the complexity and probably would only support a subset of the Python-expressions, I did not write such an evaluator yet.

4.5   Renderer

The renderer takes a parse-tree and your data, evaluates all embedded expressions and control-structures, expands the macros, escapes special characters (and tries to prevent double-escapes) and returns the result as Unicode-string.

4.6   TemplateBase

The TemplateBase class on the one hand implements parts of the user-interface, on the other hand provides the functionality for user-defined macros in the template. Remember that a macro in the template is exactly the same as a (sub-)template!

4.7   Compiler

I also wrote a small (about 100 lines) experimental compiler which compiles the templates (or: the parsed trees) to pure Python-code. It's quite simple, and may even speed up the rendering a bit. But since pyratemp is already very fast, and compiling only has advantages if you render a template many times, I haven't developed the compiler any further.

5.1   yaml2pyratemp

yaml2pyratemp.py is a simple command-line-interface to pyratemp which fills a template with the data from YAML- or JSON-files or from the command-line and prints the result to stdout in UTF-8. By default, special html-characters are escaped (except for *.tex), and two additional variables are defined: date and mtime_CCYYMMDD, both containing the current date in the "YYYY-MM-DD"-format.

Again, read the yaml2pyratemp-docstrings for details.

Note that yaml2pyratemp needs the Python-modules simplejson and/or yaml to read JSON- and/or YAML-files.

Usage (see also yaml2pyratemp.py --help):

yaml2pyratemp.py [-s] <-d NAME=VALUE> <-f DATAFILE [-n NR_OF_ENTRY]> TEMPLATEFILES
    -s      syntax-check only (don't render the template)
    -d      define variables (these also override the values from files)
    -f      use variables from YAML/JSON file(s)
    -n      use nth entry of the YAML/JSON-file
            (YAML: n-th entry, JSON: n-th element of the root-array)

For the 2nd example of Quickstart, a JSON-file might look like:

{
  "title" : "filling JSON into pyratemp",
  "special_chars" : "µ<߀",
  "number" : 13,
  "mylist" : [ "JSON", "YAML", "manually-defined variables" ]
}

To fill the template, with using a different value for number than in the JSON-file, invoke yaml2pyratemp as follows:

$ yaml2pyratemp.py -d number=42 -f example.json example.html > filled.html

Now, the result is in filled.html

6.1   Syntax Errors

To check a pyratemp-template for syntax-errors, simply let pyratemp parse the template. You can do this i.e. via yaml2pyratemp.py without any data and with -s:

$ yaml2pyratemp.py -s TEMPLATEFILE(s)

If there are syntax-errors, pyratemp raises a TemplateSyntaxError, and yaml2pyratemp displays a detailed error-message:

$ yaml2pyratemp.py -s TEMPLATEFILE
file 'TEMPLATEFILE':
  TemplateSyntaxError: line ##, col ##: ...

6.2   Fillout/Render test

For a complete test, you have to render the template. Of course you can do this in your application, with real data, but probably it's easier to test it outside of the application with some "dummy data".

This again can be done by yaml2pyratemp. Simply create a YAML- or JSON-file with your dummy data, and invoke yaml2pyratemp. If the YAML-/JSON-file contains all necessary data, the rendered template will be written to stdout, e.g.:

$ yaml2pyratemp.py -f dummydata.json example.html
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
          "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
...

Otherwise, if some data is missing (or invalid) or some expressions are invalid, pyratemp raises a TemplateRenderError and tells you what data is missing, e.g.:

$ yaml2pyratemp.py -f dummydata.json example.html
file 'example.html':
  TemplateRenderError: Cannot eval expression 'title' (NameError: name 'title' is not defined)