template: Template amounts are now written with fuller expressions.

This makes it possible to support transactions that divide funds in
ways other than a simple percentage split.  For example, there
might be tax withheld, or a flat fee expense on each transaction.

README.rst

@@ -24,15 +24,15 @@ Writing templates
 A template looks like a Ledger entry with a couple of differences:
 
 * You don't write the first payee line.
-* You allocate percentages to each account, rather than specific currency amounts.
+* Instead of writing specific currency amounts, you write simple expressions to calculate amounts.
 
 Here's a simple template for Patreon patron payments::
 
   [DEFAULT]
   template patreon income =
     ;Tag: Value
-    Income:Patreon  -100%%
+    Income:Patreon  -{amount}
-    Accrued:Accounts Receivable:Patreon  100%%
+    Accrued:Accounts Receivable:Patreon  {amount}
 
 Let's walk through this line by line.
 
@@ -42,12 +42,41 @@ Every setting in your configuration file has to be in a section.  ``[DEFAULT]``
 
 The first line of the template is a Ledger tag.  The program will leave all kinds of tags and Ledger comments alone, except to indent them nicely.
 
-The next two lines split the money across accounts.  They follow almost the same format as they do in Ledger: there's an account named, followed by a tab or two or more spaces, and then an amount.  However, rather than a currency amount, specify a percentage amount between -100% and 100%.  You can specify any number of splits you want, as long as the negative splits total -100% and the positive splits total 100%.  ``%`` is a special character in INI file syntax; writing ``%%`` is the way to get a literal ``%`` in the string.  Alternatively, you can omit the ``%`` altogether; import2ledger will still treat any floating point value at the end of the line as a percentage.
+The next two lines split the money across accounts.  They follow almost the same format as they do in Ledger: there's an account named, followed by a tab or two or more spaces, and then an expression.  Each time import2ledger generates an entry, it will evaluate this expression using the data it imported to calculate the actual currency amount to write.  Your expression can use numbers, basic arithmetic operators (including parentheses for grouping), and imported data referred to as ``{variable_name}``.
 
-If the amount of currency being imported doesn't split evenly, spare change will be allocated to the last split to keep the entry balanced.
+import2ledger uses decimal math to calculate each amount, and rounds to the number of digits appropriate for that currency.  If the amount of currency being imported doesn't split evenly, spare change will be allocated to the last split to keep the entry balanced on both sides.
 
 Refer to the `Python documentation for INI file structure <https://docs.python.org/3/library/configparser.html#supported-ini-file-structure>`_ for full details of the syntax.  Note that import2ledger doesn't use ``;`` as a comment prefix, since that's the primary comment prefix in Ledger.
 
+Template variables
+~~~~~~~~~~~~~~~~~~
+
+import2ledger templates have access to a few variables for each transaction that can be included anywhere in the entry, not just amount expressions.  In other parts of the entry, they're treated as strings, and you can control their formatting using `Python's format string syntax <https://docs.python.org/3/library/string.html#formatstrings>`_.  For example, this template sets different payees for each side of the transaction::
+
+  template patreon income =
+    Income:Patreon  -{amount}
+    ;Payee: {payee}
+    Accrued:Accounts Receivable:Patreon  {amount}
+    ;Payee: Patreon
+
+Every template can use the following variables:
+
+================== ==========================================================
+Name               Contents
+================== ==========================================================
+amount             The total amount of the transaction, as a simple decimal
+                   number (not currency-formatted)
+------------------ ----------------------------------------------------------
+currency           The three-letter code for the transaction currency
+------------------ ----------------------------------------------------------
+date               The date of the transaction, in your configured output
+                   format
+------------------ ----------------------------------------------------------
+payee              The name of the transaction payee
+================== ==========================================================
+
+Specific importers and hooks may provide additional variables.
+
 Supported templates
 ~~~~~~~~~~~~~~~~~~~
 
@@ -70,40 +99,14 @@ Patreon
 
   This template can use these variables:
 
-  ``country_name``
+  ============== ============================================================
-    The full name of the country VAT was withheld for
+  Name           Contents
-
+  ============== ============================================================
-  ``country_code``
+  country_name   The full name of the country VAT was withheld for
-    The two-letter ISO country code of the country VAT was withheld for
+  -------------- ------------------------------------------------------------
-
+  country_code   The two-letter ISO country code of the country VAT was
-Template variables
+                 withheld for
-~~~~~~~~~~~~~~~~~~
+  ============== ============================================================
-
-import2ledger templates have access to a few variables for each transaction that can be included in the output entry using `Python's format string syntax <https://docs.python.org/3/library/string.html#formatstrings>`_.  For most uses, all you need to do is write ``{variable_name}`` in the template, and the variable's value will be filled in there.  For example, this template sets different payees for each side of the transaction::
-
-  template patreon income =
-    Income:Patreon  -100%%
-    ;Payee: {payee}
-    Accrued:Accounts Receivable:Patreon  100%%
-    ;Payee: Patreon
-
-You can use the following variables:
-
-================== ==========================================================
-Name               Contents
-================== ==========================================================
-amount             The total amount of the transaction, as a simple decimal
-                   number (not currency-formatted)
------------------- ----------------------------------------------------------
-currency           The three-letter code for the transaction currency
------------------- ----------------------------------------------------------
-date               The date of the transaction, in your configured output
-                   format
------------------- ----------------------------------------------------------
-payee              The name of the transaction payee
-================== ==========================================================
-
-Custom hooks may add more template variables.
 
 Other output options
 ~~~~~~~~~~~~~~~~~~~~
@@ -112,6 +115,8 @@ Various options control how import2ledger formats dates and currency amounts.  R
 
   date_format = %%Y-%%m-%%d
 
+(``%`` is a special character in the configuration file syntax.  ``%%`` represents a literal percent sign.)
+
 Configuration sections
 ~~~~~~~~~~~~~~~~~~~~~~
 

import2ledger/config.py

@@ -241,6 +241,7 @@ class Configuration:
             signed_currencies=[code.strip().upper() for code in section_config['signed_currencies'].split(',')],
             signed_currency_fmt=section_config['signed_currency_format'],
             unsigned_currency_fmt=section_config['unsigned_currency_format'],
+            template_name=config_key,
         )
 
     def setup_logger(self, logger, section_name=None):

import2ledger/template.py

@@ -1,49 +1,153 @@
 import collections
 import datetime
 import decimal
+import functools
+import io
+import operator
+import re
+import tokenize
 
 import babel.numbers
 
-class AccountSplitter:
+from . import errors
-    Split = collections.namedtuple('Split', ('account', 'percentage'))
 
-    def __init__(self, sign, signed_currencies, signed_currency_fmt, unsigned_currency_fmt):
+class TokenTransformer:
-        self.splits = []
+    def __init__(self, source):
-        self.sign = sign
+        try:
+            source = source.readline
+        except AttributeError:
+            pass
+        self.in_tokens = tokenize.tokenize(source)
+
+    @classmethod
+    def from_bytes(cls, b):
+        return cls(io.BytesIO(b).readline)
+
+    @classmethod
+    def from_str(cls, s, encoding='utf-8'):
+        return cls.from_bytes(s.encode(encoding))
+
+    def __iter__(self):
+        for ttype, tvalue, _, _, _ in self.in_tokens:
+            try:
+                transformer = getattr(self, 'transform_' + tokenize.tok_name[ttype])
+            except AttributeError:
+                raise ValueError("{} token {!r} not supported".format(ttype, tvalue))
+            yield from transformer(ttype, tvalue)
+
+    def _noop_transformer(self, ttype, tvalue):
+        yield (ttype, tvalue)
+
+    transform_ENDMARKER = _noop_transformer
+
+    def transform_ENCODING(self, ttype, tvalue):
+        self.in_encoding = tvalue
+        return self._noop_transformer(ttype, tvalue)
+
+    def transform(self):
+        out_bytes = tokenize.untokenize(self)
+        return out_bytes.decode(self.in_encoding)
+
+
+class AmountTokenTransformer(TokenTransformer):
+    SUPPORTED_OPS = frozenset('+-*/()')
+
+    transform_NAME = TokenTransformer._noop_transformer
+
+    def transform_NUMBER(self, ttype, tvalue):
+        yield (tokenize.NAME, 'Decimal')
+        yield (tokenize.OP, '(')
+        yield (tokenize.STRING, repr(tvalue))
+        yield (tokenize.OP, ')')
+
+    def transform_OP(self, ttype, tvalue):
+        if tvalue == '{':
+            try:
+                name_type, name_value, _, _, _ = next(self.in_tokens)
+                close_type, close_value, _, _, _ = next(self.in_tokens)
+                if (name_type != tokenize.NAME
+                    or close_type != tokenize.OP
+                    or close_value != '}'):
+                    raise ValueError()
+            except (StopIteration, ValueError):
+                raise ValueError("opening { does not name variable")
+            yield (tokenize.NAME, name_value)
+        elif tvalue in self.SUPPORTED_OPS:
+            yield from self._noop_transformer(ttype, tvalue)
+        else:
+            raise ValueError("unsupported operator {!r}".format(tvalue))
+
+
+class AccountSplitter:
+    def __init__(self, signed_currencies, signed_currency_fmt, unsigned_currency_fmt,
+                 template_name):
+        self.splits = collections.OrderedDict()
         self.signed_currency_fmt = signed_currency_fmt
         self.unsigned_currency_fmt = unsigned_currency_fmt
         self.signed_currencies = set(signed_currencies)
-        self.unsplit = sign * 100
+        self.template_name = template_name
+        self._last_template_vars = object()
 
-    def add(self, account, percentage_s):
+    def add(self, account, amount_expr):
-        int_pctage = decimal.Decimal(percentage_s.rstrip('%'))
+        try:
-        self.unsplit -= int_pctage
+            clean_expr = AmountTokenTransformer.from_str(amount_expr).transform()
-        percentage = None if self.unsplit == 0 else (int_pctage / 100)
+        except ValueError as error:
-        self.splits.append(self.Split(account, percentage))
+            raise errors.UserInputConfigurationError(error.args[0], amount_expr)
+        self.splits[account] = compile(clean_expr, self.template_name, 'eval')
+
+    def _currency_decimal(self, amount, currency):
+        return decimal.Decimal(babel.numbers.format_currency(amount, currency, '###0.###'))
+
+    def _balance_amounts(self, amounts, to_amount):
+        cmp_func = operator.lt if to_amount > 0 else operator.gt
+        should_balance = functools.partial(cmp_func, 0)
+        remainder = to_amount
+        to_account = None
+        for account, amount in amounts.items():
+            if should_balance(amount):
+                remainder -= amount
+                to_account = account
+        if to_account is None:
+            pass
+        elif (abs(remainder) / abs(to_amount)) >= decimal.Decimal('.1'):
+            raise errors.UserInputConfigurationError(
+                "template can't balance amounts to {}".format(to_amount),
+                self.template_name,
+            )
+        else:
+            amounts[to_account] += remainder
+
+    def _build_amounts(self, template_vars):
+        amount_vars = {k: v for k, v in template_vars.items() if isinstance(v, decimal.Decimal)}
+        amount_vars['Decimal'] = decimal.Decimal
+        amounts = collections.OrderedDict(
+            (account, self._currency_decimal(eval(amount_expr, amount_vars),
+                                             template_vars['currency']))
+            for account, amount_expr in self.splits.items()
+        )
+        self._balance_amounts(amounts, template_vars['amount'])
+        self._balance_amounts(amounts, -template_vars['amount'])
+        return amounts
+
+    def _iter_splits(self, template_vars):
+        amounts = self._build_amounts(template_vars)
+        if template_vars['currency'] in self.signed_currencies:
+            amt_fmt = self.signed_currency_fmt
+        else:
+            amt_fmt = self.unsigned_currency_fmt
+        for account, amount in amounts.items():
+            amt_s = babel.numbers.format_currency(amount, template_vars['currency'], amt_fmt)
+            yield '    {:45}  {:>19}\n'.format(account, amt_s)
 
     def render_next(self, template_vars):
-        try:
+        if template_vars is not self._last_template_vars:
-            account, percentage = next(self._split_iter)
+            self._split_iter = self._iter_splits(template_vars)
-        except (AttributeError, StopIteration):
+            self._last_template_vars = template_vars
-            self._split_iter = iter(self.splits)
+        return next(self._split_iter)
-            account, percentage = next(self._split_iter)
-            self._remainder = self.sign * template_vars['amount']
-            self._currency = template_vars['currency']
-            if self._currency in self.signed_currencies:
-                self._amt_fmt = self.signed_currency_fmt
-            else:
-                self._amt_fmt = self.unsigned_currency_fmt
-        if percentage is None:
-            amount = self._remainder
-        else:
-            amount = decimal.Decimal(babel.numbers.format_currency(
-                template_vars['amount'] * percentage, self._currency, '###0.###'))
-        self._remainder -= amount
-        amt_s = babel.numbers.format_currency(amount, self._currency, self._amt_fmt)
-        return '    {:45}  {:>19}\n'.format(account, amt_s)
 
 
 class Template:
+    ACCOUNT_SPLIT_RE = re.compile(r'(?:\t|  )\s*')
     DATE_FMT = '%Y/%m/%d'
     SIGNED_CURRENCY_FMT = '¤#,##0.###;¤-#,##0.###'
     UNSIGNED_CURRENCY_FMT = '#,##0.### ¤¤'
@@ -51,12 +155,11 @@ class Template:
     def __init__(self, template_s, signed_currencies=frozenset(),
                  date_fmt=DATE_FMT,
                  signed_currency_fmt=SIGNED_CURRENCY_FMT,
-                 unsigned_currency_fmt=UNSIGNED_CURRENCY_FMT):
+                 unsigned_currency_fmt=UNSIGNED_CURRENCY_FMT,
+                 template_name='<template>'):
         self.date_fmt = date_fmt
-        self.pos_splits = AccountSplitter(
+        self.splitter = AccountSplitter(
-            1, signed_currencies, signed_currency_fmt, unsigned_currency_fmt)
+            signed_currencies, signed_currency_fmt, unsigned_currency_fmt, template_name)
-        self.neg_splits = AccountSplitter(
-            -1, signed_currencies, signed_currency_fmt, unsigned_currency_fmt)
 
         lines = [s.lstrip() for s in template_s.splitlines(True)]
         for index, line in enumerate(lines):
@@ -72,10 +175,14 @@ class Template:
             if line[0].isalpha():
                 if start_index < index:
                     self._add_str_func(lines[start_index:index])
-                account, percentage_s = line.rsplit(None, 1)
+                line = line.strip()
-                splitter = self.neg_splits if percentage_s.startswith('-') else self.pos_splits
+                match = self.ACCOUNT_SPLIT_RE.search(line)
-                splitter.add(account, percentage_s)
+                if match is None:
-                self.format_funcs.append(splitter.render_next)
+                    raise errors.UserInputError("no amount expression found", line)
+                account = line[:match.start()]
+                amount_expr = line[match.end():]
+                self.splitter.add(account, amount_expr)
+                self.format_funcs.append(self.splitter.render_next)
                 start_index = index + 1
         if start_index <= index:
             self._add_str_func(lines[start_index:])

tests/data/templates.ini

@@ -1,22 +1,29 @@
 [Simplest]
 template =
- Accrued:Accounts Receivable  100%%
+ Accrued:Accounts Receivable  {amount}
- Income:Donations  -100%%
+ Income:Donations  -{amount}
 
 [FiftyFifty]
 template =
- Accrued:Accounts Receivable  100%%
+ Accrued:Accounts Receivable  {amount}
- Income:Donations  -50%%
+ Income:Donations  -.5 * {amount}
- Income:Sales  -50%%
+ Income:Sales  -.5*{amount}
 
 [Complex]
 template =
  ;Tag: Value
  ;TransactionID: {txid}
- Accrued:Accounts Receivable  100%%
+ Accrued:Accounts Receivable  {amount}
  ;Entity: Supplier
- Income:Donations:Specific  -95.5%%
+ Income:Donations:Specific    -.955*  {amount}
  ;Program: Specific
  ;Entity: {entity}
- Income:Donations:General  -4.5%%
+ Income:Donations:General     -.045  * {amount}
  ;Entity: {entity}
+
+[Multivalue]
+template =
+ Expenses:Taxes  {tax}
+ Accrued:Accounts Receivable  {amount} - {tax}
+ Income:RBI         -.1*{amount}
+ Income:Donations   -.9*{amount}

tests/data/test_config.ini

@@ -5,15 +5,15 @@ signed_currencies = EUR
 [Templates]
 output_path = Template.output
 one =
- Accrued:Accounts Receivable  100%%
+ Accrued:Accounts Receivable  {amount}
- Income:Donations  -100%%
+ Income:Donations  -{amount}
 two =
  ;Tag1: {value}
- Income:Donations  -100%%
+ Income:Donations  -{amount}
  ;IncomeTag: Donations
- Expenses:Fundraising  5%%
+ Expenses:Fundraising  .05 * {amount}
  ;ExpenseTag: Fundraising
- Accrued:Accounts Receivable  95%%
+ Accrued:Accounts Receivable  .95 * {amount}
  ;AccrualTag: Receivable
 
 [Date]

tests/data/test_main.ini

@@ -5,8 +5,8 @@ signed_currencies = USD
 
 [One]
 template patreon cardfees =
- Accrued:Accounts Receivable  -100%%
+ Accrued:Accounts Receivable  -{amount}
- Expenses:Fees:Credit Card  100%%
+ Expenses:Fees:Credit Card  {amount}
 template patreon svcfees =
- Accrued:Accounts Receivable  -100%%
+ Accrued:Accounts Receivable  -{amount}
- Expenses:Fundraising  100%%
+ Expenses:Fundraising  {amount}

tests/test_config.py

@@ -31,6 +31,7 @@ def test_defaults():
     assert kwargs == {
         'date_fmt': '%Y-%m-%d',
         'signed_currency_fmt': kwargs['signed_currency_fmt'],
+        'template_name': 'one',
         'unsigned_currency_fmt': kwargs['unsigned_currency_fmt'],
     }
 
@@ -43,7 +44,7 @@ def test_template_parsing():
     except IndexError as error:
         assert False, error
     assert "\n;Tag1: {value}\n" in tmpl_s
-    assert "\nIncome:Donations  -100%\n" in tmpl_s
+    assert "\nIncome:Donations  -{amount}\n" in tmpl_s
     assert "\n;IncomeTag: Donations\n" in tmpl_s
 
 @pytest.mark.parametrize('arg_s', [None, '-', 'output.ledger'])

tests/test_templates.py

@@ -72,3 +72,17 @@ def test_balancing():
         "  Income:Donations  -0.50 USD",
         "  Income:Sales  -0.51 USD",
     ]
+
+def test_multivalue():
+    tmpl = template_from('Multivalue')
+    rendered = tmpl.render('DD', decimal.Decimal('150.00'), 'USD', DATE,
+                           tax=decimal.Decimal('12.50'))
+    lines = [normalize_whitespace(s) for s in rendered.splitlines()]
+    assert lines == [
+        "",
+        "2015/03/14 DD",
+        "  Expenses:Taxes  12.50 USD",
+        "  Accrued:Accounts Receivable  137.50 USD",
+        "  Income:RBI  -15.00 USD",
+        "  Income:Donations  -135.00 USD",
+    ]