NPO-Accounting/import2ledger
6
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

hooks.ledger_entry: Look up templates dynamically.

Browse source 
If there's a 'ledger entry' key in the entry data, use that value as the
name of the template to load.  Thanks to this, nbpy2017 could collapse
multiple importers into one.

Otherwise, build a default template name based on the importer source, and
try to use that.

All the configuration names now end with "ledger entry" instead of starting
with "template".  This makes it clearer what they're for, in case we
support other kinds of output templates in the future.

I ended up changing the names of some of the importers so the default
template name was nice, rather than specifying template names for all of
them, to reduce the amount of name discrepancies across the codebase.
This commit is contained in:
Brett Smith 2017-12-31 18:52:30 -05:00
parent 475de6db4e
commit db59d2fc8c
13 changed files with 68 additions and 105 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

14
CODE.rst
View file

@ -39,9 +39,6 @@ Class method ``can_handle(source_file)``
``__iter__()``
Returns a iterator of entry data dicts.
Class attribute ``TEMPLATE_KEY``
A string with the full key to load the corresponding template from the user's configuration (e.g., ``'template patreon income'``).
Hooks
~~~~~
@ -62,17 +59,6 @@ Hooks make arbitrary transformations to entry data dicts. Every entry data dict
Class attribute ``KIND``
This should be one of the values of the ``hooks.HOOK_KINDS`` enum. This information determines what order hooks run in.
Templates
~~~~~~~~~
Templates receive entry data dicts and format them into final output entries.
``__init__(template_str)``
Initializes the template from a single string, as read from the user's configuration.
``render(entry_data)``
Returns a string with the output entry, using the given entry data.
Loading importers and hooks
---------------------------

18
README.rst
View file

@ -29,7 +29,7 @@ A template looks like a Ledger entry with a couple of differences:
Here's a simple template for Patreon patron payments::
[DEFAULT]
template patreon income =
patreon income ledger entry =
;Tag: Value
Income:Patreon -{amount}
Accrued:Accounts Receivable:Patreon {amount}
@ -38,7 +38,7 @@ Let's walk through this line by line.
Every setting in your configuration file has to be in a section. ``[DEFAULT]`` is the default section, and import2ledger reads configuration settings from here if you don't specify another one. This documentation explains how to use sections later.
``template patreon income =`` specifies which entry template this is. Every template is found from a setting with a name in the pattern ``template <SOURCE> <TYPE>``. The remaining lines are indented further than this name; this defines a multiline value. Don't worry about the exact indentation of your template; import2ledger will indent its output nicely.
``patreon income ledger entry =`` specifies which entry template this is. Every template is found from a setting with a name in the pattern ``<SOURCE> <TYPE> ledger entry``. The remaining lines are indented further than this name; this defines a multiline value. Don't worry about the exact indentation of your template; import2ledger will indent its output nicely.
The first line of the template is a Ledger tag. The program will leave all kinds of tags and Ledger comments alone, except to indent them nicely.
@ -104,13 +104,13 @@ You can define the following templates.
Amazon Affiliate Program
^^^^^^^^^^^^^^^^^^^^^^^^
``template amazon earnings``
``amazon earnings ledger entry``
Imports one transaction per month, summarizing all earnings over the month. Generated from Amazon's "Fee Earnings" report CSV.
Benevity
^^^^^^^^
``template benevity payments``
``benevity donation ledger entry``
Imports one transaction per row in Benevity's donations report CSV.
This template can use these variables:
@ -140,16 +140,16 @@ Benevity
Patreon
^^^^^^^
``template patreon income``
``patreon income ledger entry``
Imports one transaction per patron per month. Generated from Patreon's monthly patron report CSVs.
``template patreon cardfees``
``patreon cardfees ledger entry``
Imports one expense transaction per month for that month's credit card fees. Generated from Patreon's earnings report CSV.
``template patreon svcfees``
``patreon servicefees ledger entry``
Imports one expense transaction per month for that month's Patreon service fees. Generated from Patreon's earnings report CSV.
``template patreon vat``
``patreon vat ledger entry``
Imports one transaction per country per month each time Patreon withheld VAT. Generated from Patreon's VAT report CSV.
This template can use these variables:
@ -166,7 +166,7 @@ Patreon
Stripe
^^^^^^
``template stripe payments``
``stripe payment ledger entry``
Imports one transaction per payment. Generated from Stripe's payments CSV export.
This template can use these variables:

11
TODO.rst
View file

@ -4,19 +4,10 @@ TODO
Template multiplexing with action hooks
---------------------------------------
The big idea: make it easier for hooks to customize *what* template(s) are rendered by moving more of the process into hooks—including template rendering itself.
Required:
* Make the main loop seed the entry data with information about the importer used.
* Move template rendering into a hook, where the template to load is determined by a value in the entry data.
Extra customizations after that's done:
* Add a hook that simply reads information from a configuration file section ``[template variables]`` and adds it to the entry data.
* Add a hook that changes what template to use based on other entry data. (This needs more specification.)
OR, should this be a hook that just adds more entry data, and trusts the Ledger entry template to use it?
New importers
-------------

3
import2ledger/__main__.py
View file

@ -34,8 +34,9 @@ class FileImporter:
'source_stem': in_path.stem,
}
for importer in importers:
source_vars['importer_class'] = importer.__name__
source_vars['importer_module'] = importer.__module__
in_file.seek(0)
source_vars['template'] = importer.TEMPLATE_KEY
for entry_data in importer(in_file):
entry_data = collections.ChainMap(entry_data, source_vars)
for hook in self.hooks:

13
import2ledger/hooks/ledger_entry.py
View file

@ -283,7 +283,7 @@ class LedgerEntryHook:
template_s = section_config[config_key]
except KeyError:
raise errors.UserInputConfigurationError(
"template not defined in [{}]".format(section_name),
"Ledger template not defined in [{}]".format(section_name),
config_key,
)
return Template(
@ -297,9 +297,16 @@ class LedgerEntryHook:
def run(self, entry_data):
try:
template = self._load_template(self.config, None, entry_data['template'])
template_key = entry_data['ledger template']
except KeyError:
template_key = '{} {} ledger entry'.format(
strparse.rslice_words(entry_data['importer_module'], -1, '.', 1),
entry_data['importer_class'][:-8].lower(),
)
try:
template = self._load_template(self.config, None, template_key)
except errors.UserInputConfigurationError as error:
if error.strerror.startswith('template not defined '):
if error.strerror.startswith('Ledger template not defined '):
have_template = False
else:
raise

1
import2ledger/importers/amazon.py
View file

@ -43,7 +43,6 @@ class EarningsImporter(_csv.CSVImporterBase):
'Date Shipped': '1979-07-09',
}
NEEDED_FIELDS = frozenset(SENTINEL_ROW.keys())
TEMPLATE_KEY = 'template amazon earnings'
ENTRY_SEED = {
'currency': 'USD',
'payee': 'Amazon',

3
import2ledger/importers/benevity.py
View file

@ -1,7 +1,7 @@
from . import _csv
from .. import strparse
class PaymentImporter(_csv.CSVImporterBase):
class DonationsImporter(_csv.CSVImporterBase):
HEADER_FIELDS = {
'Currency': 'currency',
'Disbursement ID': 'disbursement_id',
@ -26,7 +26,6 @@ class PaymentImporter(_csv.CSVImporterBase):
'Transaction ID': 'transaction_id',
'Donation Frequency': 'frequency',
}
TEMPLATE_KEY = 'template benevity payments'
DATE_FMT = '%Y-%m-%d'
NOT_SHARED = 'Not shared by donor'

19
import2ledger/importers/nbpy2017.py
View file

@ -121,7 +121,10 @@ def _parse_invoice(parser_class, source_file):
except AttributeError:
return None
class ImporterBase:
class InvoiceImporter:
INVOICE_CLASS = Invoice2017
LEDGER_TEMPLATE_KEY_FMT = 'nbpy2017 {0} ledger entry'
@classmethod
def _parse_invoice(cls, source_file):
return _parse_invoice(cls.INVOICE_CLASS, source_file)
@ -135,17 +138,5 @@ class ImporterBase:
def __iter__(self):
for entry in self.invoice:
if entry['status'] == self.YIELD_STATUS:
entry['ledger entry'] = self.LEDGER_TEMPLATE_KEY_FMT.format(entry['status'].lower())
yield entry
class Invoice2017Importer(ImporterBase):
TEMPLATE_KEY = 'template nbpy2017 invoice'
INVOICE_CLASS = Invoice2017
YIELD_STATUS = STATUS_INVOICED
class Payment2017Importer(ImporterBase):
TEMPLATE_KEY = 'template nbpy2017 payment'
INVOICE_CLASS = Invoice2017
YIELD_STATUS = STATUS_PAID

8
import2ledger/importers/patreon.py
View file

@ -17,7 +17,6 @@ class IncomeImporter(_csv.CSVImporterBase):
ENTRY_SEED = {
'currency': 'USD',
}
TEMPLATE_KEY = 'template patreon income'
def __init__(self, input_file):
super().__init__(input_file)
@ -48,16 +47,14 @@ class FeeImporterBase(_csv.CSVImporterBase):
}
class PatreonFeeImporter(FeeImporterBase):
class ServiceFeesImporter(FeeImporterBase):
AMOUNT_FIELD = 'Patreon Fee'
NEEDED_FIELDS = frozenset(['Month', AMOUNT_FIELD])
TEMPLATE_KEY = 'template patreon svcfees'
class CardFeeImporter(FeeImporterBase):
class CardFeesImporter(FeeImporterBase):
AMOUNT_FIELD = 'Processing Fees'
NEEDED_FIELDS = frozenset(['Month', AMOUNT_FIELD])
TEMPLATE_KEY = 'template patreon cardfees'
class VATImporter(FeeImporterBase):
@ -67,4 +64,3 @@ class VATImporter(FeeImporterBase):
'Country Code': 'country_code',
'Country Name': 'country_name',
}
TEMPLATE_KEY = 'template patreon vat'

1
import2ledger/importers/stripe.py
View file

@ -17,7 +17,6 @@ class PaymentImporter(_csv.CSVImporterBase):
'Description': 'description',
'id': 'payment_id',
}
TEMPLATE_KEY = 'template stripe payments'
DATE_FMT = '%Y-%m-%d'
def _read_row(self, row):

74
tests/data/imports.yml
View file

@ -11,7 +11,7 @@
currency: USD
- source: PatreonEarnings.csv
importer: patreon.PatreonFeeImporter
importer: patreon.ServiceFeesImporter
expect:
- payee: Patreon
date: !!python/object/apply:datetime.date [2017, 9, 1]
@ -23,7 +23,7 @@
currency: USD
- source: PatreonEarnings.csv
importer: patreon.CardFeeImporter
importer: patreon.CardFeesImporter
expect:
- payee: Patreon
date: !!python/object/apply:datetime.date [2017, 9, 1]
@ -83,9 +83,10 @@
description: "Payment for invoice #100"
- source: nbpy2017a.html
importer: nbpy2017.Invoice2017Importer
importer: nbpy2017.InvoiceImporter
expect:
- payee: Python Person A
ledger entry: nbpy2017 invoice ledger entry
date: !!python/object/apply:datetime.date [2017, 10, 19]
amount: !!python/object/apply:decimal.Decimal ["80.00"]
tickets_sold: !!python/object/apply:decimal.Decimal ["1"]
@ -96,41 +97,8 @@
status: Invoice
invoice_id: "83"
invoice_date: !!python/object/apply:datetime.date [2017, 10, 19]
- source: nbpy2017b.html
importer: nbpy2017.Invoice2017Importer
expect:
- payee: Python Person B
date: !!python/object/apply:datetime.date [2017, 12, 3]
amount: !!python/object/apply:decimal.Decimal ["50.00"]
tickets_sold: !!python/object/apply:decimal.Decimal ["1"]
ticket_rate: !!python/object/apply:decimal.Decimal ["42.50"]
shirts_sold: !!python/object/apply:decimal.Decimal ["0"]
shirt_rate: !!python/object/apply:decimal.Decimal ["25.50"]
status: Invoice
currency: USD
invoice_date: !!python/object/apply:datetime.date [2017, 12, 3]
invoice_id: "304"
- source: nbpy2017c.html
importer: nbpy2017.Invoice2017Importer
expect:
- payee: Python Person C
date: !!python/object/apply:datetime.date [2017, 10, 5]
amount: !!python/object/apply:decimal.Decimal ["55.00"]
tickets_sold: !!python/object/apply:decimal.Decimal ["1"]
ticket_rate: !!python/object/apply:decimal.Decimal ["21.25"]
shirts_sold: !!python/object/apply:decimal.Decimal ["1"]
shirt_rate: !!python/object/apply:decimal.Decimal ["25.50"]
status: Invoice
currency: USD
invoice_date: !!python/object/apply:datetime.date [2017, 10, 5]
invoice_id: "11"
- source: nbpy2017a.html
importer: nbpy2017.Payment2017Importer
expect:
- payee: Python Person A
ledger entry: nbpy2017 payment ledger entry
date: !!python/object/apply:datetime.date [2017, 10, 19]
amount: !!python/object/apply:decimal.Decimal ["80.00"]
tickets_sold: !!python/object/apply:decimal.Decimal ["1"]
@ -145,9 +113,22 @@
stripe_id: ch_ahr0ue8lai1ohqu4Gei4Biem
- source: nbpy2017b.html
importer: nbpy2017.Payment2017Importer
importer: nbpy2017.InvoiceImporter
expect:
- payee: Python Person B
ledger entry: nbpy2017 invoice ledger entry
date: !!python/object/apply:datetime.date [2017, 12, 3]
amount: !!python/object/apply:decimal.Decimal ["50.00"]
tickets_sold: !!python/object/apply:decimal.Decimal ["1"]
ticket_rate: !!python/object/apply:decimal.Decimal ["42.50"]
shirts_sold: !!python/object/apply:decimal.Decimal ["0"]
shirt_rate: !!python/object/apply:decimal.Decimal ["25.50"]
status: Invoice
currency: USD
invoice_date: !!python/object/apply:datetime.date [2017, 12, 3]
invoice_id: "304"
- payee: Python Person B
ledger entry: nbpy2017 payment ledger entry
date: !!python/object/apply:datetime.date [2017, 12, 3]
amount: !!python/object/apply:decimal.Decimal ["50.00"]
tickets_sold: !!python/object/apply:decimal.Decimal ["1"]
@ -162,9 +143,22 @@
invoice_id: "304"
- source: nbpy2017c.html
importer: nbpy2017.Payment2017Importer
importer: nbpy2017.InvoiceImporter
expect:
- payee: Python Person C
ledger entry: nbpy2017 invoice ledger entry
date: !!python/object/apply:datetime.date [2017, 10, 5]
amount: !!python/object/apply:decimal.Decimal ["55.00"]
tickets_sold: !!python/object/apply:decimal.Decimal ["1"]
ticket_rate: !!python/object/apply:decimal.Decimal ["21.25"]
shirts_sold: !!python/object/apply:decimal.Decimal ["1"]
shirt_rate: !!python/object/apply:decimal.Decimal ["25.50"]
status: Invoice
currency: USD
invoice_date: !!python/object/apply:datetime.date [2017, 10, 5]
invoice_id: "11"
- payee: Python Person C
ledger entry: nbpy2017 payment ledger entry
date: !!python/object/apply:datetime.date [2017, 10, 5]
amount: !!python/object/apply:decimal.Decimal ["55.00"]
tickets_sold: !!python/object/apply:decimal.Decimal ["1"]
@ -191,7 +185,7 @@
currency: USD
- source: Benevity.csv
importer: benevity.PaymentImporter
importer: benevity.DonationsImporter
expect:
- date: !!python/object/apply:datetime.date [2017, 10, 28]
currency: USD

4
tests/data/test_main.ini
View file

@ -4,10 +4,10 @@ loglevel = critical
signed_currencies = USD
[One]
template patreon cardfees =
patreon cardfees ledger entry =
Accrued:Accounts Receivable -{amount}
Expenses:Fees:Credit Card {amount}
template patreon svcfees =
patreon servicefees ledger entry =
;SourcePath: {source_abspath}
;SourceName: {source_name}
Accrued:Accounts Receivable -{amount}

2
tests/test_hook_ledger_entry.py
View file

@ -27,7 +27,7 @@ def template_vars(payee, amount, currency='USD', date=DATE, other_vars=None):
'currency': currency,
'date': date,
'payee': payee,
'template': 'template',
'ledger template': 'template',
}
if other_vars is None:
return call_vars