From db59d2fc8ceb2964b25865137df31419e7899fad Mon Sep 17 00:00:00 2001 From: Brett Smith Date: Sun, 31 Dec 2017 18:52:30 -0500 Subject: [PATCH] hooks.ledger_entry: Look up templates dynamically. If there's a 'ledger entry' key in the entry data, use that value as the name of the template to load. Thanks to this, nbpy2017 could collapse multiple importers into one. Otherwise, build a default template name based on the importer source, and try to use that. All the configuration names now end with "ledger entry" instead of starting with "template". This makes it clearer what they're for, in case we support other kinds of output templates in the future. I ended up changing the names of some of the importers so the default template name was nice, rather than specifying template names for all of them, to reduce the amount of name discrepancies across the codebase. --- CODE.rst | 14 ------ README.rst | 18 +++---- TODO.rst | 11 +---- import2ledger/__main__.py | 3 +- import2ledger/hooks/ledger_entry.py | 13 +++-- import2ledger/importers/amazon.py | 1 - import2ledger/importers/benevity.py | 3 +- import2ledger/importers/nbpy2017.py | 21 +++----- import2ledger/importers/patreon.py | 8 +--- import2ledger/importers/stripe.py | 1 - tests/data/imports.yml | 74 +++++++++++++---------------- tests/data/test_main.ini | 4 +- tests/test_hook_ledger_entry.py | 2 +- 13 files changed, 68 insertions(+), 105 deletions(-) diff --git a/CODE.rst b/CODE.rst index abeafb0..2f092d5 100644 --- a/CODE.rst +++ b/CODE.rst @@ -39,9 +39,6 @@ Class method ``can_handle(source_file)`` ``__iter__()`` Returns a iterator of entry data dicts. -Class attribute ``TEMPLATE_KEY`` - A string with the full key to load the corresponding template from the user's configuration (e.g., ``'template patreon income'``). - Hooks ~~~~~ @@ -62,17 +59,6 @@ Hooks make arbitrary transformations to entry data dicts. Every entry data dict Class attribute ``KIND`` This should be one of the values of the ``hooks.HOOK_KINDS`` enum. This information determines what order hooks run in. -Templates -~~~~~~~~~ - -Templates receive entry data dicts and format them into final output entries. - -``__init__(template_str)`` - Initializes the template from a single string, as read from the user's configuration. - -``render(entry_data)`` - Returns a string with the output entry, using the given entry data. - Loading importers and hooks --------------------------- diff --git a/README.rst b/README.rst index b9a0c46..1cdf6f5 100644 --- a/README.rst +++ b/README.rst @@ -29,7 +29,7 @@ A template looks like a Ledger entry with a couple of differences: Here's a simple template for Patreon patron payments:: [DEFAULT] - template patreon income = + patreon income ledger entry = ;Tag: Value Income:Patreon -{amount} Accrued:Accounts Receivable:Patreon {amount} @@ -38,7 +38,7 @@ Let's walk through this line by line. Every setting in your configuration file has to be in a section. ``[DEFAULT]`` is the default section, and import2ledger reads configuration settings from here if you don't specify another one. This documentation explains how to use sections later. -``template patreon income =`` specifies which entry template this is. Every template is found from a setting with a name in the pattern ``template ``. The remaining lines are indented further than this name; this defines a multiline value. Don't worry about the exact indentation of your template; import2ledger will indent its output nicely. +``patreon income ledger entry =`` specifies which entry template this is. Every template is found from a setting with a name in the pattern `` ledger entry``. The remaining lines are indented further than this name; this defines a multiline value. Don't worry about the exact indentation of your template; import2ledger will indent its output nicely. The first line of the template is a Ledger tag. The program will leave all kinds of tags and Ledger comments alone, except to indent them nicely. @@ -104,13 +104,13 @@ You can define the following templates. Amazon Affiliate Program ^^^^^^^^^^^^^^^^^^^^^^^^ -``template amazon earnings`` +``amazon earnings ledger entry`` Imports one transaction per month, summarizing all earnings over the month. Generated from Amazon's "Fee Earnings" report CSV. Benevity ^^^^^^^^ -``template benevity payments`` +``benevity donation ledger entry`` Imports one transaction per row in Benevity's donations report CSV. This template can use these variables: @@ -140,16 +140,16 @@ Benevity Patreon ^^^^^^^ -``template patreon income`` +``patreon income ledger entry`` Imports one transaction per patron per month. Generated from Patreon's monthly patron report CSVs. -``template patreon cardfees`` +``patreon cardfees ledger entry`` Imports one expense transaction per month for that month's credit card fees. Generated from Patreon's earnings report CSV. -``template patreon svcfees`` +``patreon servicefees ledger entry`` Imports one expense transaction per month for that month's Patreon service fees. Generated from Patreon's earnings report CSV. -``template patreon vat`` +``patreon vat ledger entry`` Imports one transaction per country per month each time Patreon withheld VAT. Generated from Patreon's VAT report CSV. This template can use these variables: @@ -166,7 +166,7 @@ Patreon Stripe ^^^^^^ -``template stripe payments`` +``stripe payment ledger entry`` Imports one transaction per payment. Generated from Stripe's payments CSV export. This template can use these variables: diff --git a/TODO.rst b/TODO.rst index 0185219..8b242c9 100644 --- a/TODO.rst +++ b/TODO.rst @@ -4,19 +4,10 @@ TODO Template multiplexing with action hooks --------------------------------------- -The big idea: make it easier for hooks to customize *what* template(s) are rendered by moving more of the process into hooks—including template rendering itself. - -Required: - -* Make the main loop seed the entry data with information about the importer used. - -* Move template rendering into a hook, where the template to load is determined by a value in the entry data. - -Extra customizations after that's done: - * Add a hook that simply reads information from a configuration file section ``[template variables]`` and adds it to the entry data. * Add a hook that changes what template to use based on other entry data. (This needs more specification.) + OR, should this be a hook that just adds more entry data, and trusts the Ledger entry template to use it? New importers ------------- diff --git a/import2ledger/__main__.py b/import2ledger/__main__.py index 57ec1e9..5d7896d 100644 --- a/import2ledger/__main__.py +++ b/import2ledger/__main__.py @@ -34,8 +34,9 @@ class FileImporter: 'source_stem': in_path.stem, } for importer in importers: + source_vars['importer_class'] = importer.__name__ + source_vars['importer_module'] = importer.__module__ in_file.seek(0) - source_vars['template'] = importer.TEMPLATE_KEY for entry_data in importer(in_file): entry_data = collections.ChainMap(entry_data, source_vars) for hook in self.hooks: diff --git a/import2ledger/hooks/ledger_entry.py b/import2ledger/hooks/ledger_entry.py index 46b46b2..97584d9 100644 --- a/import2ledger/hooks/ledger_entry.py +++ b/import2ledger/hooks/ledger_entry.py @@ -283,7 +283,7 @@ class LedgerEntryHook: template_s = section_config[config_key] except KeyError: raise errors.UserInputConfigurationError( - "template not defined in [{}]".format(section_name), + "Ledger template not defined in [{}]".format(section_name), config_key, ) return Template( @@ -297,9 +297,16 @@ class LedgerEntryHook: def run(self, entry_data): try: - template = self._load_template(self.config, None, entry_data['template']) + template_key = entry_data['ledger template'] + except KeyError: + template_key = '{} {} ledger entry'.format( + strparse.rslice_words(entry_data['importer_module'], -1, '.', 1), + entry_data['importer_class'][:-8].lower(), + ) + try: + template = self._load_template(self.config, None, template_key) except errors.UserInputConfigurationError as error: - if error.strerror.startswith('template not defined '): + if error.strerror.startswith('Ledger template not defined '): have_template = False else: raise diff --git a/import2ledger/importers/amazon.py b/import2ledger/importers/amazon.py index 6927e97..fc8202e 100644 --- a/import2ledger/importers/amazon.py +++ b/import2ledger/importers/amazon.py @@ -43,7 +43,6 @@ class EarningsImporter(_csv.CSVImporterBase): 'Date Shipped': '1979-07-09', } NEEDED_FIELDS = frozenset(SENTINEL_ROW.keys()) - TEMPLATE_KEY = 'template amazon earnings' ENTRY_SEED = { 'currency': 'USD', 'payee': 'Amazon', diff --git a/import2ledger/importers/benevity.py b/import2ledger/importers/benevity.py index c304103..8b29ebd 100644 --- a/import2ledger/importers/benevity.py +++ b/import2ledger/importers/benevity.py @@ -1,7 +1,7 @@ from . import _csv from .. import strparse -class PaymentImporter(_csv.CSVImporterBase): +class DonationsImporter(_csv.CSVImporterBase): HEADER_FIELDS = { 'Currency': 'currency', 'Disbursement ID': 'disbursement_id', @@ -26,7 +26,6 @@ class PaymentImporter(_csv.CSVImporterBase): 'Transaction ID': 'transaction_id', 'Donation Frequency': 'frequency', } - TEMPLATE_KEY = 'template benevity payments' DATE_FMT = '%Y-%m-%d' NOT_SHARED = 'Not shared by donor' diff --git a/import2ledger/importers/nbpy2017.py b/import2ledger/importers/nbpy2017.py index d871fae..b99a7da 100644 --- a/import2ledger/importers/nbpy2017.py +++ b/import2ledger/importers/nbpy2017.py @@ -121,7 +121,10 @@ def _parse_invoice(parser_class, source_file): except AttributeError: return None -class ImporterBase: +class InvoiceImporter: + INVOICE_CLASS = Invoice2017 + LEDGER_TEMPLATE_KEY_FMT = 'nbpy2017 {0} ledger entry' + @classmethod def _parse_invoice(cls, source_file): return _parse_invoice(cls.INVOICE_CLASS, source_file) @@ -135,17 +138,5 @@ class ImporterBase: def __iter__(self): for entry in self.invoice: - if entry['status'] == self.YIELD_STATUS: - yield entry - - -class Invoice2017Importer(ImporterBase): - TEMPLATE_KEY = 'template nbpy2017 invoice' - INVOICE_CLASS = Invoice2017 - YIELD_STATUS = STATUS_INVOICED - - -class Payment2017Importer(ImporterBase): - TEMPLATE_KEY = 'template nbpy2017 payment' - INVOICE_CLASS = Invoice2017 - YIELD_STATUS = STATUS_PAID + entry['ledger entry'] = self.LEDGER_TEMPLATE_KEY_FMT.format(entry['status'].lower()) + yield entry diff --git a/import2ledger/importers/patreon.py b/import2ledger/importers/patreon.py index 1df117f..ae347be 100644 --- a/import2ledger/importers/patreon.py +++ b/import2ledger/importers/patreon.py @@ -17,7 +17,6 @@ class IncomeImporter(_csv.CSVImporterBase): ENTRY_SEED = { 'currency': 'USD', } - TEMPLATE_KEY = 'template patreon income' def __init__(self, input_file): super().__init__(input_file) @@ -48,16 +47,14 @@ class FeeImporterBase(_csv.CSVImporterBase): } -class PatreonFeeImporter(FeeImporterBase): +class ServiceFeesImporter(FeeImporterBase): AMOUNT_FIELD = 'Patreon Fee' NEEDED_FIELDS = frozenset(['Month', AMOUNT_FIELD]) - TEMPLATE_KEY = 'template patreon svcfees' -class CardFeeImporter(FeeImporterBase): +class CardFeesImporter(FeeImporterBase): AMOUNT_FIELD = 'Processing Fees' NEEDED_FIELDS = frozenset(['Month', AMOUNT_FIELD]) - TEMPLATE_KEY = 'template patreon cardfees' class VATImporter(FeeImporterBase): @@ -67,4 +64,3 @@ class VATImporter(FeeImporterBase): 'Country Code': 'country_code', 'Country Name': 'country_name', } - TEMPLATE_KEY = 'template patreon vat' diff --git a/import2ledger/importers/stripe.py b/import2ledger/importers/stripe.py index 99b03c4..1c91e26 100644 --- a/import2ledger/importers/stripe.py +++ b/import2ledger/importers/stripe.py @@ -17,7 +17,6 @@ class PaymentImporter(_csv.CSVImporterBase): 'Description': 'description', 'id': 'payment_id', } - TEMPLATE_KEY = 'template stripe payments' DATE_FMT = '%Y-%m-%d' def _read_row(self, row): diff --git a/tests/data/imports.yml b/tests/data/imports.yml index 1b7f967..420a719 100644 --- a/tests/data/imports.yml +++ b/tests/data/imports.yml @@ -11,7 +11,7 @@ currency: USD - source: PatreonEarnings.csv - importer: patreon.PatreonFeeImporter + importer: patreon.ServiceFeesImporter expect: - payee: Patreon date: !!python/object/apply:datetime.date [2017, 9, 1] @@ -23,7 +23,7 @@ currency: USD - source: PatreonEarnings.csv - importer: patreon.CardFeeImporter + importer: patreon.CardFeesImporter expect: - payee: Patreon date: !!python/object/apply:datetime.date [2017, 9, 1] @@ -83,9 +83,10 @@ description: "Payment for invoice #100" - source: nbpy2017a.html - importer: nbpy2017.Invoice2017Importer + importer: nbpy2017.InvoiceImporter expect: - payee: Python Person A + ledger entry: nbpy2017 invoice ledger entry date: !!python/object/apply:datetime.date [2017, 10, 19] amount: !!python/object/apply:decimal.Decimal ["80.00"] tickets_sold: !!python/object/apply:decimal.Decimal ["1"] @@ -96,41 +97,8 @@ status: Invoice invoice_id: "83" invoice_date: !!python/object/apply:datetime.date [2017, 10, 19] - -- source: nbpy2017b.html - importer: nbpy2017.Invoice2017Importer - expect: - - payee: Python Person B - date: !!python/object/apply:datetime.date [2017, 12, 3] - amount: !!python/object/apply:decimal.Decimal ["50.00"] - tickets_sold: !!python/object/apply:decimal.Decimal ["1"] - ticket_rate: !!python/object/apply:decimal.Decimal ["42.50"] - shirts_sold: !!python/object/apply:decimal.Decimal ["0"] - shirt_rate: !!python/object/apply:decimal.Decimal ["25.50"] - status: Invoice - currency: USD - invoice_date: !!python/object/apply:datetime.date [2017, 12, 3] - invoice_id: "304" - -- source: nbpy2017c.html - importer: nbpy2017.Invoice2017Importer - expect: - - payee: Python Person C - date: !!python/object/apply:datetime.date [2017, 10, 5] - amount: !!python/object/apply:decimal.Decimal ["55.00"] - tickets_sold: !!python/object/apply:decimal.Decimal ["1"] - ticket_rate: !!python/object/apply:decimal.Decimal ["21.25"] - shirts_sold: !!python/object/apply:decimal.Decimal ["1"] - shirt_rate: !!python/object/apply:decimal.Decimal ["25.50"] - status: Invoice - currency: USD - invoice_date: !!python/object/apply:datetime.date [2017, 10, 5] - invoice_id: "11" - -- source: nbpy2017a.html - importer: nbpy2017.Payment2017Importer - expect: - payee: Python Person A + ledger entry: nbpy2017 payment ledger entry date: !!python/object/apply:datetime.date [2017, 10, 19] amount: !!python/object/apply:decimal.Decimal ["80.00"] tickets_sold: !!python/object/apply:decimal.Decimal ["1"] @@ -145,9 +113,22 @@ stripe_id: ch_ahr0ue8lai1ohqu4Gei4Biem - source: nbpy2017b.html - importer: nbpy2017.Payment2017Importer + importer: nbpy2017.InvoiceImporter expect: - payee: Python Person B + ledger entry: nbpy2017 invoice ledger entry + date: !!python/object/apply:datetime.date [2017, 12, 3] + amount: !!python/object/apply:decimal.Decimal ["50.00"] + tickets_sold: !!python/object/apply:decimal.Decimal ["1"] + ticket_rate: !!python/object/apply:decimal.Decimal ["42.50"] + shirts_sold: !!python/object/apply:decimal.Decimal ["0"] + shirt_rate: !!python/object/apply:decimal.Decimal ["25.50"] + status: Invoice + currency: USD + invoice_date: !!python/object/apply:datetime.date [2017, 12, 3] + invoice_id: "304" + - payee: Python Person B + ledger entry: nbpy2017 payment ledger entry date: !!python/object/apply:datetime.date [2017, 12, 3] amount: !!python/object/apply:decimal.Decimal ["50.00"] tickets_sold: !!python/object/apply:decimal.Decimal ["1"] @@ -162,9 +143,22 @@ invoice_id: "304" - source: nbpy2017c.html - importer: nbpy2017.Payment2017Importer + importer: nbpy2017.InvoiceImporter expect: - payee: Python Person C + ledger entry: nbpy2017 invoice ledger entry + date: !!python/object/apply:datetime.date [2017, 10, 5] + amount: !!python/object/apply:decimal.Decimal ["55.00"] + tickets_sold: !!python/object/apply:decimal.Decimal ["1"] + ticket_rate: !!python/object/apply:decimal.Decimal ["21.25"] + shirts_sold: !!python/object/apply:decimal.Decimal ["1"] + shirt_rate: !!python/object/apply:decimal.Decimal ["25.50"] + status: Invoice + currency: USD + invoice_date: !!python/object/apply:datetime.date [2017, 10, 5] + invoice_id: "11" + - payee: Python Person C + ledger entry: nbpy2017 payment ledger entry date: !!python/object/apply:datetime.date [2017, 10, 5] amount: !!python/object/apply:decimal.Decimal ["55.00"] tickets_sold: !!python/object/apply:decimal.Decimal ["1"] @@ -191,7 +185,7 @@ currency: USD - source: Benevity.csv - importer: benevity.PaymentImporter + importer: benevity.DonationsImporter expect: - date: !!python/object/apply:datetime.date [2017, 10, 28] currency: USD diff --git a/tests/data/test_main.ini b/tests/data/test_main.ini index 5babf5c..1d4dfb2 100644 --- a/tests/data/test_main.ini +++ b/tests/data/test_main.ini @@ -4,10 +4,10 @@ loglevel = critical signed_currencies = USD [One] -template patreon cardfees = +patreon cardfees ledger entry = Accrued:Accounts Receivable -{amount} Expenses:Fees:Credit Card {amount} -template patreon svcfees = +patreon servicefees ledger entry = ;SourcePath: {source_abspath} ;SourceName: {source_name} Accrued:Accounts Receivable -{amount} diff --git a/tests/test_hook_ledger_entry.py b/tests/test_hook_ledger_entry.py index e748be0..feb876c 100644 --- a/tests/test_hook_ledger_entry.py +++ b/tests/test_hook_ledger_entry.py @@ -27,7 +27,7 @@ def template_vars(payee, amount, currency='USD', date=DATE, other_vars=None): 'currency': currency, 'date': date, 'payee': payee, - 'template': 'template', + 'ledger template': 'template', } if other_vars is None: return call_vars