Everything it said was a problem has been done legitimately in our books at
one point or another.
* Variation in contract can happen in different line items of an invoice or
"group of contractor" situations.
* Variation in cost can happen because one invoice spans a period of time,
like donation matching programs. There is probably still value in a tool
that checks to make sure we use consistent rates each day, but that
affects all kinds of transactions, not just accruals, so it would be
done better in a separate tool.
* Variation in account happens because invoices legitimately span accrual
accounts, like donation matching programs with fees payable.
So: it's gone, good riddance.
This adds almost all the metadata that's relevant to accruals.
I considered adding statement, but that cuased rows to get spaced out a lot,
and statement's kind of a low-value column, so I decided against it.
Ultimately I would like to make this configurable but that's for the
future.
It is more common than I realized that we split an invoice by
entity on the accrual side, so this supports that better.
It still disregards inconsistency between accrual entity and payment entity
for reporting purposes, to help keep reporting clean around automatic
imports.
The changes to BaseReport._report shook out because at this point, the group
key is effectively arbitrary and shouldn't be used for any reporting
purposes.
This works fine with how we're currently using it, makes transformation
methods easier to implement, and avoids potential bugs where a balance is
initialized with a bad mapping.
So far we've been implicitly relying on this by the user passing search
terms that filter out the opening balance transaction. That will stop
happening with the aging report, so we need to do it ourselves.
This was an early mistake, it makes data consistency mistakes too
easy, and I only used it once so far in actual code. Going to fix
this now so I can more safely build on top of this data structure.
It turns out the provided implementation gets us most of the way there,
we just needed to add handling for the special case of zero balances.
Now it's confirmed with tests.
The test changes make them order-sensitive, which they should be.
It's important that our loader methods return date-sorted entries
just like Beancount itself would.
This makes the output more useful for broad searches like on an
entity. Invoices that cross FY boundaries will appear to be paid
without being accrued, and so would appear when we were just
filtering zeroed-out invoices.
If we integrate the aging report into this module in the future,
that'll need to follow different logic, and just filter out
zeroed-out invoices. But the basic balance report and outgoing
report are more workaday tools, where more filtering makes them
more useful.
It makes sense to let the bookkeeper skip validations in situations
where the metadata requires information that might not be available
when entered. It does not make sense to skip validations that *must*
be available and affect the structure of the books, like project and
entity.
This commit ensures every plugin hook has a test for flagged
transactions, even for hooks that currently have the desired
behavior where no code changes were required for the test to
pass.
Building a string and loading it means Beancount can never cache any
load. It only caches top-level file loads because options in the
top-level file can change the semantics of included entries.
Instead use load_file as much as possible, and filter entries as
needed.
The old loading strategy didn't load options, which yielded some
spurious errors. It also created awkward duplication of plugin
information in the code as well as the books.
Implement a new loading strategy that works by reading one of the
"main files" under the books/ subdirectory and includes entries
for additional FYs beyond that.
This is still not ideal in a lot of ways. In particular, Beancount can't
cache any results, causing any load to be slower than it theoretically could
be. I expect more commits to follow. But some of them might require
restructuring the books, and that should happen separately.
This module basically reimplements the old payment-report+income-report,
in a single tool (after setting aside some of the checks that have moved
to the plugin). The aging report can be implemented here too when we need
that.
I believe we still want this in principle, but we're not currently enforcing
it the way I thought we were, and we very regularly write Payables without
this supporting documentation (for trip reimbursement, regular service fees,
etc.). Enforcing this now would be way too noisy in the books, we need to
devise a separate plan to enforce this if we want it.
This makes methods like _replace available in real code, and caught the
bug where we can't use @functools.lru_cache with Transaction arguments,
because they're unhashable due to their mutable members.
This gets closer to our real intentions: anything that checks link
metadata should check rt-id. MetaRepoLinks is the exception, not
the rule, in ignoring rt-id.
Unfortunately this is becoming more trouble than it's worth as we
call more and more untyped Beancount functions.
disallow_untyped_defs provides most of the value of what we really
want here, so go ahead and turn this off.
Ultimately I would like to make it possible to configure the software
entirely through this file, rather than the hodgepodge system we have
now. But that can come later.
payment-report and accrual-report query to find the last date a
series of postings had a non/zero balance. This method is a good
building block for that.
This is something reporting tools will want a lot. This will make it
easier for them to look at just postings without worrying about the
parent transaction.
See docstring for full rationale. This greatly reduces the need for other
plugin code to handle the case of `post.units.number is None`, eliminating
the need for entire methods and letting it do plain numeric comparisons.
This lets us import the plugin module without importing all of the included
hooks. This provides better isolation and error reporting in case there's
something like a syntax problem in one of the hooks: it doesn't cause
importing any plugin module to fail.
Extend the base class from checking 1 metadata value to checking N.
This is preparation for RT#10643, letting payables be documented with
invoice or contract.
This does unify error reporting, because now we always report all
type/invalid value errors *plus* a missing error if appropriate.
I think this consistency and thoroughness is appropriate, although
it did require some adjustments to the tests.
Python 3.6 does not implement __class_getitem__, and because of that
it's not possible to introspect when things like Hook[Transaction]
are called. Sidestep the issue with a more explicit assignment.
When we send checks, we don't have a check document anywhere (for
security reasons), we just note the check number. Update the
validation to match. RT#10507.
The main impetus of this change is to rename accounts that were outside
Beancount's accepted five root accounts, to move them into that
structure. This includes:
Accrued:*Payable: → Liabilities:Payable:*
Accrued:*Receivable: → Assets:Receivable:*
UneanedIncome:* → Liabilities:UnearnedIncome:*
Note the last change did inspire in a change to our validation rules. We no
longer require income-type on unearned income, because it's no longer
considered income at all. Once it's earned and converted to an Income
account, that has an income-type of course.
This did inspire another rename that was not required, but
provided more consistency with the other account names above:
Assets:Prepaid* → Assets:Prepaid:*
Where applicable, I have generally extended tests to make sure one of each
of the five account types is tested. (This mostly meant adding an Equity
account to the tests.) I also added tests for key parts of the hierarchy,
like Assets:Receivable and Liabilities:Payable, where applicable.
As part of this change, Account.is_real_asset() got renamed to
Account.is_cash_equivalent(), to better self-document its purpose.
Because these are the classes that get instantiated many times while
iterating transactions, the performance benefit of defining __slots__
is worth the development overhead.
The main motivation for this change is to make sure that higher-level
code deals with the fact that self.units.number can be None, and has
an easy way to do so.
I'm not sure all our code is *currently* doing the right thing for this
case, because I'm not sure it will ever actually come up. It's possible
that earlier Beancount plugins fill in decimal amounts for postings
that are originally loaded with self.units.number=None. I'll have to see
later whether this case comes up in reality, and then deal with it if so.
For now the safest strategy seems to be that most code should operate
when self.units.number is None.
These were tightened up by running mypy with --disallow-any-generics
and cleaning up reported errors.
There are still a couple of cases I don't know how to deal with
right now that prevent me from setting that option permanently:
* plugin/__init__.py sets up a dict where the key is a Directive
and its value is a set of hooks that correspond to that type
of directive. The relationship between key and value is not
expressable through the type system, and any other solution would
make the code way more involved and less dynamic.
* The type variable used in _GenericRange still isn't quite right,
in the sense that specifying a value for CT creates more errors
I don't know how to deal with. Protocols seem like the right
solution for that but they apparently didn't land for Python 3.7
that I can see.
Mostly this meant giving annotations to low-value functions like
the error classes and __init_subclass__, but it's worth it for
the future strictness+documentation value.
This just returns a constant for now, but we know it may need to be
configurable in the future. Other code can start using this now
to be configurable in the future.
I wasn't too worried about this earlier because the cache mainly stores
a bunch of numbers, but there's a little more than that: the generated
URLs also include original attachment filenames, which might be sensitive
(referencing people's names, bank names, etc.). Tighten security
accordingly.
This returns a cached version of the new rtutil.RT class,
so that a single process can easily grab references to one
instance with one cache of ticket/attachment data, etc.
For now, this is basically the Python version of
ledger-tag-convert.plx. It knows how to create RT web links from
ticket and attachment IDs. It confirms that those objects actually
exist too. It may grow to encompass other functionality in the
future.
This loads settings from the same environment variables and ~/.rtrc
file as the rt CLI.
Note that it does *not* support RTCONFIG and the config file
searching, because right now that seems like more work for more
trouble to me.
Our version of Posting is interface-compatible with Beancount's,
but makes stronger guarantees about the data types for our
higher-level code to rely on.
As I'm pushing to reimplement payment-report/income-report, I see query and
reporting tools will want easy access to these kinds of views, so here it
starts.
* Rename _typing to beancount_types to better reflect what it is.
* LessComparable isn't a Beancount type, so move that to
plugin.core with its dependent helper classes.
* Errors are a core Beancount concept, so move that module to the
top level and have it include appropriate type definitions.
I feel like posting hooks a case of premature optimization in early
development. This approach reduces the number of special cases in
the code and allows us to more strongly reason about hooks in the
type system.
This is the layer that keeps track of the different groups of hooks and
can filter them before runtime. The idea here is that you'll be able
to do things like skip hooks that require network access when you don't
have it, or skip CPU-intensive hooks when you don't need them, etc.
Python's own enum works fine for the simple values that expenseAllocation
uses, but it won't work as well for metadata like taxImplication where
many of the values aren't legal Python identifiers. Introduce our own
MetadataEnum class with the necessary functionality, and switch to that.
This is the simplest version of a common validation we're going to do:
make sure that a particular piece of metadata has one of a set of
values.
This checker needs some bounds checking but I wanted to err on the
side of committing this early because it introduces so much base
infrastructure.
