After some efforts to make this current incarnation of the daemon
work for situations with excessively long output, I discovered two
problems.
First, bean-query seems to always use a pager in interactive mode,
and it seems impossible to dissuade bean-query from this behavior
without an upstream change to Beancount. As such, my first effort
was to force the pager to be `/usr/bin/cat`, which I thought would be
sufficient to solve the problem entirely.
After that change coupled with much debugging with IPC::Run, I found
that detecting the beancount> prompt using a regular expression of
/^\s*beancount\s*\>\s*/m, or similar efforts with simply
/\s*beancount\s*/ or other versions like /\s*beancount\s*\>\s*$/ms
/\s*beancount\s*\>\s*$/s, simply would not detect the end of the
string, so the process hung on this line in the main loop:
pump $runningBeanQuery until $rbcOut =~ /^\s*beancount\s*\>\s*/m;
Ironically, when there is *not* copious output, that line worked just
fine. It was baffling.
I switched to Expect, and found the exact same behavior with:
$runningBeanQuery->expect(undef, -re => '^\s*beancount\s*\>\s*')
with that line, it works fine in the StartRunningBeanQuery()
subroutine, but when used in the main loop, Expect never finds that
string. Again, I tried it with various regexes as specified above to
no avail.
*However*, merely changing the line to:
$runningBeanQuery->expect(undef, 'beancount>')
(i.e., not using the regex feature in the Expect library), fixed the
problem entirely.
My working theory is some complex interaction bug between the IO::Pty
module, regular expression matching, and output timing (since this
problem only shows up when there is copious output). I think in
particular the fact that bean-query creates a subprocess for the
`PAGER` command in the pty when the output is copious is somehow
related to the problem.
Regardless, this solution now works in all the cases where I'm using
the daemon, so I'm not investigating further.
Part of the SELECT statement that was entered gets repeated in the
output. It's some sort of timing issue with IPC::Run or the IO::Pty
because it's often not the entire input but only part of it.
It does however appear that there is usually a blank line right
before output, and in text mode, we can also search for the line of
-'s as a backup to assure we get the proper output and don't send the
garbled input back as the output across the named pipe.
This is a bit of a hacky solution and likely error-prone.
By using IPC::Run, the bean-query process need not be restarted every
time. This shows significant speed improvement.
Note that now, the process will not notice when the Beancount files
change underneath it, or when they query format changes. Additional
code will be needed to support this.
I think it was in error to go for a lock on the data and clear it —
even if it's data the client had set — when we timeout waiting for
the named pipe to appear. If the named pipe never appeared, then
something is likely really wrong with the server anyway, and as such,
we might as well expect the server to do the cleanup.
If the server never creates the named pipe, then this loop originally
hung forever. The count it now waits for is arbitrary, and probably
could be smaller, since the current implementation of the server
creates the pipe within 3-6 seconds under normal conditions.
Because the client defaulted to dying, this left the server in a
strange state as the query question wasn't cleared and the
“fake failure” named pipe was never opened and processed, since it
couldn't be.
The server did eventually recover when the client fully died;
however, the client has enough information to give up its query when
it knows failure has occurred, so that now happens.
The format option on bean-query command-line is now an option that
the client can pass along. The setting defaults to whatever
`bean-query` was going to default to.
On the a classic “concurrency is hard” front, I am slightly proud of
myself that I didn't hit this race condition but anticipated that it
might occur later. Giving entire control to fifoName to the server
makes more sense here, and the client should not submit its query any
time either fifoName or question is set.
As an extra safeguard, the client will die if it gets the lock and
the state for submitting a query isn't right.
Finally, added some fifoName removal debugging on the server.
This is a very basic daemon to run bean-query so that other Perl programs
can call it. The speed savings is not really there yet, as to get that, it
would need to leave bean-query running, perhaps timing out and reloading
files as needed. That's the long term plan.
Right now, all that this is useful for is to run another Perl process that
wants to submit and receive answers to bean-query.
