hledger-prices flags, commit policy..

[Simon Michael]

Cc'ing this from https://github.com/simonmichael/hledger/pull/562 to invite your comments:

@ony: Thanks for your comments. Re costs vs prices, believe me it's confusing for native english speakers too. :) The language is fuzzy. 

Here's my basic feedback. As a potential user, the flags are not clear. I can't tell from the names and docs what they do and when I would use them.

This makes me not entirely comfortable approving this as it is; I want to help raise the standard of all our docs and UX, especially in the main repo.

On the other hand this is described as an [experimental addon](http://hledger.org/manual.html#experimental-add-ons), and addons are supposed to be the place where development is unconstrained and I don't get in the way. Also I'm not using this myself yet, and you are. Maybe I should be sending pull requests to you instead. Do you have the necessary access to self-approve/merge it yourself ?

I'm cc'ing this to the list for wider comment, in case anybody has thoughts on this PR or on the larger topic of merging policy.

[Simon Michael]

On the PR I suggest replacing eg "--cost" with "--transaction-price", to be consistent with our main docs and terminology.

I use "transaction price" in the docs because "cost" is unidirectional and doesn't fit so well for a sale transaction. And, to communicate the relationship with, and difference from, "market prices". (One is a price set in a transaction, the other is set by an outside source, eg a stock market).

I must admit --transaction-price is a bit cumbersome. Is there another more familiar word that covers both buying and selling ?

I thought of "exchange"... in an alternate world we could have talked about "the price it was exchanged at", maybe used -X instead of -B. But that's not possible here because "-X/--exchange COMMODITY" has a well-established meaning in Ledger.

Should we re-embrace the "cost" and "value" terms, deprecating "transaction price" and explaining in docs that "cost" also covers selling ?

[Christian G. Warden]

I think ledger has the terminology right, --basis/--cost and --market.
These are common terms to talking about the price of something as of
the time of a transaction vs as of a later date.

> --
> You received this message because you are subscribed to the Google Groups "hledger" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to hledger+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Simon Michael]

I reviewed Ledger's price/value-related options to refresh my memory. This was a useful exercise. There are many, and some of them are different from what I thought. Here they are (some aren't relevant to this discussion, I've included them just for future reference):

Ledger 3.1.1 flags related to price/valuation

Non-line-wrapped version: https://gist.github.com/simonmichael/df414479a102a16d267689d5092c736f

COMMANDS

     balance

              --basis (-B)     Report in terms of cost basis, not amount or value.  This is the only form of report which is guaranteed to always balance to zero, when no report-query

                               is specified.  Only show totals for the top-most accounts.

     register

              --exchange commodity (-X)

                               Render all values in the given commodity, if a price conversion rate can be determined.  Rates are always displayed relative to the date of the posting

                               they are calculated for.  This means a register report is a historical value report.  For current values, it may be preferable to use the balance report.

              --gain (-G)      Show any gains (or losses) in commodity values over time.

              --historical (-H)

                               Value commodities at the time of their acquisition.

              --market (-V)    Show current market values for all amounts.  This is determined in a somewhat magical fashion.  It is probably more straightforward to use --exchange

                               option.

OPTIONS

     --basis (-B)

              Report the cost basis on all posting.  Alias for --cost

     --cost   Report the cost basis on all posting.  Alias for --basis.

     --exchange COMMODITY [, COMMODITY, ...] (-X)

              Display values in terms of the given COMMODITY.  The latest available price is used.

     --gain (-G)

              Report net gain or loss for commodities that have a price history.

     --lot-prices

              Report the price at which each commodity in a balance report was purchased.

     --lots   Report the date and price at which each commodity was purchased in a balance report.

     --lots-actual

              Preserve the uniqueness of commodities so they aren't merged during reporting without printing the lot annotations.

     --market (-V)

              Use the latest market value for all commodities.

     --price (-I)

              Use the price of the commodity purchase for performing calculations.

     --price-db FILE

     --price-exp STR (-Z)

              Set the expected freshness of price quotes, in INT minutes.  That is, if the last known quote for any commodity is older than this value, and if --download is being used,

              then the Internet will be consulted again for a newer price.  Otherwise, the old price is still considered to be fresh enough.  Alias for --leeway.

     --prices-format FMT

              Set the format for the prices report.

     --pricedb-format FMT

              Set the format expected for the historical price file.

     --revalued

              Report discrepancy in values for manual reports by inserting <Revalued> postings.  This is implied when using the --exchange (-X) or --market (-V) option.

     --revalued-only

              Show only <Revalued> postings.

     --revalued-total

              Display the sum of the revalued postings as the running total, which serves to show unrealized capital in a gain/losses report.

     --unrealized

              Show generated unrealized gain and loss accounts in the balance report.

     --unrealized-gains

              Allow the user to specify what account name should be used for unrealized gains.  Defaults to Equity:Unrealized Gains.  Often set in one's ~/.ledgerrc file to change the

              default.

     --unrealized-losses

              Allow the user to specify what account name should be used for unrealized losses.  Defaults to Equity:Unrealized Losses.  Often set in one's ~/.ledgerrc file to change the

              default.

And here are hledger's:

  -B --cost               convert amounts to their cost at transaction time

                          (using the transaction price, if any)

  -V --value              convert amounts to their market value on the report

                          end date (using the most recent applicable market

                          price, if any)

"basis" is the reason we have -B, but I find it vague - there are probably other kinds of basis ? (What are they ? I know about "cash basis" and "accrual basis".)

"market" - a market is one source of prices allowing valuation, but not the only one. 

"cost" is familiar and I think correctly suggests something fixed at transaction time. 

"value" is familiar and I think correctly suggests something that can change over time. (Ledger also supports --value, even though it doesn't appear in the --help output above).

So I am feeling good about "cost" and "value", and hledger's current flags, and using "cost" more in UI and docs. 

I see that we could increase compatibility for Ledger users by adding --basis and --market as aliases for --cost and --value, though maybe they should be somehow less visible to reduce confusion.

[Simon Michael]

I've pushed some changes for review https://github.com/simonmichael/hledger/pull/572 . See what you think. 

This is quite a design swamp!

[Simon Michael]

On Jun 10, 2017, at 4:16 PM, Simon Michael <si...@joyful.com> wrote:

I've pushed some changes for review https://github.com/simonmichael/hledger/pull/572 . See what you think. 

This is quite a design swamp!

Oops, that was meant for the "change how --cleared/--pending/--uncleared work ? #553" thread.

[Carel Fellinger]

On Sat, Jun 10, 2017 at 04:16:59PM -0700, Simon Michael wrote:

> I've pushed some changes for review
> https://github.com/simonmichael/hledger/pull/572>

> See what you think.

> This is quite a design swamp!

We dutch, whenever we see a swamp we just think a little drainage
that's all it takes. Clay ground is great for potatoes and such.

The same here, I think getting the nomenclature right is the first
step. You see, no matter how persuading I am, my daughters keep
tripping over the uncleared bit, and they aren't burdened by previous
experience and expectations from dodgy docs. They just keep including
pending. The term unmarked on the other hand works all the time.

Yesterday I tried to explain it again, using: cleared, clearing and
uncleared. Okee, yeah I remember..but then, the eldest complained, P
for uncleared? I really have to look that up in the help--she just
found "?" and got all the help she needed:)--screen al the time. So I
tried: cleared, pending clearance and uncleared. W'll see how that
sticks, and whether it helps keeping --uncleared clear.

So to me the first step is to check what the ledger people are upto in
their ledger 3 setup, the docs are contradicting whether --uncleared
includes or excludes pending. And then depending on that go for
unmarked or for pending clearance, whatever is closed to what they do.


The third step is what we are doing now, find a way to filter on those
states. We have two competing approaches so far:

1: Christian's toggle state (lower case) and toggle complement (uppercase)
this proved very easy to explain and use both from the commandline and
within hledger-ui
for: it gives direct access to all state combinations
use of lower/upper to get the state/comlement helps getting
the knowledge in the muscles (so very very easy in the ui)
against: it needs three extra commandline flags
to lower the mental burden those should start with --not-

2: Simon's combined states
not yet there from within hledger-ui, but easy to explain and use
from the commandline
for: mininal usage of commandline flags
against: complex in ui (still to be tested)

Helas, there emerged a shop stopper: the warping behaviour of such
toggling makes us loose focus. I'll start a separate thread on this,
as not to complicate the current discussion. Besides it will need
coding best kept in its own branch.

The point being, that toggling uncleared twice will put the cursor (focus)
at the very end of the list, toggling pending will end me up some where
half way. We seem to end at the last of the intermediate selection, and
I rather ended up where I started from. The way it is it's hard to keep
track of the very transaction you're trying to check/reconcile, unless
it happens to be the last of the list.

As a last note, it seems that not showing the state in the register
display is not that sorely missed, the toggling gives the info you
need at a finger tip. Or so it seems, but to really check we need
to get rid of this warping.

--
groetjes, carel

[Simon Michael]

On Jun 13, 2017, at 10:36 AM, Carel Fellinger <cf-20...@xs4all.nl> wrote:

On Sat, Jun 10, 2017 at 04:16:59PM -0700, Simon Michael wrote:

I've pushed some changes for review
https://github.com/simonmichael/hledger/pull/572>
See what you think.

This is quite a design swamp!

We dutch, whenever we see a swamp we just think a little drainage
that's all it takes.  Clay ground is great for potatoes and such.

Love it! 

The same here, I think getting the nomenclature right is the first
step.  You see, no matter how persuading I am, my daughters keep
tripping over the uncleared bit, and they aren't burdened by previous
experience and expectations from dodgy docs.  They just keep including
pending. The term unmarked on the other hand works all the time.

Yesterday I tried to explain it again, using: cleared, clearing and
uncleared.  Okee, yeah I remember..but then, the eldest complained, P
for uncleared?  I really have to look that up in the help--she just
found "?" and got all the help she needed:)--screen al the time.  So I
tried: cleared, pending clearance and uncleared.  W'll see how that
sticks, and whether it helps keeping --uncleared clear.

Thanks to you and your daughters - such user testing is really valuable. (And fun to read. The NL test department is setting a high standard!)

We may take a meandering path to the right solution, with experimentation and some wrong turns, but hopefully we'll get there in the end.

Re the above. I have never used pending, and until recently my world was simple: UNCLEARED/CLEARED. Would that work for you ? What do you use pending for, if you don't mind me asking possibly for a second time.. ?

So to me the first step is to check what the ledger people are upto in
their ledger 3 setup, the docs are contradicting whether --uncleared
includes or excludes pending.  And then depending on that go for
unmarked or for pending clearance, whatever is closed to what they do.

Ledger's --uncleared includes pending, while the docs use "uncleared" as one of the three states. That's how the swamp got started. As with hledger, the ui & docs show signs of bias towards the two-state uncleared/cleared model, as if pending was added later or is considered less important.

http://ledger-cli.org/3.0/doc/ledger3.html#Transactions-and-Comments cunningly doesn't name the unmarked state. But http://ledger-cli.org/3.0/doc/ledger3.html#Transaction-state says clearly:

A transaction can have a “state”: cleared, pending, or uncleared.

The third step is what we are doing now, find a way to filter on those
states.  We have two competing approaches so far:

 1:  Christian's toggle state (lower case) and toggle complement (uppercase)
     this proved very easy to explain and use both from the commandline and
     within hledger-ui
     for:  it gives direct access to all state combinations
           use of lower/upper to get the state/comlement helps getting
   the knowledge in the muscles (so very very easy in the ui)
     against:  it needs three extra commandline flags
               to lower the mental burden those should start with --not-

 2:  Simon's combined states
     not yet there from within hledger-ui, but easy to explain and use
     from the commandline
     for:  mininal usage of commandline flags
     against:  complex in ui (still to be tested)

Helas, there emerged a shop stopper: the warping behaviour of such
toggling makes us loose focus.  I'll start a separate thread on this,
as not to complicate the current discussion.  Besides it will need
coding best kept in its own branch.

You're right, I didn't work on this yet. It should be quite fixable though, we have some other actions that do a better job of preserving the selection/scroll position.

The point being, that toggling uncleared twice will put the cursor (focus)
at the very end of the list,  toggling pending will end me up some where
half way.  We seem to end at the last of the intermediate selection, and
I rather ended up where I started from.  The way it is it's hard to keep
track of the very transaction you're trying to check/reconcile, unless
it happens to be the last of the list.

As a last note, it seems that not showing the state in the register
display is not that sorely missed, the toggling gives the info you
need at a finger tip.  Or so it seems, but to really check we need
to get rid of this warping.

Good to know.

[Simon Michael]

We are working on this now in today's hledger dev stream.

https://hangouts.google.com/group/4IWx6VVkYpb61Bvm1

Latest thoughts: go further with Carel's "unmarked" suggestion. Replace "uncleared" with "unmarked" entirely in the UI. Probably don't even include --uncleared as a synonym, make a clean break. Mention it in docs though.

It would be most intuitive if all three states had a positive name (without "un").

Also, the current state names are a bit stressful. I was thinking, make them: "usual", "pleasant", "clear". The complementary flags are of course --unusual, --unpleasant, --unclear. Consistent, less stressful, more accurate ? No ?

Ok, trying unmarked/pending/cleared.

[Simon Michael]

This has landed in master. Also multiple status filters in hledger-ui. The last is fiddly as expected..

Please build and use master as much as possible, to see if we like these changes. The next major release is in about two weeks, IIRC.

Thanks!

[Carel Fellinger]

On Thu, Jun 15, 2017 at 07:21:31PM -0700, Simon Michael wrote:
> This has landed in master. Also multiple status filters in
> hledger-ui. The last is fiddly as expected..

> Please build and use master as much as possible, to see if we like
> these changes. The next major release is in about two weeks, IIRC.

Fantastic, works like a charm. And it struck me how simple the docs
on state, state flags and state toggles have become, whow. And to my
surprise the combine-state-toggling turned out to be the favourite here.

Reconciling/auditing is done a lot over here. Not only to check bank
statements, but also to check all advance payments we do for each
other are properly repaid and repaid just once. Sofar we used the
posting states to keep track in the Assets:Accounts Payable:RELATION
and Liablility:Accounts Receivable:RELATION. I'me hoping the checking
can be done visually from within hledger-ui instead of needing the
special program I made years ago geared to gnucash.


Today I lined up the girls for some user interface testing to compare
simon's implementation of combine-state-toggling:

U: toggle unmarked Initially show selected only,
P: toggle pending but toggle once one is selected.
C: toggle cleared Start again when all three are on.

and christofer's implementation of toggling-(complement-)states:

u: show unmarked only U: show all but unmarked
p: show pending only P: show all but pending
c: show cleared only C: show all but cleared

As you can see from the above, documenting the state/complement-state
thingy is straight forward and simple. To me it looked like a winner,
but much to my surprise explaining it to me wife and the youngest
daughter just got me blank faces staring at me: what the hell are you
doing. The eldest remarked: you're making us do some state calculus.

Apparently it's a programmers oddity to think of complementing states,
but the user is always right, especially when you're outnumbered :).
They unanimous and categorically rejected the complement state toggles
and choose for simon's implementation just on the bases of the explanation.

Okee, the user is not always right, they're sometimes deluded, yes.

So I did some more testing and forced the youngest to try it out,
expecting to change her mind as the complementing state is cleary
superiour. Heilas, I was the one to change mind:

when we need to reconcile bank statements, we need to be able
to look at the past to check there are no left over pending or
uncleared postings. And it turns out that's quite easy with
simon's implementation in the register screen:

select cleared, go down one transaction [needed due to warping]
add pending to the selection and drop cleared
check there no transactions above cursor

same with cleared/unmarked

trying this in christofer's implementation is a lot harder on the
brain, you really have to calculate the proper key to press,
whereas it's (visually) obvious in simon's implemantation.
Ofcourse it is learnable, but it's not nice from the start.


There are three minor points left: the warping, the lack of state info
and the docs on combining states in the ui as wel as in the cli.

1: Warping, to be delt with in a separate thread, makes you lose track
of the transaction group you're inspecting because you get warped
to the end of the selection list if the transaction under the
cursor is not in the new selection. To circumvent this odd
behaviour one has to pay attention to place the cursor on a
transaction that stays in the new selection before the toggling.
There is room for improvement there.

2: When a combined state view is presented it could/would be handy if
some 'electric' combined state was shown in the register screen.
It could well be this point becomes moot once the warping is fixed.

3: I'm not convinced the current wording for the combining nature of
states in the cli and the ui is readable enough for others. I've
tried a different wording for simon's implementation, to me it
seems more readable, probably because I like me own words :]

--
groetjes, carel

[Simon Michael]

Good to hear your thoughts!

I have used the new combinable toggles only briefly, and I must say I didn't like them. They forced me to think about the tool, rather than what I was trying to do.

I imagine Christian's scheme will too, though I have not tried it yet (I also don't like giving up six valuable keys. Might want u, p, c to set that mark on the selected txn or something).

I'll test it both ways some more. But I'm thinking one-filter-at-a-time is the better default behaviour.

Re 1, yes warping must be fixed asap.

I didn't understand 2.. ?

Other thoughts:

Sigh, should we relent on including a backwards-compatible --uncleared flag. I don't want to annoy folks by seeming gratuitously incompatible. Perhaps till next release, with a deprecation notice.

Maybe allow status:u, status:p, status:c as synonyms for status:, status:!, status:*. So you don't need to think about shell quoting rules (* requires escaping in fish shell, eg).

[Simon Michael]

hledger-ui in master now has this temporary flag, for easier testing:

     --status-toggles=N    choose how status toggles work:

                            1 UPC toggles X/all

                            2 UPC cycles X/not-X/all

                            3 UPC toggles each X

Not yet implemented, PR welcome:

                            4 upc sets X, UPC sets not-X

                            5 upc toggles X, UPC toggles not-X

[Simon Michael]

PS and 1 is the default. The help screen is also updated by this flag, but not any of the other docs.

[Simon Michael]

Also: I thought I understood the warping problem but I really don't - a clear reproducible example would be great, eg based on examples/status.journal (with more txns if needed).