Future of bean-web

[Dominik Aumayr]

Hi, 

I would like to contribute to beancount by helping to rewriting bean-web, in order to

  - make the navigation between different views (balance sheet, income statement, etc.) and filters easier

  - make the UI (and UX) a bit prettier and make different features like filters (component, tag, year, ...) easily accessible via JavaScript (popovers, AJAX, etc.)

  - update the Python framework (to Flask maybe?) to make use of HTML templates (making changes easier)

  - and decouple bean-web from bean-report (no more HTML coming out of bean-report, moving all HTML templates to bean-web)

Is there a need for a contribution like this? Do you, Martin Blais, have any plans to change bean-web? And do you have opinions about the points mentioned above? 

I would really like to get involved with this, but need some hints as where to start and a discussion about where to take this contribution to bean-web/beancount. 

Best regards,

- Dominik

[Martin Blais]

On Mon, Nov 30, 2015 at 2:08 PM, Dominik Aumayr <dom...@aumayr.name> wrote:

Hi, 

I would like to contribute to beancount by helping to rewriting bean-web,

Sounds like a great idea to me! :-)

bean-web needs some love and attention. It's aging.

Or a new version.

 

in order to

  - make the navigation between different views (balance sheet, income statement, etc.) and filters easier

  - make the UI (and UX) a bit prettier and make different features like filters (component, tag, year, ...) easily accessible via JavaScript (popovers, AJAX, etc.)

  - update the Python framework (to Flask maybe?) to make use of HTML templates (making changes easier) 

  - and decouple bean-web from bean-report (no more HTML coming out of bean-report, moving all HTML templates to bean-web)

Here are some aging thoughts for bean-web:

- I've been meaning to remove the rendering of all the postings in journals; I think this detail is only needed on-demand, for specific transactions when debugging, akin to getting the "bean-doctor context" or "bean-doctor linked" invoked from Emacs/Vi. For a while I thought I could implement a client-side cursor that would allow one to "expand" a journal row to render its transaction, but that would still require rendering all of them in the HTML output, which is a bit overkill (it's slow for very long journals). Ideally, rendering a transaction's full detail should only be done on-demand. So maybe that's done in a request from a popup or via a link. Not sure how to do it, but the default journal shouldn't have to render all this detail.

- In a journal, it would be nice to be able to toggle the visibility of (a) balance assertion rows, and (b) everything else than transactions. I'm thinking of some hotkeys, perhaps with a help menu.

- In the balance renderings, it would be nice to have collapsible trees (I used to have them in a Beancount v1) that when you collapse, the total amount of the subaccounts appear. If you expand, they would disappear and the detail amounts on the subtree items replace them. I've been meaning to add that for a while, just no time for writing FE code.

- About the framework: I'm specifically using the most simple framework out there in order to minimize the number dependencies. Bottle is literally one file of Python, and it doesn't change very much, which I like. If you'd like to use something that introduces more external installs, it would probably be best to do the reimplementation in a separate package (which I encourage you to do, there's no problem with that). I'm really trying hard to limit the amount of stuff Beancount itself depends on, so very sensitive about adding any new dependencies, I really don't want to introduce breakage because of external packages. I don't even depend on Pandas in it, for that reason, though it might have been useful for it.

- An idea: One idea I've been having for a rewrite is to remove the hard-coded views and replace them instead with a FROM expression that would dynamically generate them. This way, all available reports any subset of transactions could be inspected easily. I imagined a textarea from the root page where the user could type an expression that would link to the view page (probably starting a balsheet). There would still be links to commonly used views (e.g. the last few years) but there would be no need to litter that root page with all the possibilities, e.g., for all tags. It's a bit of a mess right now.

- About decoupling bean-web from bean-report: This would be going in the reverse direction of where I've started to move this. I've been meaning for bean-web to reduce more and more to being just a thin shell around common reporting code that just happens to render those to HTML. The point of doing this is that the very same code used to produce CSV or text reports should be used to render HTML as well, so that those reports' data don't differ at all. Of course, you're always welcome to write another UI that doesn't use those reports if that turns out being a limitation (and I can imagine it might be for some things, and certainly makes creating new reports slower).

Is there a need for a contribution like this?

Absolutely. And thanks for asking!

While I have been shifting my own focus to using the shell more and more in order to support it better, I think it would be awesome to have a tool that provides new reports and visualizations (e.g., a treemap of assets, like this toy experiment I did but better: https://bitbucket.org/blais/beancount/src/9d56d5bfcf9310c3cd9a5fff0df4bf5a915dd911/experiments/treemap/bean-treemap?at=default&fileviewer=file-view-default). 

Tables of monthly aggregates seem to be a recurring topic.

Do you, Martin Blais, have any plans to change bean-web? And do you have opinions about the points mentioned above? 

(See my brain dump above.) 

I would really like to get involved with this, but need some hints as where to start and a discussion about where to take this contribution to bean-web/beancount.

I think the best way to proceed is to drive this from the perspective of what you would really want to see for yourself. What are you itching to have? I'd suggest maybe writing a very short doc on what you'd like to achieve just to organize your ideas, maybe share it with us, and then start to implement the features that aren't already available in bean-web and make that project public so we can kick the tires on it. I'd love to see how others would like to render their data.

If you'd like to make some incremental improvements to good old bean-web, some initial thoughts above (I'll admit it's not very ambitious), happy to discuss more. I'm enclined towards avoiding killing bean-web... I think a very simple UI like bean-web should always be provided along with Beancount and it doesn't have to be the best, it should just be good enough and always work; I see bean-web as a baseline, not so much as "the best we could do," but something stable and simple we can always rely on and that implements much of what we can do that can work across everyone's input files. That's an inherent limitation right there. IMO more elaborate presentations should probably be borne in a separate codebase, at least at first, until they mature (there's no disadvantage to that BTW, that shouldn't be a problem at all). That's probably a more exciting way to build something new. If you want to improve it in place, I'd recommend starting slow and small, and writing unit tests if it involves new Python code, trying to keep with its current style as much as possible.

I think in general the challenge is making presentations "generic" enough to work for many other users. I'm finding this in writing code for new Beancount features, for example: If I had to just write the scripts for myself as one-offs -- that's often how they start, BTW - it would take about 1/50th of the time it takes to bring them into Beancount and make sure all the corners are ironed and polished perfectly so that others can use them. In other words, I end up spending the bulk of my time coding specifically so that the tools are reusable and defined in the most general and simplest way (case in point: the recent price script I just release took forever to complete because I wasn't happy with it, it wasn't simple enough for others to use and depended on things that appeared only in my input file, I had to clean that up, and in the process I had to introduce some concepts, and ended up spending several weekends working on it... not all entirely fun; I ended up having to clean up how export code works to avoid crosstalk between the feature and so I had document those new semantics, I didn't really feel like doing that at all but it was the right thing to do in the end and the end result is much nicer and less confusing... I find joy in "finishing things" which is probably unwise in the bigger picture of my life, anyhow). And oftentimes, users will point out things that make me realize some things I hadn't thought about. It's really time-consuming. That's mainly why I'm trying to keep things really simple in the core, so that it's manageable, so that we end up with something that works with not too many pending bugs.

(Final note: For contributions to the Beancount code itself beyond two-line patches and bug fixes we'd have to sign a contributor agreement like most other OSS projects, e.g. something like this: https://github.com/fourier/ztree#copyright-issues or like http://clojure.org/contributing. In case you're not familiar with this process, this is pretty common, not something nefarious, for the integrity of copyright and ensuring the code remains squarely in GNU GPL domain regardless who committed to it.)

Best regards,

- Dominik

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/5b5e4e2d-a744-4696-9d0a-27497194cc26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dominik Aumayr]

Thanks Martin!

To quickly show where I want to take the web interface for beancount I created a quick-and-dirty web application:

https://github.com/aumayr/beancount-web
(can also be installed via "pip install git+ssh://g...@github.com/aumayr/beancount-web.git" for quick testing)

It is a separate package which uses an artificial API (api.py) to make calls into beancount and extract the data needed for the HTML templates. The API simply returns dicts with the data; this could be built into beancount itself: As a new format besides text, CSV and HTML bean-report could also return the data as JSON-dicts (this is what api.py tries to emulate). The web frontend, now somewhat detached from beancount itself, can now render the data as it wishes.

This decoupling also means that beancount itself can be as dependency-independent as you mentioned it should be, while the web frontend can be built using its own requirements (Flask, and a custom beancount-pygments-lexer, in the sample code above).

Some notable features:

- New template/styling with sidebar
- Filters in header (they do not really work that well at the moment)
- Syntax highlighting of the beancount-file (Source file)
- Charts to show some examples how they could be utilized (Balance Sheet, Income Statement, individual Accounts/Journals)

Some notable drawbacks at the moment:

- Absolutely no error handling
- No edge-cases tested nor handled
- Just tested with my own beancount-file (very simple and only half a year old) and with bean-example output
- No automated tests
- Slow for very large files (> 5 years in case of bean-example files)
- Filters (Tag, Year) not working properly (Tag better than Year), and Component-filter missing (Componets-links link to individual accounts)
- Source code highlighting works well enough for the moment, but the Lexer is far form complete (just a quick hack)

Can you test it in your browser? What do you think about it in terms of visual appeal, features and the decoupling via JSON/dicts that beancount.report emmits?


PS: I will collect your feature-ideas in a TODO-file in the beancount-web-repository so they do not get lost.

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhOWuUYpqjLmV%2BSM9jSp0vy8OP2mai4KYXDEo11%3DjB-_sA%40mail.gmail.com.

[redst...@gmail.com]

To quickly show where I want to take the web interface for beancount I created a quick-and-dirty web application:

https://github.com/aumayr/beancount-web

Looks like a good start!

I'm a beancount user, and ended up using beancount's web interface a lot more than I thought I would. So I'm excited to see someone updating the web interface (unfortunately, I'm not skilled with web related programming), which I think would be highly beneficial for the community of users.

One thing I noticed right off the bat: the graph on the top is pretty and useful, but for some accounts like the expense account, a plot of some kind of summation (monthly average, most likely) would be more useful than plotting the cumulative total.

I'd be happy to provide humble feedback for my use case at least, if that's helpful. I'm excited to see where this goes!

[Dominik Aumayr]

> Looks like a good start!

Thanks :-)

> One thing I noticed right off the bat: the graph on the top is pretty and useful, but for some accounts like the expense account, a plot of some kind of summation (monthly average, most likely) would be more useful than plotting the cumulative total.

I did put some more work into the charts (up to commit 7a5bbeb):

- New types of charts (Net worth, etc.)
- Multiple charts per page
- Charts with multiple operating_currencies
- Chart legends (toggle currency if there are multiple operating_currencies)

Can you give it a shot and report back which charts you find useful and which you would like to see? This is very flexible now and adding new ones takes no time.

> I'd be happy to provide humble feedback for my use case at least, if that's helpful. I'm excited to see where this goes!

And I am very grateful for your feedback - helps a lot! Everything is very fresh and flexible right now, so if you have ideas how we can improve this fresh start of beancount-web please feel free to share them :-)

> --
> You received this message because you are subscribed to the Google Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
> To post to this group, send email to bean...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/a16af970-6dad-4012-aca4-5580afe708de%40googlegroups.com.

[Dave Stephens]

This looks great.

A repeating theme here is having the ability to look at comparison data, ie: expenses by category by month or year. 

I added the ability to filter the output by month in this pull-request for bean-web.  Would be useful to have that feature in beancount-web.  

https://bitbucket.org/blais/beancount/pull-requests/2/enh-added-monthview-and-ability-to-review/diff

I also tried to make an output that showed 12 months of data on a single page, but didn't get very far with that - maybe something to add to beancount-web if you can think of a way to do it.

Thanks for the work on this!

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/BF372C66-45F1-4728-B7F4-5D29320C2491%40aumayr.name.

[Dominik Aumayr]

> I added the ability to filter the output by month in this pull-request for bean-web. Would be useful to have that feature in beancount-web.
> https://bitbucket.org/blais/beancount/pull-requests/2/enh-added-monthview-and-ability-to-review/diff

Haha I used this, manually patching my local beancount-installation with this diff, with bean-web for the last months :-D Thanks for that!

> A repeating theme here is having the ability to look at comparison data, ie: expenses by category by month or year.
>

> I also tried to make an output that showed 12 months of data on a single page, but didn't get very far with that - maybe something to add to beancount-web if you can think of a way to do it.

Can you elaborate on those two ideas a little bit? What exactly do you want to see?

There are the "Monthly totals"-graphs for each account and for the Income Statement (Income, Expenses). Do those make sense to you?

> Thanks for the work on this!

Happy to help!

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CALRNXuGr6JStbH1VOm%3Di-mPi6-sucxWxcScLG7uF3-fyBwRhsw%40mail.gmail.com.

[Dave Stephens]

Haha I used this, manually patching my local beancount-installation with this diff, with bean-web for the last months :-D Thanks for that!

Nice.  Glad someone uses this.

 

Can you elaborate on those two ideas a little bit? What exactly do you want to see?

Sure - basically I want to see what I spent my money on by month.  So the current income statement view, but with columns for each month (or year).  Does that make more sense?   

 

There are the "Monthly totals"-graphs for each account and for the Income Statement (Income, Expenses). Do those make sense to you?

Yes, these are great.  Would be good to be able to set the start date of the charts as an option.  My charts are starting at the date I have accounts opening in my beancount file, which is about 10 years before I have any data.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/0B8FC9E0-DDD0-418B-A50F-21FC4C76475C%40aumayr.name.

[Dominik Aumayr]

> Sure - basically I want to see what I spent my money on by month. So the current income statement view, but with columns for each month (or year). Does that make more sense?

I gave it a first try:

https://github.com/aumayr/beancount-web/commit/57fcfb040a4b8b429b3c49cd31d48b2570da38c2
(on the monthly-ie-totals branch, see https://github.com/aumayr/beancount-web/tree/monthly-ie-totals)

The new view is at http://localhost:5000/monthly_income_stmt/ . Is this "monthly Expenses overview table" what you meant? If yes I still have to clean up a bit before I merge into master, as this is a bit messy (code-wise) right now :-)

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CALRNXuGZ2yhjpeepq4gEoxs2kRGmzqLx7d95-6QVBzhiBFbnzg%40mail.gmail.com.

[Jacques Gagnon]

I like this view, a few thing to add IMHO:

• Sums displayed not only for the leaf of the table but also for all parent.
• Ability to expend minimize the tree.
• A sum of all account displayed at the end of each month columns.

In general I would personally like a page where we can customize that kind of monthly report:

• Select accounts to display (and at which level to group them by, between tree root and the leaf)
• Ability to generate graph

This is extreme case but I'll like to make some graph like this:

http://i.stack.imgur.com/3ZdAH.png

(clusters of stacked bar)

bars could be for example: expenses, liabilities, income and last one being the sum of all 3.

[redst...@gmail.com]

Looks nice! I hope the following comments are helpful:

Negative numbers:

-----------------

Most people are used to seeing positive numbers in Expense and Liability accounts, especially on graphs. Perhaps include an option to flip the negative numbers to positives?

Barcharts for drilled-down expense categories:

----------------------------------------------

It would be very helpful to compare any account in the hierarchy by month. Eg: Monthly expenditure in the Expense:Groceries account.

Treemaps:

---------

To generalize "drill-downs," a hierarchical chart ([1]) is among the most helpful for personal accounting, especially for Expense accounts. Some way to display this would be nice.

In summary, finding a way of generalizing all of this, so it is cohesive and customizable would be nice. I really liked Martin's idea of displaying custom queries on graphs. To expand:

- one can type a custom query into the webpage, and have it display results in both tabular and graph form

- the graph displayed would have some basic customizability (eg: radio buttons for bar/treemap/line, cumulative vs. per-posting/transaction, flip numbers, etc.)

- a config file holds a set of the user's most favorite set of queries and corresponding graphs, that are readily available any time without messing around with queries or configuring a graph

- an internal config file holds a set of sensible default graphs. they are like the user config file above, except, they cover the most common cases (eg: expense analysis, cashflow, net worth, etc.)

Thanks for working on this. This is probably what I most miss with beancount (or any other accounting software - beancount is actually the one furthest ahead), so I find it very valuable! I wish I could contribute directly, but currently don't have the bandwidth or expertise. I'll keep providing feedback if that's helpful.

[1] https://en.wikipedia.org/wiki/Treemapping

[redst...@gmail.com]

At the risk of inundating this thread with my own wishlist, let me write a few more ideas out. These are perhaps not related to what you're doing at the moment with graphs, but here they are anyway.

Data up-to-date indication:

---------------------------

When displaying the balance sheet, for Assets and Liabilities, I typically always compare my current balance to what the bank/credit-card/financial institution currently shows. Once they compare correctly, I put in a 'balance' directive for that account.

It would be immensely helpful if account names were optionally color coded in the balance sheet display to help with this process, like so:

Show account in:

- green:  if the latest posting is a balance check that passed (i.e., known-good)

- red:    if the latest posting is a balance check that failed (i.e., known-bad)

- yellow: if the latest posting is not a balance check (i.e., unknown)

Account subtotals:

------------------

Include sub-totals for nested accounts? Possibly in a different color, or different column, to set them apart? Or shown only when the parent is collapsed (as Martin might have mentioned somewhere)?

Collapsible parents:

--------------------

Ability to allow parents to be "collapsed." My hierarchy can be too deep and detailed, and some times, I want more of an overview of a category. It would be nice if this played well with graphs: collapsing would for instance, change the nature of a treemap to not show detailes of subaccounts of the collapsed account.

[Dominik Aumayr]

> • Sums displayed not only for the leaf of the table but also for all parent.

I did not yet figure out how to get these numbers, only the numbers for the leafs. Do you have any experience how to convince beancount to go give you all the numbers for an account tree?

> • Ability to expend minimize the tree.

I am on it. The client-side code (JavaScript, styling, ...) is not the issue here, as I already have these working. What is still missing are the sums for non-leafs, as mentioned above. Once I can get those from beancount I can finish implementing the tree.

> • A sum of all account displayed at the end of each month columns.

Done, available in HEAD (again, monthly-ie-totals branch).

> In general I would personally like a page where we can customize that kind of monthly report:
> • Select accounts to display (and at which level to group them by, between tree root and the leaf)
> • Ability to generate graph

This is what the "Component"-filter in the header (right) should become, once I know how to implement that properly. Instead of the current links to simple a Account-view this should become a selectable list/tree of all accounts.

> This is extreme case but I'll like to make some graph like this:
> http://i.stack.imgur.com/3ZdAH.png
> (clusters of stacked bar)
> bars could be for example: expenses, liabilities, income and last one being the sum of all 3.

Again, if I can convince beancount to give me these numbers (not only for leafs) then this should be a no-brainer to implement (eg. show a stacked bar chart for the visible/selected tree nodes).

If you have any idea how to get those numbers via the beancount functions, please let me know (preferably via a PR) :-)

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/e85071b8-b543-4aa7-a070-732323b21dfa%40googlegroups.com.

[Dominik Aumayr]

> Account subtotals:
> ------------------
> Include sub-totals for nested accounts? Possibly in a different color, or different column, to set them apart? Or shown only when the parent is collapsed (as Martin might have mentioned somewhere)?

I would love to have that as well, but as mentioned earlier I currently don't know how to get those numbers. The code I use only returns the numbers for leaf-nodes.

> Collapsible parents:
> --------------------
> Ability to allow parents to be "collapsed." My hierarchy can be too deep and detailed, and some times, I want more of an overview of a category. It would be nice if this played well with graphs: collapsing would for instance, change the nature of a treemap to not show detailes of subaccounts of the collapsed account.

I have the client-side code already, but again, I cannot convince beancount to give me the numbers for parent-accounts, only leafs.

> Data up-to-date indication:
> ---------------------------
> When displaying the balance sheet, for Assets and Liabilities, I typically always compare my current balance to what the bank/credit-card/financial institution currently shows. Once they compare correctly, I put in a 'balance' directive for that account.
>
> It would be immensely helpful if account names were optionally color coded in the balance sheet display to help with this process, like so:
>
> Show account in:
> - green: if the latest posting is a balance check that passed (i.e., known-good)
> - red: if the latest posting is a balance check that failed (i.e., known-bad)
> - yellow: if the latest posting is not a balance check (i.e., unknown)

Is this a best/common practice?

> Barcharts for drilled-down expense categories:
> ----------------------------------------------
> It would be very helpful to compare any account in the hierarchy by month. Eg: Monthly expenditure in the Expense:Groceries account.

This is already possible: Simply click on an account name to open the Account view. There should be a table chart with monthly balances. Is this what you are looking for? (Note: This does currently not work for some accounts, and I don't know why. But it should work for most.)

> Treemaps:
> ---------
> To generalize "drill-downs," a hierarchical chart ([1]) is among the most helpful for personal accounting, especially for Expense accounts. Some way to display this would be nice.

> [1] https://en.wikipedia.org/wiki/Treemapping

Excellent idea! I use treemaps to analyze my file system (http://www.derlien.com/), and it is very helpful! I will look for a JavaScript-library that does implement treemaps, and will implement it if I can find a good one.

> In summary, finding a way of generalizing all of this, so it is cohesive and customizable would be nice. I really liked Martin's idea of displaying custom queries on graphs. To expand:
> - one can type a custom query into the webpage, and have it display results in both tabular and graph form
> - the graph displayed would have some basic customizability (eg: radio buttons for bar/treemap/line, cumulative vs. per-posting/transaction, flip numbers, etc.)

This could become it's own view/subpage: A "Query"-view in the "Debug"-section in the sidebar. Will be put on my todo-list :-)

> --
> You received this message because you are subscribed to the Google Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
> To post to this group, send email to bean...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/5557fe4f-fb1f-4e12-b241-19d3206a67a6%40googlegroups.com.

[Jacques Gagnon]

I see 2 way of getting this data.

1. The sums are compute directly from beancount-web

2. Additional query are made to get the data and a union of all query data is displayed.

For the later on the bean-query API Martin recently added the root function which can be used to group data query at any level between the root and leafs. 

I wonder however what would be the time impact of multiple query on a large file vs just computing it from the base data.

Jacques Gagnon
GTalk/E-Mail: darth...@gmail.com
WLM (MSN): clou...@msn.com

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/FEC6AE41-B60E-46F2-8D9E-A6CA9CA82575%40aumayr.name.

[redst...@gmail.com]

>On Tuesday, December 8, 2015 at 2:11:49 AM UTC-8, Dominik Aumayr wrote:

>> Account subtotals: 

>> ------------------ 

>> Include sub-totals for nested accounts? Possibly in a different color, or different column, to set them apart? Or shown only when the parent is collapsed (as Martin might have mentioned somewhere)? 

>

>I would love to have that as well, but as mentioned earlier I currently don't know how to get those numbers. The code I use only returns the numbers for leaf-nodes. 

When I click on an account in bean-web, it shows all postings under that 

account, including all children, and gives a total for the 

parent+children. But I don't know how you can get this total in the backend.

Hmm, Martin might be the best person to ask.

>> Data up-to-date indication: 

>> --------------------------- 

>> When displaying the balance sheet, for Assets and Liabilities, I typically always compare my current balance to what the bank/credit-card/financial institution currently shows. Once they compare correctly, I put in a 'balance' directive for that account. 

>> 

>> It would be immensely helpful if account names were optionally color coded in the balance sheet display to help with this process, like so: 

>> 

>> Show account in: 

>> - green:  if the latest posting is a balance check that passed (i.e., known-good) 

>> - red:    if the latest posting is a balance check that failed (i.e., known-bad) 

>> - yellow: if the latest posting is not a balance check (i.e., unknown) 

>

>Is this a best/common practice? 

This is an idea that I've had for a while. I'm not aware of where this would

be a best/common practice apart from personal finance software that attempts

to sync with institutions. The closest I can think of are these: Gnucash has

the "reconcile" functionality and indication. Yodlee has an indicator showing

when the last refresh was. My idea came from the fact that this is the first

thing I do after importing, to figure out the "health" of my latest data

update. Currently, I'm forced to go through this process manually, and it's

exhausting.

>> Barcharts for drilled-down expense categories: 

>> ---------------------------------------------- 

>> It would be very helpful to compare any account in the hierarchy by month. Eg: Monthly expenditure in the Expense:Groceries account. 

>

>This is already possible: Simply click on an account name to open the Account view. There should be a table chart with monthly balances. Is this what you are looking for? (Note: This does currently not work for some accounts, and I don't know why. But it should work for most.) 

Hmm, I see the "Monthly totals" link, but when I click on it, I get a blank

graph. I updated from the repo a minute ago. Is this what I should be clicking

on?

[Dominik Aumayr]

> When I click on an account in bean-web, it shows all postings under that
> account, including all children, and gives a total for the
> parent+children. But I don't know how you can get this total in the backend.
>
> Hmm, Martin might be the best person to ask.

I figured it out on my own, altough the solution is not the finest (the numbers are correct, but the code is ugly). The api.py file is not meant to be a production-ready file anyway, just a quick way to get what a hypothetical beancount-JSON-api might look like. New code can be found in HEAD.

> >> Barcharts for drilled-down expense categories:
> >> ----------------------------------------------
> >> It would be very helpful to compare any account in the hierarchy by month. Eg: Monthly expenditure in the Expense:Groceries account.
> >
> >This is already possible: Simply click on an account name to open the Account view. There should be a table chart with monthly balances. Is this what you are looking for? (Note: This does currently not work for some accounts, and I don't know why. But it should work for most.)
>
> Hmm, I see the "Monthly totals" link, but when I click on it, I get a blank
> graph. I updated from the repo a minute ago. Is this what I should be clicking
> on?

The graph is broken for some Accounts, and I still have to figure out why.
But in master there is a new menu item which shows a big table with monthly aggregations of each account (full tree like Trial Balance). Does this work for you?

> >> Data up-to-date indication:
> >> ---------------------------
> >> When displaying the balance sheet, for Assets and Liabilities, I typically always compare my current balance to what the bank/credit-card/financial institution currently shows. Once they compare correctly, I put in a 'balance' directive for that account.
> >>
> >> It would be immensely helpful if account names were optionally color coded in the balance sheet display to help with this process, like so:
> >>
> >> Show account in:
> >> - green: if the latest posting is a balance check that passed (i.e., known-good)
> >> - red: if the latest posting is a balance check that failed (i.e., known-bad)
> >> - yellow: if the latest posting is not a balance check (i.e., unknown)
> >
> >Is this a best/common practice?
>
> This is an idea that I've had for a while. I'm not aware of where this would
> be a best/common practice apart from personal finance software that attempts
> to sync with institutions. The closest I can think of are these: Gnucash has
> the "reconcile" functionality and indication. Yodlee has an indicator showing
> when the last refresh was. My idea came from the fact that this is the first
> thing I do after importing, to figure out the "health" of my latest data
> update. Currently, I'm forced to go through this process manually, and it's
> exhausting.

This should be implemented in beancount first, as beancount-web tries to have no application logic whatsoever (except for api.py, which should eventually be implemented in beancount as well).

@Martin, what do you think about this feature?

> --
> You received this message because you are subscribed to the Google Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
> To post to this group, send email to bean...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/e9d0cfdc-1db9-45cc-8f74-b6046c185bca%40googlegroups.com.

[Dominik Aumayr]

>> • Sums displayed not only for the leaf of the table but also for all parent.
>
> I did not yet figure out how to get these numbers, only the numbers for the leafs. Do you have any experience how to convince beancount to go give you all the numbers for an account tree?
>
>> • Ability to expend minimize the tree.
>
> I am on it. The client-side code (JavaScript, styling, ...) is not the issue here, as I already have these working. What is still missing are the sums for non-leafs, as mentioned above. Once I can get those from beancount I can finish implementing the tree.

I figured out how to get those numbers, and just merged the "collapsable tree"-feature, as well as the "sums for all parents"-feature into master.

[redst...@gmail.com]

On Tuesday, December 8, 2015 at 12:44:40 PM UTC-8, Dominik Aumayr wrote:

>> • Sums displayed not only for the leaf of the table but also for all parent.
>
> I did not yet figure out how to get these numbers, only the numbers for the leafs. Do you have any experience how to convince beancount to go give you all the numbers for an account tree?
>
>> • Ability to expend minimize the tree.
>
> I am on it. The client-side code (JavaScript, styling, ...) is not the issue here, as I already have these working. What is still missing are the sums for non-leafs, as mentioned above. Once I can get those from beancount I can finish implementing the tree.

I figured out how to get those numbers, and just merged the "collapsable tree"-feature, as well as the "sums for all parents"-feature into master.

Great! I can't seem to see the commit though. Could you please point me to it?

[Jacques Gagnon]

Really nice!

The extend/minimize work for me in chrome but do not show up in Iceweasel 38.4.0 (Debian Firefox)

Also the color of the parent sum is really faint in Chrome (Linux). But ok in Iceweasel

This table got all the data I need, I'll try to make a chart like I wanted this weekend.

chartist-js look like a nice library to play with.

I see they support clustered or stacked bar chart, but not both at the same time.

But after looking at the lib source I think I could put something together.

[redst...@gmail.com]

Wow, I'm not at all seeing the collapsible parents or the parent sums. Tried on both Chrome and IE11 on Windows 7. I'm not even seeing the parent sum numbers in the html source. Any idea why? Is this dependent on a specific version of beancount? I'm using Martin's top of the tree.

Should I be able to see this on the main page as soon as I go to http://localhost:5000 ?

[Dominik Aumayr]

Current HEAD in master is 098316cef16ba6cc14a8a9f36d2cdda35b9c2cfa from about 7 hours ago (see https://github.com/aumayr/beancount-web/commits/master). Did you get the lastest source code?

If yes then you should see it in every table that has a tree of accounts in it, eg. Balance sheet, Income statement, Trial balance, Monthly Income statement, etc.

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/f9968b86-8255-4a2d-bd37-0470f644041a%40googlegroups.com.

[Dominik Aumayr]

> The extend/minimize work for me in chrome but do not show up in Iceweasel 38.4.0 (Debian Firefox)

I haven't tested it in Firefox yet (as I develop on a Mac using Chrome and Safari), but will do so in the future.

> Also the color of the parent sum is really faint in Chrome (Linux). But ok in Iceweasel

The colors are very light indeed, but if you hover your mouse over a table then they should get darker. I did this because otherwise the tables become very cluttered visually.

> This table got all the data I need, I'll try to make a chart like I wanted this weekend.
> chartist-js look like a nice library to play with.
>
> I see they support clustered or stacked bar chart, but not both at the same time.
> But after looking at the lib source I think I could put something together.

Nice! A few pointers:

- The generation of charts is somewhat generic, so first you provide a dictionary with the relevant data and options for the chart like so: https://github.com/aumayr/beancount-web/blob/master/beancount_web/templates/charts/_chart_monthly_net_worth.html#L3:L17

- Then you have to add a new 'stackedbar' option in charts.js, where you set up the chart itself, here: https://github.com/aumayr/beancount-web/blob/master/beancount_web/static/javascript/charts.js#L43:L56

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/6aad823a-af9d-45df-b4fb-ea3a1152aa71%40googlegroups.com.

[redst...@gmail.com]

Thanks, I see the sums now. My bad - I had multiple installs of beancount-web and it was using one I didn't expect.

Here is what I'm seeing:

http://i.imgur.com/6JDngTL.png?1

The parent and child sums show in the same font. Is this a problem at my end? Perhaps the parent could be bold? Maybe even in a separate "parent" column by itself?

Also, I'm not seeing what is being referred to as "extend/minimize". Should I be able to see it here?

Thank you for working on this!

[jere...@gmail.com]

This is very nice --- exactly what I was hoping someone would write!

You do warn that it is slow with large files, and currently it is unfortunately not very usable with my journal which totals to about 350KB (representing several years of personal transactions), as every report takes about 30 seconds to produce.

When I have some time I'll try to investigate how that might be improved, unless you or someone else gets to it first.

[redst...@gmail.com]

Hi Dominik,

I love the progress I'm seeing with this, and this is very useful!

Would you like feedback, bug reports etc. via github? It seems to me like github's infrastructure for these would make it easier, but if you can state your preference, I'll post accordingly.

[redst...@gmail.com]

It's working fine for me on IE11. The font/colors (black, gray, and the darker mouseover)  for the collapsible tree feature is excellent and intuitive!

[Dominik Aumayr]

> Would you like feedback, bug reports etc. via github? It seems to me like github's infrastructure for these would make it easier, but if you can state your preference, I'll post accordingly.

Thanks for giving it a try, and giving feedback!

We can kick off general discussions about features of beancount-web (and it's relation to beancount) here, and if you find a bug or have direct feedback about a component I would be happy if you directly open an issue on Github (https://github.com/aumayr/beancount-web/issues/new). Also, if a discussion about a specific feature results in a decision ("we should implement that", etc.) we can use Github issues as a TODO-list (tag "enhancement").

> --
> You received this message because you are subscribed to the Google Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
> To post to this group, send email to bean...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/5372491b-b78c-4484-82e3-462f907b98ca%40googlegroups.com.

[Dominik Aumayr]

>> Treemaps:
>> ---------
>> To generalize "drill-downs," a hierarchical chart ([1]) is among the most helpful for personal accounting, especially for Expense accounts. Some way to display this would be nice.
>> [1] https://en.wikipedia.org/wiki/Treemapping
>
> Excellent idea! I use treemaps to analyze my file system (http://www.derlien.com/), and it is very helpful! I will look for a JavaScript-library that does implement treemaps, and will implement it if I can find a good one.

Just merged the treemap-graphs into master, and am pretty happy with the treemaps!

One problem that I encountered: What about subaccounts that have a negative balance? Right now the treemap ignores negative balances (I invert the balances for Income- an Equity-subaccounts), but that is not a good solution…

[redst...@gmail.com]

On Wednesday, December 9, 2015 at 6:35:11 AM UTC-8, Dominik Aumayr wrote:

>> Treemaps:
>> ---------
>> To generalize "drill-downs," a hierarchical chart ([1]) is among the most helpful for personal accounting, especially for Expense accounts. Some way to display this would be nice.
>> [1] https://en.wikipedia.org/wiki/Treemapping
>
> Excellent idea! I use treemaps to analyze my file system (http://www.derlien.com/), and it is very helpful! I will look for a JavaScript-library that does implement treemaps, and will implement it if I can find a good one.

Just merged the treemap-graphs into master, and am pretty happy with the treemaps!

Yes, it looks great! I like the clean, short labels and mouseover.

I assume that you're thinking of putting something in like arbitrary start and end dates, or better yet, allowing for an arbitrary query? For eg, I typically want to see my expenses minus taxes.

Any idea how colors are being allocated?

 

One problem that I encountered: What about subaccounts that have a negative balance? Right now the treemap ignores negative balances (I invert the balances for Income- an Equity-subaccounts), but that is not a good solution…

Yes, this is one drawback of Treemaps. However, I think that for what it would mainly be used for in personal finance (getting a quick view of the big picture of each of Expense/Income/Asset/Liabilities), it is mostly safe to simply ignore them in the treemap, especially for small numbers.

However, one possible exception and an approach: show the magnitude of negative subaccounts, but with a special marker. Eg: show only negative accounts in red (or stripes or whatever). Allow for this to be an option, and by default, turn it off - i.e., ignore negative subaccounts like you're doing. The place where turning it on would help is to find problems. For instance, a significant negative number under an Expense or an Asset account (or a flipped Income or Liability account) might be an indication of a problem.

[jere...@gmail.com]

I determined the source of the slowness with large files, and fixed it. I'll send a pull request soon. Essentially it came down to the fact that the monthly reports were requiring a full pass through all entries for each month, leading to quadratic scaling. The problem is that clamp processes all entries, including ones before the start date, in order to compute starting balances, even though this information wasn't actually needed. For net worth calculation, the holdings were being recomputed fresh for each end date, rather than just adding to the holdings from the previous end date.

One thing that did become apparent, though, in the course of fixing these things is that it might not be ideal for beancount-web to live in a separate repository from beancount, because it tends to be useful to redactor the existing report and ops code in beancount when implementing new functionality in beancount-web, and this is very hard to manage with separate repositories.

[Dominik Aumayr]

> I assume that you're thinking of putting something in like arbitrary start and end dates, or better yet, allowing for an arbitrary query? For eg, I typically want to see my expenses minus taxes.

Yes, this could be a part of the custom-query-view. I have not started working on it though.

> Any idea how colors are being allocated?

The templates for the treemaps can specify a parameter which is the level of accounts it should change colors for, called chart.options.treemapColorLevel.
When the template, for example, sets chart.options.treemapColorLevel to 1, then the colors will change for every account on level 2 (0-indexed), eg. "Gifts" in "Expenses:Gifts:Christmas".

Code can be found here: https://github.com/aumayr/beancount-web/blob/8279cd03dc334dd11742a7c254c9ecb1497474b7/beancount_web/static/javascript/charts.js#L64:L68

> Yes, this is one drawback of Treemaps. However, I think that for what it would mainly be used for in personal finance (getting a quick view of the big picture of each of Expense/Income/Asset/Liabilities), it is mostly safe to simply ignore them in the treemap, especially for small numbers.
>
> However, one possible exception and an approach: show the magnitude of negative subaccounts, but with a special marker. Eg: show only negative accounts in red (or stripes or whatever). Allow for this to be an option, and by default, turn it off - i.e., ignore negative subaccounts like you're doing. The place where turning it on would help is to find problems. For instance, a significant negative number under an Expense or an Asset account (or a flipped Income or Liability account) might be an indication of a problem.

Good idea, caputred it in https://github.com/aumayr/beancount-web/issues/10

> --
> You received this message because you are subscribed to the Google Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
> To post to this group, send email to bean...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/49c56699-63d7-4952-a0c5-c6ad481883a3%40googlegroups.com.

[Dominik Aumayr]

> I determined the source of the slowness with large files, and fixed it. I'll send a pull request soon. Essentially it came down to the fact that the monthly reports were requiring a full pass through all entries for each month, leading to quadratic scaling. The problem is that clamp processes all entries, including ones before the start date, in order to compute starting balances, even though this information wasn't actually needed. For net worth calculation, the holdings were being recomputed fresh for each end date, rather than just adding to the holdings from the previous end date.

Perfekt, thank you for your help! As mentioned on Github I just pushed a refactoring of api.py, so merging could become a bit of a mess...

> One thing that did become apparent, though, in the course of fixing these things is that it might not be ideal for beancount-web to live in a separate repository from beancount, because it tends to be useful to redactor the existing report and ops code in beancount when implementing new functionality in beancount-web, and this is very hard to manage with separate repositories.

I think moving beancount-web into the beancount-repository is a bit early (as it changes a lot right now), and may not be in the immediate interest of Martin. (Maybe in the future when beancount-web is stable and mature).

But I also have that problem a lot, so something along those lines would be helpful for developting. Maybe put a clone of beancount into the beancount-web repository and then:

- merge changes in beancount into the copy of beancount in beancount-web (when Martin changes something) and
- create PRs to beancount off the copy of beancount in beancount-web

Could that be a practical solution? Do you have any other ides?

> Am 10.12.2015 um 09:30 schrieb jere...@gmail.com:
>
> I determined the source of the slowness with large files, and fixed it. I'll send a pull request soon. Essentially it came down to the fact that the monthly reports were requiring a full pass through all entries for each month, leading to quadratic scaling. The problem is that clamp processes all entries, including ones before the start date, in order to compute starting balances, even though this information wasn't actually needed. For net worth calculation, the holdings were being recomputed fresh for each end date, rather than just adding to the holdings from the previous end date.
>
> One thing that did become apparent, though, in the course of fixing these things is that it might not be ideal for beancount-web to live in a separate repository from beancount, because it tends to be useful to redactor the existing report and ops code in beancount when implementing new functionality in beancount-web, and this is very hard to manage with separate repositories.
>

> --
> You received this message because you are subscribed to the Google Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
> To post to this group, send email to bean...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/96e91929-3092-40db-9cdd-e1e607ca9d10%40googlegroups.com.

[Martin Blais]

Lots to add on this exciting thread, but swamped at work and cannot mix oss & work time, will catch up on the weekend with comments. But quickly, the main issue if dependency management. You're moving much faster right now in a separate repo and that's a great thing. You could create a branch in Beancount repo with the changes to reporting to support your app and integration into the mainline code can come later , that's one idea.
Have to run,

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/3E2D151F-3E04-46FF-A02D-6FFBCF581794%40aumayr.name.

[jere...@gmail.com]

Yes, having beancount-web developed in a separate repository from Martin's main beancount repository definitely makes sense. Whether to include the beancount code in that repository or to use a separate beancount branch associated with beancount-web is less clear. I guess separate repositories is simpler, even though it may mean that a lot of pull requests have to be in the form of two pull requests.

[Martin Blais]

You can also create custom reports in the separate repo, that's also not a problem.

I could integrate them later if they're generic enough and I can maintain them.

Or if you need something for a specific problem, you can ask. Reports are easy to build IMO.

On Thu, Dec 10, 2015 at 7:55 PM, <jere...@gmail.com> wrote:

Yes, having beancount-web developed in a separate repository from Martin's main beancount repository definitely makes sense.  Whether to include the beancount code in that repository or to use a separate beancount branch associated with beancount-web is less clear.  I guess separate repositories is simpler, even though it may mean that a lot of pull requests have to be in the form of two pull requests.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/49f9635e-166c-411f-98fb-247544913f29%40googlegroups.com.

[Jeremy Maitin-Shepard]

On Thursday, December 10, 2015 at 5:03:41 PM UTC-8, Martin Blais wrote:

You can also create custom reports in the separate repo, that's also not a problem.

I could integrate them later if they're generic enough and I can maintain them.

Or if you need something for a specific problem, you can ask. Reports are easy to build IMO.

The particular thing in question was factoring out the contents of the inner loop of beancount/ops/holdings.py:get_final_holdings into a separate function that constructs a Holding given a lot, a number, a price_map, a date, and an optional account name (set to None if unspecified).  For now I've just copied that logic into a function in (my branch of) beancount-web.

[Jeremy Maitin-Shepard]

On Thursday, December 10, 2015 at 12:30:59 AM UTC-8, Jeremy Maitin-Shepard wrote:

I determined the source of the slowness with large files, and fixed it.  I'll send a pull request soon.  Essentially it came down to the fact that the monthly reports were requiring a full pass through all entries for each month, leading to quadratic scaling.  The problem is that clamp processes all entries, including ones before the start date, in order to compute starting balances, even though this information wasn't actually needed.  For net worth calculation, the holdings were being recomputed fresh for each end date, rather than just adding to the holdings from the previous end date.

I submitted a pull request with these changes (rebased).  This fixes the speed issues with the monthly reports, at least for journals the size of mine / example.beancount.

The "General Journal" view still has a number of issues:
1. It is very slow if there are many entries.
2. The balance and changes columns don't really make sense when showing all entries.  They basically just end up reflecting currency conversions, and dont't provide any useful information.
3. When limited to a subtree, the change and balance columns are useful, but if there are many currencies involved, as in example.beancount, every balance entry contains the balance for every currency, even ones that did not change in the transaction/posting, which makes the page longer and doesn't seem useful.

It turned out that for example.beancount, over half of the time (13 out of 22 seconds) was spent just on calls to _inventory_to_json due to the large number of currencies involved.  I suggest it be changed instead to show only the balances for currencies involved in the transaction/posting (using a new version of _inventory_to_json).  This would probably eliminate almost all of the time due to _inventory_to_json.  The remaining 8 seconds was spent actually rendering the page; eliminating that might be harder/require some sort of lazy generation.

I also suggest that if the journal for all accounts is shown, then the balance and changes columns are eliminated.

[Dominik Aumayr]

> The "General Journal" view still has a number of issues:
> 1. It is very slow if there are many entries.
> 2. The balance and changes columns don't really make sense when showing all entries. They basically just end up reflecting currency conversions, and dont't provide any useful information.
> 3. When limited to a subtree, the change and balance columns are useful, but if there are many currencies involved, as in example.beancount, every balance entry contains the balance for every currency, even ones that did not change in the transaction/posting, which makes the page longer and doesn't seem useful.
>
> It turned out that for example.beancount, over half of the time (13 out of 22 seconds) was spent just on calls to _inventory_to_json due to the large number of currencies involved. I suggest it be changed instead to show only the balances for currencies involved in the transaction/posting (using a new version of _inventory_to_json). This would probably eliminate almost all of the time due to _inventory_to_json. The remaining 8 seconds was spent actually rendering the page; eliminating that might be harder/require some sort of lazy generation.
>
> I also suggest that if the journal for all accounts is shown, then the balance and changes columns are eliminated.

I tried to implement the suggestions here:

https://github.com/aumayr/beancount-web/commit/34dc11b12caee21437bd4a43fe57f0529a8cb010

- Hide balance and change in journal for all accounts
- Update _inventory_to_json to only show currencies that have changed

Do the code-changes look correct?

> The remaining 8 seconds was spent actually rendering the page; eliminating that might be harder/require some sort of lazy generation.

Any ideas how to implement that?

> --
> You received this message because you are subscribed to the Google Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
> To post to this group, send email to bean...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/ef492368-2458-44dd-b9fd-7888a081245c%40googlegroups.com.

[Jeremy Maitin-Shepard]

On Sat, Dec 12, 2015 at 12:36 AM, Jeremy Maitin-Shepard <jer...@jeremyms.com> wrote:

I submitted a pull request with these changes (rebased).  This fixes the speed issues with the monthly reports, at least for journals the size of mine / example.beancount.

I realized there is a licensing problem with my changes: the get_holding_from_position function is code that was factored out of beancount, which is licensed under GPLv2, while you have listed MIT as the license for beancount-web.

I think having a different license than beancount will be problematic, since I imagine code movement between the two projects may be common.  The MIT license is GPLv2 compatible, but GPLv2 is more restrictive.  That basically means that by including that function the whole of beancount-web cannot be considered MIT licensed anymore.

[Jeremy Maitin-Shepard]

On Sat, Dec 12, 2015 at 11:04 AM, Dominik Aumayr <dom...@aumayr.name> wrote:

I tried to implement the suggestions here:

https://github.com/aumayr/beancount-web/commit/34dc11b12caee21437bd4a43fe57f0529a8cb010

- Hide balance and change in journal for all accounts
- Update _inventory_to_json to only show currencies that have changed

Do the code-changes look correct?

Thanks.  In terms of behavior, the only thing that might be worth changing is to include the balance for all currencies in changes, even if the balance is 0, since it is probably good to make that explicit.

I realized that because inventory is just a list, rather than a dict keyed by currency, restricting to only some of the currencies doesn't really make it faster.  To fix that we'd have to make a variant of iterate_with_balance, that would just give the aggregate balance and changes per currency (using a dict) rather than individual lots.

 
 

> The remaining 8 seconds was spent actually rendering the page; eliminating that might be harder/require some sort of lazy generation.

Any ideas how to implement that?

Unfortunately I don't have much experience with web programming, so it is hard for me to know what would be fast, and I also don't know what the options and tradeoffs are as far as lazy-loading a scrollable list.  If the slowness is due to python generating the HTML, maybe there is a way to optimize the template code.  We could also just send the json to the client and generate the html using javascript.

[Dominik Aumayr]

> Thanks. In terms of behavior, the only thing that might be worth changing is to include the balance for all currencies in changes, even if the balance is 0, since it is probably good to make that explicit.
>
> I realized that because inventory is just a list, rather than a dict keyed by currency, restricting to only some of the currencies doesn't really make it faster. To fix that we'd have to make a variant of iterate_with_balance, that would just give the aggregate balance and changes per currency (using a dict) rather than individual lots.

OK. Are you planning on implementing the new variant of iterate_with_balance?

> Unfortunately I don't have much experience with web programming, so it is hard for me to know what would be fast, and I also don't know what the options and tradeoffs are as far as lazy-loading a scrollable list. If the slowness is due to python generating the HTML, maybe there is a way to optimize the template code. We could also just send the json to the client and generate the html using javascript.

As I have more experience with web programming than with working with beancount I will focus on that aspect of the performace-issues ;-)
The server-side rendering of the journal looks like the biggest slowdown, so I will try to generate the HTML on the client with JavaScript.

> --
> You received this message because you are subscribed to the Google Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
> To post to this group, send email to bean...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAKJfoCEh4de6LxMD_L0XVJPi6cygaFtAj_Kw_ZgkuLY_PvtsPA%40mail.gmail.com.

[Dominik Aumayr]

> I realized there is a licensing problem with my changes: the get_holding_from_position function is code that was factored out of beancount, which is licensed under GPLv2, while you have listed MIT as the license for beancount-web.

To be honest I did not really give too much thought about the licensing of beancount-web when I started it a few weeks ago. I will look into what would be best for the project.

> I think having a different license than beancount will be problematic, since I imagine code movement between the two projects may be common. The MIT license is GPLv2 compatible, but GPLv2 is more restrictive. That basically means that by including that function the whole of beancount-web cannot be considered MIT licensed anymore.

Generally speaking this true right now, of course, as we implement api.py to give the results needed for beancount-web. But the goal is to move all the api.py code into custom reports in beancount itself (once they are stable), so there should be no big "code-overlap" between the two projects.

> --
> You received this message because you are subscribed to the Google Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
> To post to this group, send email to bean...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAKJfoCHaPi3PH7dOjUUQVXns_mDSbe%2BRQkA3Yc%2BM33VGcHxrXQ%40mail.gmail.com.

[Dominik Aumayr]

> As I have more experience with web programming than with working with beancount I will focus on that aspect of the performace-issues ;-)
> The server-side rendering of the journal looks like the biggest slowdown, so I will try to generate the HTML on the client with JavaScript.

I commited many changes to master today, including the client-side rendering of Journal entries. How do these changes affect the performace for you?

[Jacques Gagnon]

I've got a bean-web  instance with a bean-example file running on pythonanywhere.

Balance-sheet page was taking 20sec to load previously, now it's only take 2sec.

So good improvement.

btw its at darthcloud.pythonanywhere.com if someone curious to see the new bean-web without installing.

[redst...@gmail.com]

On Sunday, December 13, 2015 at 1:24:08 PM UTC-8, Dominik Aumayr wrote:

> As I have more experience with web programming than with working with beancount I will focus on that aspect of the performace-issues ;-) 
> The server-side rendering of the journal looks like the biggest slowdown, so I will try to generate the HTML on the client with JavaScript. 

I commited many changes to master today, including the client-side rendering of Journal entries. How do these changes affect the performace for you? 

Getting to be quite usable on data of 10+ years! My journal was too big to get to even work with beancount-web previously, but now, it works! It still crashes when I click on Expenses (says "Internal Server Error The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.", but seems to work okay otherwise.

Is it already possible to see accounts by year, or is this a feature to be done?

[Dominik Aumayr]

> Getting to be quite usable on data of 10+ years! My journal was too big to get to even work with beancount-web previously, but now, it works! It still crashes when I click on Expenses (says "Internal Server Error The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.", but seems to work okay otherwise.

Can you start `beancount-web` with `--debug` enabled and post the stacktrace to a new issue on Github? Then we can resolve the issue together.

> Is it already possible to see accounts by year, or is this a feature to be done?

That's next on the list ;-)

> --
> You received this message because you are subscribed to the Google Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
> To post to this group, send email to bean...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/111e0934-4f5f-4013-b531-5c60b4519954%40googlegroups.com.

[Simon Michael]

On 12/13/15 1:41 PM, Jacques Gagnon wrote:
> btw its at darthcloud.pythonanywhere.com if someone curious to see the new
> bean-web without installing.

That's me, thanks for doing that..

Looking excellent! Great work bean-web devs!

[Martin Blais]

Hi Dominik,

I tried your web front-end this weekend and I have to compliment you on it, it's really beautiful and it works well.

It's also styled much nicer than bean-web and I love how you've implemented some of the new behavior, e.g. how the aggregate amounts dim depending on tree collapse state, it's perfect. This is a really awesome UI and if you have the stamina to work through all the kinks that surely will come up over time with an increased user base I'm pretty sure it stand a good chance of becoming the de-facto front-end for Beancount users.  Congratulations on building something so nice! :-)

I love your treemap view, I had experimented a little bit with something like that in the past here:

https://bitbucket.org/blais/beancount/src/c5252218c11b9f96c57e1423fb3f38314916149b/experiments/treemap/bean-treemap?at=default&fileviewer=file-view-default

Your treemap is much better integrated, in any case.

Your commodities view is awesome.

Some notes to consider:

- Given how "pro" this looks, you may consider adding a global option to render the numbers in their absolute value, that is, negate the income, equity and liabilities numbers everywhere. I've been considering adding this to Beancount myself (both for input syntax and output reports actually), wondering if I could find a way to do that in a way you could reuse. This would make the presention easier to follow for "real" accountants by removing the surprising signs.

- I've been meaning to remove the rendering of postings in journals, to make them collapsible (and collapsed by default) like you did, but to only fetch the detail from the server when expanding a particular transaction or hovering over a particular link. This would have two advantages: (1) the journals would render much faster, by avoiding rendering all of these postings details by default, which is slow, and which is really an overkill amount of detail for the most part, and (2) the journals would look smaller on-screen and be much easier to read.

- It would be useful to render the "context" for the top four menus, e.g. if one restricts by time, a tag, or a component. Currently it only appears in the URL.

Lots more to say, will add more later and try to reply to the threads below.

On Fri, Dec 4, 2015 at 10:52 AM, Dominik Aumayr <dom...@aumayr.name> wrote:

Thanks Martin!

To quickly show where I want to take the web interface for beancount I created a quick-and-dirty web application:

https://github.com/aumayr/beancount-web
(can also be installed via "pip install git+ssh://g...@github.com/aumayr/beancount-web.git" for quick testing)

It is a separate package which uses an artificial API (api.py) to make calls into beancount and extract the data needed for the HTML templates. The API simply returns dicts with the data; this could be built into beancount itself: As a new format besides text, CSV and HTML bean-report could also return the data as JSON-dicts (this is what api.py tries to emulate). The web frontend, now somewhat detached from beancount itself, can now render the data as it wishes.

This decoupling also means that beancount itself can be as dependency-independent as you mentioned it should be, while the web frontend can be built using its own requirements (Flask, and a custom beancount-pygments-lexer, in the sample code above).

Some notable features:

- New template/styling with sidebar
- Filters in header (they do not really work that well at the moment)
- Syntax highlighting of the beancount-file (Source file)
- Charts to show some examples how they could be utilized (Balance Sheet, Income Statement, individual Accounts/Journals)

Some notable drawbacks at the moment:

- Absolutely no error handling
- No edge-cases tested nor handled
- Just tested with my own beancount-file (very simple and only half a year old) and with bean-example output
- No automated tests
- Slow for very large files (> 5 years in case of bean-example files)
- Filters (Tag, Year) not working properly (Tag better than Year), and Component-filter missing (Componets-links link to individual accounts)
- Source code highlighting works well enough for the moment, but the Lexer is far form complete (just a quick hack)

Can you test it in your browser? What do you think about it in terms of visual appeal, features and the decoupling via JSON/dicts that beancount.report emmits?


PS: I will collect your feature-ideas in a TODO-file in the beancount-web-repository so they do not get lost.

> Am 01.12.2015 um 06:54 schrieb Martin Blais <bl...@furius.ca>:
>
> On Mon, Nov 30, 2015 at 2:08 PM, Dominik Aumayr <dom...@aumayr.name> wrote:
> Hi,
>
> I would like to contribute to beancount by helping to rewriting bean-web,
>
> Sounds like a great idea to me! :-)
> bean-web needs some love and attention. It's aging.
> Or a new version.
>
>
> in order to
>
>   - make the navigation between different views (balance sheet, income statement, etc.) and filters easier
>   - make the UI (and UX) a bit prettier and make different features like filters (component, tag, year, ...) easily accessible via JavaScript (popovers, AJAX, etc.)
>   - update the Python framework (to Flask maybe?) to make use of HTML templates (making changes easier)
>   - and decouple bean-web from bean-report (no more HTML coming out of bean-report, moving all HTML templates to bean-web)
>
> Here are some aging thoughts for bean-web:
>
> - I've been meaning to remove the rendering of all the postings in journals; I think this detail is only needed on-demand, for specific transactions when debugging, akin to getting the "bean-doctor context" or "bean-doctor linked" invoked from Emacs/Vi. For a while I thought I could implement a client-side cursor that would allow one to "expand" a journal row to render its transaction, but that would still require rendering all of them in the HTML output, which is a bit overkill (it's slow for very long journals). Ideally, rendering a transaction's full detail should only be done on-demand. So maybe that's done in a request from a popup or via a link. Not sure how to do it, but the default journal shouldn't have to render all this detail.
>
> - In a journal, it would be nice to be able to toggle the visibility of (a) balance assertion rows, and (b) everything else than transactions. I'm thinking of some hotkeys, perhaps with a help menu.
>
> - In the balance renderings, it would be nice to have collapsible trees (I used to have them in a Beancount v1) that when you collapse, the total amount of the subaccounts appear. If you expand, they would disappear and the detail amounts on the subtree items replace them. I've been meaning to add that for a while, just no time for writing FE code.
>
> - About the framework: I'm specifically using the most simple framework out there in order to minimize the number dependencies. Bottle is literally one file of Python, and it doesn't change very much, which I like. If you'd like to use something that introduces more external installs, it would probably be best to do the reimplementation in a separate package (which I encourage you to do, there's no problem with that). I'm really trying hard to limit the amount of stuff Beancount itself depends on, so very sensitive about adding any new dependencies, I really don't want to introduce breakage because of external packages. I don't even depend on Pandas in it, for that reason, though it might have been useful for it.
>
> - An idea: One idea I've been having for a rewrite is to remove the hard-coded views and replace them instead with a FROM expression that would dynamically generate them. This way, all available reports any subset of transactions could be inspected easily. I imagined a textarea from the root page where the user could type an expression that would link to the view page (probably starting a balsheet). There would still be links to commonly used views (e.g. the last few years) but there would be no need to litter that root page with all the possibilities, e.g., for all tags. It's a bit of a mess right now.
>
> - About decoupling bean-web from bean-report: This would be going in the reverse direction of where I've started to move this. I've been meaning for bean-web to reduce more and more to being just a thin shell around common reporting code that just happens to render those to HTML. The point of doing this is that the very same code used to produce CSV or text reports should be used to render HTML as well, so that those reports' data don't differ at all. Of course, you're always welcome to write another UI that doesn't use those reports if that turns out being a limitation (and I can imagine it might be for some things, and certainly makes creating new reports slower).
>
>
> Is there a need for a contribution like this?
>
> Absolutely. And thanks for asking!
> While I have been shifting my own focus to using the shell more and more in order to support it better, I think it would be awesome to have a tool that provides new reports and visualizations (e.g., a treemap of assets, like this toy experiment I did but better: https://bitbucket.org/blais/beancount/src/9d56d5bfcf9310c3cd9a5fff0df4bf5a915dd911/experiments/treemap/bean-treemap?at=default&fileviewer=file-view-default).
>
> Tables of monthly aggregates seem to be a recurring topic.
>
>
> Do you, Martin Blais, have any plans to change bean-web? And do you have opinions about the points mentioned above?
>
> (See my brain dump above.)
>
>
> I would really like to get involved with this, but need some hints as where to start and a discussion about where to take this contribution to bean-web/beancount.
>
> I think the best way to proceed is to drive this from the perspective of what you would really want to see for yourself. What are you itching to have? I'd suggest maybe writing a very short doc on what you'd like to achieve just to organize your ideas, maybe share it with us, and then start to implement the features that aren't already available in bean-web and make that project public so we can kick the tires on it. I'd love to see how others would like to render their data.
>
> If you'd like to make some incremental improvements to good old bean-web, some initial thoughts above (I'll admit it's not very ambitious), happy to discuss more. I'm enclined towards avoiding killing bean-web... I think a very simple UI like bean-web should always be provided along with Beancount and it doesn't have to be the best, it should just be good enough and always work; I see bean-web as a baseline, not so much as "the best we could do," but something stable and simple we can always rely on and that implements much of what we can do that can work across everyone's input files. That's an inherent limitation right there. IMO more elaborate presentations should probably be borne in a separate codebase, at least at first, until they mature (there's no disadvantage to that BTW, that shouldn't be a problem at all). That's probably a more exciting way to build something new. If you want to improve it in place, I'd recommend starting slow and small, and writing unit tests if it involves new Python code, trying to keep with its current style as much as possible.
>
> I think in general the challenge is making presentations "generic" enough to work for many other users. I'm finding this in writing code for new Beancount features, for example: If I had to just write the scripts for myself as one-offs -- that's often how they start, BTW - it would take about 1/50th of the time it takes to bring them into Beancount and make sure all the corners are ironed and polished perfectly so that others can use them. In other words, I end up spending the bulk of my time coding specifically so that the tools are reusable and defined in the most general and simplest way (case in point: the recent price script I just release took forever to complete because I wasn't happy with it, it wasn't simple enough for others to use and depended on things that appeared only in my input file, I had to clean that up, and in the process I had to introduce some concepts, and ended up spending several weekends working on it... not all entirely fun; I ended up having to clean up how export code works to avoid crosstalk between the feature and so I had document those new semantics, I didn't really feel like doing that at all but it was the right thing to do in the end and the end result is much nicer and less confusing... I find joy in "finishing things" which is probably unwise in the bigger picture of my life, anyhow). And oftentimes, users will point out things that make me realize some things I hadn't thought about. It's really time-consuming. That's mainly why I'm trying to keep things really simple in the core, so that it's manageable, so that we end up with something that works with not too many pending bugs.
>
> (Final note: For contributions to the Beancount code itself beyond two-line patches and bug fixes we'd have to sign a contributor agreement like most other OSS projects, e.g. something like this: https://github.com/fourier/ztree#copyright-issues or like http://clojure.org/contributing. In case you're not familiar with this process, this is pretty common, not something nefarious, for the integrity of copyright and ensuring the code remains squarely in GNU GPL domain regardless who committed to it.)
>
>
>
> Best regards,
> - Dominik

>
> --
> You received this message because you are subscribed to the Google Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
> To post to this group, send email to bean...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/5b5e4e2d-a744-4696-9d0a-27497194cc26%40googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
> To post to this group, send email to bean...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhOWuUYpqjLmV%2BSM9jSp0vy8OP2mai4KYXDEo11%3DjB-_sA%40mail.gmail.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/FCEA5F2C-E5AC-41FF-8EE4-B38FECEA4697%40aumayr.name.

[Martin Blais]

On Mon, Dec 7, 2015 at 4:15 PM, <redst...@gmail.com> wrote:

At the risk of inundating this thread with my own wishlist, let me write a few more ideas out. These are perhaps not related to what you're doing at the moment with graphs, but here they are anyway.

Data up-to-date indication:

---------------------------

When displaying the balance sheet, for Assets and Liabilities, I typically always compare my current balance to what the bank/credit-card/financial institution currently shows. Once they compare correctly, I put in a 'balance' directive for that account.

It would be immensely helpful if account names were optionally color coded in the balance sheet display to help with this process, like so:

Show account in:

- green:  if the latest posting is a balance check that passed (i.e., known-good)

- red:    if the latest posting is a balance check that failed (i.e., known-bad)

- yellow: if the latest posting is not a balance check (i.e., unknown)

That's a cool idea, and I would add maybe a "gray" if the account hasn't been updated in a while (as compared to the last available date in the file).

[Martin Blais]

On Tue, Dec 8, 2015 at 3:38 PM, Dominik Aumayr <dom...@aumayr.name> wrote:

> When I click on an account in bean-web, it shows all postings under that
> account, including all children, and gives a total for the
> parent+children. But I don't know how you can get this total in the backend.
>
> Hmm, Martin might be the best person to ask.

I figured it out on my own, altough the solution is not the finest (the numbers are correct, but the code is ugly). The api.py file is not meant to be a production-ready file anyway, just a quick way to get what a hypothetical beancount-JSON-api might look like. New code can be found in HEAD.

Just did a uber quick search in your code and I'm assuming you're referring to this bit:

https://github.com/aumayr/beancount-web/blob/master/beancount_web/api.py#L330

There's already support to do that type of thing, there's an object called "Inventory" which is used to do just that:

https://bitbucket.org/blais/beancount/src/c5252218c11b9f96c57e1423fb3f38314916149b/src/python/beancount/core/inventory.py?at=default&fileviewer=file-view-default

There's also a utility function you can use to compute balances by account:

https://bitbucket.org/blais/beancount/src/c5252218c11b9f96c57e1423fb3f38314916149b/src/python/beancount/ops/summarize.py?at=default&fileviewer=file-view-default#summarize.py-574

You can adapt this to summing just the postings you need. 

Only at the very end, on the resulting Inventory instance should you extract the units or cost.

The design doc contains a discussion of all those core objects, which should help you map out how to manipulate Beancount's most common objects:

https://docs.google.com/document/d/1N7HDXuNWgLG2PqFS4Kkgv5LzAAtU6c97UVNT7tdTIjA/edit#heading=h.ug7qnawtw5ou

> >> Barcharts for drilled-down expense categories:
> >> ----------------------------------------------
> >> It would be very helpful to compare any account in the hierarchy by month. Eg: Monthly expenditure in the Expense:Groceries account.
> >
> >This is already possible: Simply click on an account name to open the Account view. There should be a table chart with monthly balances. Is this what you are looking for? (Note: This does currently not work for some accounts, and I don't know why. But it should work for most.)
>

> Hmm, I see the "Monthly totals" link, but when I click on it, I get a blank
> graph. I updated from the repo a minute ago. Is this what I should be clicking
> on?

The graph is broken for some Accounts, and I still have to figure out why.
But in master there is a new menu item which shows a big table with monthly aggregations of each account (full tree like Trial Balance). Does this work for you?

> >> Data up-to-date indication:
> >> ---------------------------
> >> When displaying the balance sheet, for Assets and Liabilities, I typically always compare my current balance to what the bank/credit-card/financial institution currently shows. Once they compare correctly, I put in a 'balance' directive for that account.
> >>
> >> It would be immensely helpful if account names were optionally color coded in the balance sheet display to help with this process, like so:
> >>
> >> Show account in:
> >> - green:  if the latest posting is a balance check that passed (i.e., known-good)
> >> - red:    if the latest posting is a balance check that failed (i.e., known-bad)
> >> - yellow: if the latest posting is not a balance check (i.e., unknown)
> >

> >Is this a best/common practice?
>

> This is an idea that I've had for a while. I'm not aware of where this would
> be a best/common practice apart from personal finance software that attempts
> to sync with institutions. The closest I can think of are these: Gnucash has
> the "reconcile" functionality and indication. Yodlee has an indicator showing
> when the last refresh was. My idea came from the fact that this is the first
> thing I do after importing, to figure out the "health" of my latest data
> update. Currently, I'm forced to go through this process manually, and it's
> exhausting.

This should be implemented in beancount first, as beancount-web tries to have no application logic whatsoever (except for api.py, which should eventually be implemented in beancount as well).

@Martin, what do you think about this feature?

I think it's a great idea, and the date checking is something you can certainly do on your side for now, or in api.py.

[Martin Blais]

On Wed, Dec 9, 2015 at 7:28 AM, Dominik Aumayr <dom...@aumayr.name> wrote:

> Would you like feedback, bug reports etc. via github? It seems to me like github's infrastructure for these would make it easier, but if you can state your preference, I'll post accordingly.

Thanks for giving it a try, and giving feedback!

We can kick off general discussions about features of beancount-web (and it's relation to beancount) here, and if you find a bug or have direct feedback about a component I would be happy if you directly open an issue on Github (https://github.com/aumayr/beancount-web/issues/new). Also, if a discussion about a specific feature results in a decision ("we should implement that", etc.) we can use Github issues as a TODO-list (tag "enhancement").

In any case, you're certainly welcome to use this list to discuss beancount-web issues, or LedgerHub issues as well!

[Martin Blais]

On Wed, Dec 9, 2015 at 4:37 PM, <redst...@gmail.com> wrote:

On Wednesday, December 9, 2015 at 6:35:11 AM UTC-8, Dominik Aumayr wrote:

>> Treemaps:
>> ---------
>> To generalize "drill-downs," a hierarchical chart ([1]) is among the most helpful for personal accounting, especially for Expense accounts. Some way to display this would be nice.
>> [1] https://en.wikipedia.org/wiki/Treemapping
>
> Excellent idea! I use treemaps to analyze my file system (http://www.derlien.com/), and it is very helpful! I will look for a JavaScript-library that does implement treemaps, and will implement it if I can find a good one.

Just merged the treemap-graphs into master, and am pretty happy with the treemaps!

Yes, it looks great! I like the clean, short labels and mouseover.

I assume that you're thinking of putting something in like arbitrary start and end dates, or better yet, allowing for an arbitrary query? For eg, I typically want to see my expenses minus taxes.

An arbitrary FROM clause from the SQL parser would be awesome indeed.

I also think that all the default views should be reduced to and defined by such queries.

[Martin Blais]

Just a heads-up and historical note on that "Holding" object: I'm hoping to make it go away. 

I'm currently working through a pretty drastic simplification of the Posting & Position objects in the booking branch in order to flatten the data structure, like this:

https://docs.google.com/document/d/1N7HDXuNWgLG2PqFS4Kkgv5LzAAtU6c97UVNT7tdTIjA/edit#heading=h.7pp5j2b1msmg

Also described here (NOTE: THIS IS NOT MERGED TO default YET):

https://bitbucket.org/blais/beancount/src/9cc0914b5deac62a9a15da0c00e1b65e70831a64/CHANGES?at=booking&fileviewer=file-view-default#CHANGES-7

I'm mentioning this because one of the goals of making this refactoring is to replace the "Holding" object by a good old "Posting" object, thus reducing the surface area of the API some more. I'm not 100% I can do it sanely, but I'll give it a good shot, I would really like for those resulting portfolios of Holdings to be no more than lists of Posting instances.

Anyhow, some movement in that area to be expected in the near-ish future.

 

[Martin Blais]

One kind of test that is easily implemented and which has been really beneficial in avoiding total undetected breakage in bean-web is a generic "scrape everything and check for errors" automated test, like I do it:

https://bitbucket.org/blais/beancount/src/c5252218c11b9f96c57e1423fb3f38314916149b/src/python/beancount/web/scrape.py?at=default&fileviewer=file-view-default

This just runs as part of my suite of unit tests and gives me a sense that changes to the core don't break the front-end. You may want to consider adding something like this to beancount-web.

[Martin Blais]

A suggestion: If you gave beancount-web a more distinct name, it might alleviate some of the confusion that will occur to newcomers to the project. "beancount-web" sounds pretty close to "bean-web" and I'm pretty sure not everyone will make the distinction right away, we don't have the most favorable combo of names. Of course that's your call, but I think a better name might also give the project a stronger sense of identity and thus some extra elan.

Some totally wild ideas off the top of my head (feel free to ignore me):

- "frijoles" (beans in spanish)

- "refritos" (https://en.wikipedia.org/wiki/Refried_beans)

- "congri" (https://en.wikipedia.org/wiki/Moros_y_Cristianos_(food) - a cuban rice & beans mix)

(No idea why all these spanish names come to mind.)

- "feijoada" (bean stew, https://en.wikipedia.org/wiki/Feijoada, if you prefer pt_br)

- "cassoulet" (https://en.wikipedia.org/wiki/Cassoulet, a french version)

- "fasolada" (greek version of the same theme)

- "bean-stew" (while I'm at it with the food themes)

- "legumes" (which are beans...)

- "fava" (cool large white beans I love to eat)

- "corn" or "squash" (bean, corn & squash are related: https://en.wikipedia.org/wiki/Three_Sisters_(agriculture))

- "pod" or "bean-pod" (sac that contains growing beans)

- "gassy" (one step further from eating beans, okay I'm going a bit nuts here)

Alright, resetting off the food theme:

- bean-sight (after "beancount" + "Insight")

- beanster (startup-like)

- beanjurious (adjective)

- bean-frontend (boring)

- bean-bag (it's comfy)

- nugget (a small bean of gold)

- some other name related to accounting (unfortunately many of those are already in use for other projects)

[Martin Blais]

On Sat, Dec 12, 2015 at 4:26 PM, Jeremy Maitin-Shepard <jer...@jeremyms.com> wrote:

Any ideas how to implement that?

Unfortunately I don't have much experience with web programming, so it is hard for me to know what would be fast, and I also don't know what the options and tradeoffs are as far as lazy-loading a scrollable list.  If the slowness is due to python generating the HTML, maybe there is a way to optimize the template code.  We could also just send the json to the client and generate the html using javascript.

About testing for performance: if you don't have a really large file, you can use bean-example to generate a really very big file if you'd like to stress your front-end (and Beancount itself).  bean-example is the program I wrote to generate a somewhat realistic example input file (https://bitbucket.org/blais/beancount/src/c5252218c11b9f96c57e1423fb3f38314916149b/examples/example.beancount?at=default&fileviewer=file-view-default)... I didn't write all that data by hand, I generate it with some randomness. It it distributed with Beancount and accepts parameters, e.g., you could easily generate a file with 30 years worth of data if you like.

Just an idea.

[Dominik Aumayr]

> I tried your web front-end this weekend and I have to compliment you on it, it's really beautiful and it works well.

Thanks, Martin!

> - Given how "pro" this looks, you may consider adding a global option to render the numbers in their absolute value, that is, negate the income, equity and liabilities numbers everywhere. I've been considering adding this to Beancount myself (both for input syntax and output reports actually), wondering if I could find a way to do that in a way you could reuse. This would make the presention easier to follow for "real" accountants by removing the surprising signs.

I like the idea, and created an issue on Github to discuss the feature: https://github.com/aumayr/beancount-web/issues/33

> - I've been meaning to remove the rendering of postings in journals, to make them collapsible (and collapsed by default) like you did, but to only fetch the detail from the server when expanding a particular transaction or hovering over a particular link. This would have two advantages: (1) the journals would render much faster, by avoiding rendering all of these postings details by default, which is slow, and which is really an overkill amount of detail for the most part, and (2) the journals would look smaller on-screen and be much easier to read.

The legs are collapsable via JavaScript right now, but it is very slow. Will implement that: https://github.com/aumayr/beancount-web/issues/34

> - It would be useful to render the "context" for the top four menus, e.g. if one restricts by time, a tag, or a component. Currently it only appears in the URL.

What exactly do you mean by "context"? (The current HEAD version shows if/which filters you have choosen).


Thanks again for the feedback!

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhOGpe1S42M079d_Ynz-gUwm7_EBVcbEAY399-RyF05mhw%40mail.gmail.com.

[Dominik Aumayr]

Created a Github issue to track and discuss this idea: https://github.com/aumayr/beancount-web/issues/35

> --
> You received this message because you are subscribed to the Google Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
> To post to this group, send email to bean...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhPtVZbnyf_U3mgMVfADF73Z5QPsQ0bvO6ZTHMM202uS1w%40mail.gmail.com.

[Dominik Aumayr]

> One kind of test that is easily implemented and which has been really beneficial in avoiding total undetected breakage in bean-web is a generic "scrape everything and check for errors" automated test, like I do it:
> https://bitbucket.org/blais/beancount/src/c5252218c11b9f96c57e1423fb3f38314916149b/src/python/beancount/web/scrape.py?at=default&fileviewer=file-view-default
>
> This just runs as part of my suite of unit tests and gives me a sense that changes to the core don't break the front-end. You may want to consider adding something like this to beancount-web.

Thats an excellent idea, thanks for the pointer (and the pointer to bean-example - I'm already using a 3-year-generated-file to test, but will add a bigger one as well).
(Again, for reference: https://github.com/aumayr/beancount-web/issues/36)

> --
> You received this message because you are subscribed to the Google Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
> To post to this group, send email to bean...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhOJjyTS5YgfqoLYpKt4UYbCMwM3SULNxJr%2BdbZmNJ16ig%40mail.gmail.com.

[redst...@gmail.com]

In the monthly view, the data displayed in the table goes left to right from most to least recent, while in the graph, it's the opposite.

I understand the rationale for this:

- for the table, I certainly don't want to scroll all the way to the right to find out my latest data

- for the graph, timeseries plots are usually plotted with least recent data at the left most

However, it's still confusing to look at the table and the graph and have to laterally invert one or the other all the time. I'd suggest the graph match the table whenever switching to monthly views.

[redst...@gmail.com]

In the Equity/Holdings view, it would be nice if clicking on the column headers sorted them. bean-web does this currently.

[Rubén Gómez]

>>>> Dominik Aumayr, o 04 Dec, 16:52:

> To quickly show where I want to take the web interface for beancount I created a quick-and-dirty web application:
> https://github.com/aumayr/beancount-web
> (can also be installed via "pip install git+ssh://g...@github.com/aumayr/beancount-web.git" for quick testing)

Today I was testing it and it's really awesome. The graph and the top
filters are very handy, with a few clicks I get the report I expected.

A strange behavior I noted is that starting with 'beancount-web JOURNAL'
give an 'Internal Server Error', however starting with '--debug' works
normally wihtout debug/error output then I can't file a bug report.


I spent some time and tuned the interface to my own taste, if someone is
interested in the "skin" I can share the CSS.
Screenshots are attached.

[Dominik Aumayr]

A strange behavior I noted is that starting with 'beancount-web JOURNAL'
give an 'Internal Server Error', however starting with '--debug' works
normally wihtout debug/error output then I can't file a bug report.

This is a known bug (https://github.com/aumayr/beancount-web/issues/50), and I will fix it in the next couple of days. Sorry for the inconvenience!

I spent some time and tuned the interface to my own taste, if someone is
interested in the "skin" I can share the CSS.
Screenshots are attached.

This looks really good! Can I include the CSS as a secondary "theme"/"skin" in beancount-web?

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/20151217174422.GE22276%40mordomo.mordomo.local.

For more options, visit https://groups.google.com/d/optout.

<bean-01.png>

<bean-02.png>

<bean-03.png>

<bean-04.png>

[Rubén Gómez]

> This is a known bug > (https://github.com/aumayr/beancount-web/issues/50),
> and I will fix it in the next couple of days. Sorry for the inconvenience!

Thanks!

> This looks really good! Can I include the CSS as a secondary
> "theme"/"skin" in beancount-web?

Sure, the files are attached. I use the font 'ShareTech Mono' [1]
because I prefer the zeros with a dash and also is more condensed than
the default one.
Its license is the SIL Open Font [2] allowing be embedded in web
applications.

[1] http://www.dafont.com/share-techmono.font
[2] http://scripts.sil.org/cms/scripts/page.php?item_id=OFL_web