sorting a ledger file

[Ralph J.Mayer]

Hi,

is there a way to sort a ledger file by date?
(the file not the output)

cat file | sort does obviously not work since every entry is three lines

Any pointers?

--
Viele Grￜￜe / Kind Regards / Cordiali Saluti / Met vriendelijke groet


Ralph J.Mayer

[Craig Earls]

Two ways. Use the 'print' command from ledger. Or use the sort and cleanup function in ledger-mode for emacs

--
Craig

On Fri, Aug 23, 2013 at 11:17 AM, Ralph J.Mayer <rma...@nerd-residenz.de> wrote:

Hi,

is there a way to sort a ledger file by date?
(the file not the output)

cat file | sort does obviously not work since every entry is three lines

Any pointers?

--

Viele Grüße / Kind Regards / Cordiali Saluti / Met vriendelijke groet


Ralph J.Mayer

--

---
You received this message because you are subscribed to the Google Groups "Ledger" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ledger-cli+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[John Wiegley]

>>>>> Ralph J Mayer <rma...@nerd-residenz.de> writes:

> is there a way to sort a ledger file by date?
> (the file not the output)

If you use comments, automated expressions, etc., then the answer is no.
Ledger 2.x had a "write" command for doing this, but I never re-implemented it
for Ledger 3.0.

John

[Craig Earls]

I forgot about that. The ledger-mode cleanup function honors those
sorts of things. It is all I use these days.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "Ledger" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ledger-cli+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

--
Craig, Corona De Tucson, AZ
enderw88.wordpress.com

[Ralph J.Mayer]

Am 23.08.13 20:25, schrieb Craig Earls:

> I forgot about that. The ledger-mode cleanup function honors those
> sorts of things. It is all I use these days.

But ... vim! *sigh*


--
Viele Gr��e / Kind Regards / Cordiali Saluti / Met vriendelijke groet


Ralph J.Mayer

[Craig Earls]

So what I did back before ledger-mode had the clean up function was to
use the print command and I had a script that added in the automatic
transactions I used. I avoided file level comments.

On Fri, Aug 23, 2013 at 11:47 AM, Ralph J.Mayer <rma...@nerd-residenz.de> wrote:
> Am 23.08.13 20:25, schrieb Craig Earls:
>
>> I forgot about that. The ledger-mode cleanup function honors those
>> sorts of things. It is all I use these days.
>
>
> But ... vim! *sigh*
>
>
>
> --

> Viele Grüße / Kind Regards / Cordiali Saluti / Met vriendelijke groet
>
>
> Ralph J.Mayer
>

[Ralph J.Mayer]

>> is there a way to sort a ledger file by date?
>> (the file not the output)
>
> If you use comments, automated expressions, etc., then the answer is no.
> Ledger 2.x had a "write" command for doing this, but I never re-implemented it
> for Ledger 3.0.

so, I should keep a 2.x somewhere for this.

thx for this awesome pice of software!


--
Viele Gr��e / Kind Regards / Cordiali Saluti / Met vriendelijke groet


Ralph J.Mayer

[John Wiegley]

>>>>> Ralph J Mayer <rma...@nerd-residenz.de> writes:

>>> is there a way to sort a ledger file by date?
>>> (the file not the output)
>>
>> If you use comments, automated expressions, etc., then the answer is no.
>> Ledger 2.x had a "write" command for doing this, but I never re-implemented it
>> for Ledger 3.0.

> so, I should keep a 2.x somewhere for this.

I don't think 2.x can parse every 3.0 file.

John

[Ralph J.Mayer]

>> so, I should keep a 2.x somewhere for this.
>
> I don't think 2.x can parse every 3.0 file.

so, any plans on reimplementing the print command someday?

[Craig Earls]

Have you asked the vim-ledger guys it is feasible?

On Fri, Aug 23, 2013 at 12:04 PM, Ralph J.Mayer <rma...@nerd-residenz.de> wrote:
>>> so, I should keep a 2.x somewhere for this.
>>
>>
>> I don't think 2.x can parse every 3.0 file.
>
>
> so, any plans on reimplementing the print command someday?
>
>
>
> --

> Viele Grüße / Kind Regards / Cordiali Saluti / Met vriendelijke groet
>
>
> Ralph J.Mayer
>

[Charles Merriam]

The print command is limited and requires a bit of eyeballing to catch the errors.

For example, this file:

bucket out

2013/1/1 Foo

   in    $0.01

Has the correct register: 

$ ledger -f x.led register

13-Jan-01 Foo                   in                            $0.01        $0.01

                                        out                          $-0.01            0 

But the incorrect print statements:

$ ledger -f x.led print

2013/01/01 Foo

    in                                         $0.01

It feels like a proper format command would be able to toss out an idempotent register with "-S date" for the sorting.  Idempotent is the fancy way of saying running the command against its own output would still reproduce the output.

[Craig Earls]

It would probably be easier to implement this sort of thing in Python
as a standalone script than to coerce ledger into doing it. Imagine a
realistic ledger file that contained a "bucket" statement (or any
other file level command or comment) in the down in the body of the
file. Ledger allows it, and some people actually use it that way, but
if you sorted the file where should the bucket statement end up? There
was a long discussion about this last year or the year before when it
came to how to handle file level comments and commands. It is the
only case I can think of where ledgers flexibility actually gets in
the way of a simple task.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "Ledger" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ledger-cli+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Ralph J.Mayer]

> If you use comments, automated expressions, etc., then the answer is no.
> Ledger 2.x had a "write" command for doing this, but I never re-implemented it
> for Ledger 3.0.

Turns out hledger has print implemented.


--
Viele Gr��e / Kind Regards / Cordiali Saluti / Met vriendelijke groet


Ralph J.Mayer

[Craig Earls]

How does it handle file level comments and auto/virtual cactus?

--
Craig

On Sun, Aug 25, 2013 at 11:32 AM, Ralph J.Mayer <rma...@nerd-residenz.de> wrote:

> If you use comments, automated expressions, etc., then the answer is no.
> Ledger 2.x had a "write" command for doing this, but I never re-implemented it
> for Ledger 3.0.

Turns out hledger has print implemented.


--

Viele Grüße / Kind Regards / Cordiali Saluti / Met vriendelijke groet


Ralph J.Mayer

[Ralph J.Mayer]

Am 25.08.13 20:42, schrieb Craig Earls:

> How does it handle file level comments and auto/virtual cactus?

I think limited. Heres a comparison
http://hledger.org/FAQ.html#how-does-hledger-relate-to-ledger

--
Viele Gr��e / Kind Regards / Cordiali Saluti / Met vriendelijke groet


Ralph J.Mayer

[Simon Michael]

hledger's print throws away top-level comments and directives, like
ledger's. I think it's not supposed to - there are data structures to
hold those things - but currently it does.

Other than patching h/ledger, sorting with the emacs mode (perhaps
hidden behind a command-line script) or an awk script sounds like the
best option.

[Craig Earls]

In the previous message cactus==xact 

--
Craig

On Mon, Aug 26, 2013 at 11:33 AM, Craig Earls <ende...@gmail.com> wrote:

OK. I am willing to work on an external script for this ( probably using leder print to do the hard work)

If any is interested in this sort of tool please me know how you personally would lik to handle non-cactus entries. 

There are many options, here are a few I can think of:

1. Comments, virtuals and autos that occurr before the first real xact will be left at the head of the file. 

2. File level comments are tied to the xact immediately following and will sort as such. There might be other smarter ways to handle this. 

3. File level commands that occur in the body of the file serve as separators and all xacts that occur after that command will stay after that command regardless of date. Should file level comments be the same?  

Optionally I can make the tool

Do a check. A simple ledger bal on the new file should be identical. 

Any other ideas?  I would probably do this in python unless there is a more convenient scripting tool (bash is right out. Don't ask!)

Craig

--
Craig

[Russell Adams]

Sorting and narrowing ledger data was a major issue for me. The
closest I got was using reports in Emacs to allow me to hyperlink to
ledger entries in any file.

When we start talking about sorting and queries (narrowing) this is
where the flat file format starts to be an issue vs a kind of
database.

Throw in the integration with version control, and this is a real
pickle. It certainly frustrated me.

I'd love to see how this is implemented, and welcome the discussion of
features.

On Mon, Aug 26, 2013 at 11:33:58AM -0700, Craig Earls wrote:
> OK. I am willing to work on an external script for this ( probably using leder print to do the hard work)
>
>
>
>
>
> If any is interested in this sort of tool please me know how you personally would lik to handle non-cactus entries.�
>
>
>
>
>
> There are many options, here are a few I can think of:
>
>
>
>
>
> 1. Comments, virtuals and autos that occurr before the first real xact will be left at the head of the file.�
>
> 2. File level comments are tied to the xact immediately following and will sort as such. There might be other smarter ways to handle this.�
>
> 3. File level commands that occur in the body of the file serve as separators and all xacts that occur after that command will stay after that command regardless of date. Should file level comments be the same? �
>
>
>
>
>
> Optionally I can make the tool
>
> Do a check. A simple ledger bal on the new file should be identical.�
>
>
>
>
>
> Any other ideas? �I would probably do this in python unless there is a more convenient scripting tool (bash is right out. Don't ask!)
>
>
>
>
>
> Craig
>
> --
> Craig
>
> On Mon, Aug 26, 2013 at 8:49 AM, Simon Michael <si...@joyful.com> wrote:
>

> > --
> > ---
> > You received this message because you are subscribed to the Google Groups "Ledger" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to ledger-cli+...@googlegroups.com.
> > For more options, visit https://groups.google.com/groups/opt_out.
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups "Ledger" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ledger-cli+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

------------------------------------------------------------------
Russell Adams RLA...@AdamsInfoServ.com

PGP Key ID: 0x1160DCB3 http://www.adamsinfoserv.com/

Fingerprint: 1723 D8CA 4280 1EC9 557F 66E8 1154 E018 1160 DCB3

[Harshad RJ]

On Tue, Aug 27, 2013 at 12:10 AM, Russell Adams <RLA...@adamsinfoserv.com> wrote:

When we start talking about sorting and queries (narrowing) this is
where the flat file format starts to be an issue vs a kind of
database.

I don't think the flat file format by itself is an impediment. If the semantics are well defined and declarative, then it is as good as a structured database, and yet, being  more convenient to humans.

I had mentioned this in a previous thread. Sorry to harp on it again, but it really would be good to reconsider this suggestion:

• Don't
  leak parsing order and other such internal details.
• Make the declaration order irrelevant.

While this might make some rare scenarios a little cumbersome, I think it will be a net win. I have implemented this exact suggestion when I joined a team building a large commercial DSL. It simplified the DSL and also the parser and processing engine. It also allowed easy development of IDEs where partial processing of input could be done in real time. That was couple of years ago and that team still thanks me for it.

--
Harshad RJ

[Craig Earls]

RJ,
I think your idea is great. John would have to comment on its
practicality of implementation at this point. I also don't know how
many people already depend on the behavior as it currently sits. I
intentionally avoid use of commands in the body of the file
specifically because of this issue. Others may differ in their use
by, for example, changing the bucket directive mid file and keeping
different financial institutions indifferent parts of the file.
Blindly sorting that by date would be disastrous. Org users would
probably be the best candidates for checking this.

Craig

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "Ledger" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ledger-cli+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.



--

[Russell Adams]

I can see the point where commands which apply across many
transactions (a manual optimization) are broken by sorting. My data
has always been verbose and each txn is atomic, so sorting wasn't an
issue.

Thanks.

[Craig Earls]

My transactions are also all atomic. With the exception of a few
virtual transactions at the top order does no matter.

On Mon, Aug 26, 2013 at 1:17 PM, Russell Adams
<RLA...@adamsinfoserv.com> wrote:
> I can see the point where commands which apply across many
> transactions (a manual optimization) are broken by sorting. My data
> has always been verbose and each txn is atomic, so sorting wasn't an
> issue.
>
> Thanks.

[Craig Earls]

OK. I am willing to work on an external script for this ( probably using leder print to do the hard work)

If any is interested in this sort of tool please me know how you personally would lik to handle non-cactus entries. 

There are many options, here are a few I can think of:

1. Comments, virtuals and autos that occurr before the first real xact will be left at the head of the file. 

2. File level comments are tied to the xact immediately following and will sort as such. There might be other smarter ways to handle this. 

3. File level commands that occur in the body of the file serve as separators and all xacts that occur after that command will stay after that command regardless of date. Should file level comments be the same?  

Optionally I can make the tool

Do a check. A simple ledger bal on the new file should be identical. 

Any other ideas?  I would probably do this in python unless there is a more convenient scripting tool (bash is right out. Don't ask!)

Craig

--
Craig

On Mon, Aug 26, 2013 at 8:49 AM, Simon Michael <si...@joyful.com> wrote:

[Craig Earls]

So, is there any interest in this project? If not I won't waste my time.

[Russell Adams]

Craig,

As much as I love ledger, I've moved my expenses back to Excel using a
giant T chart. I spent about 3 months trying to write a database
oriented ledger-like expense reporting system, and failed miserably at
the UI layer. So I wouldn't get any use out of the tool you propose.

Ledger's major flexibility is the text file format and the ability to
hand craft optimized ledgers. That is also it's downside in large
volumes of verbose (non-optimal) transactions quickly become unwieldy.

Sorting and narrowing (queries) are both related to data management,
and I'm not sure how that should be accomplished in the text files
while staying synchronized with version control tools.

I think a simple sort that may break hand optimized ledgers would be
ok if it raised an error on a break (ie: if txn count and balances
changed after sort, raise error). A tool like that should be easy to
implement.

Otherwise it comes back to the debate on txn storage.

Thanks.

[Charles Merriam]

So, is the answer to figure out how to optimize the ledger entries?

So if I had:

2013/1/1 Safeway
Expenses:Groceries $23.00
Liabilities:Chase Visa -$23.00

and I add:

account Expenses:Groceries
alias grocs

account Liabilities:Chase Visa
alias visa

should a sort give:

2013/1/1 Safeway
grocs $23.00 ; using the alias I just added
visa ; balance the transaction shortcut

This does open up the can of worms: how should a ledger file look?
is one better? when to add a bucket?

I guess I looked at documentation as the low hanging fruit, but failed
to figure out its build system in a day. It is unclear how welcome
little fixes are, and I get tired of making patches that get ignored.

> You received this message because you are subscribed to a topic in the Google Groups "Ledger" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/ledger-cli/WsIPp_6sn_k/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to ledger-cli+...@googlegroups.com.

[Craig Earls]

With respect to patches, if I have ignored any documentation patches I
am sorry, I completely missed them. As far as building the docs, cd
into the docs directory and run the gendocs script ../tools/gendocs.sh

[Russell Adams]

Charles,

I see that as a hand optimization for manually created files. There
are many directives that span txns and break if the order changes, I
think that's the primary concern with sorting.

My txns were very verbose by comparison, and they sorted easily
because they were atomic.

Thanks.

[John Wiegley]

>>>>> Russell Adams <RLA...@AdamsInfoServ.Com> writes:

> As much as I love ledger, I've moved my expenses back to Excel using a
> giant T chart.

And there was NOT much rejoicing. :(

John

[Russell Adams]

I'm disappointed in it too. :P

The fact I can't avoid after 5 years of effort is that ledger is a
tool only I can use, and I've had to grow a small ecosystem of scripts
around it to maintain it.

I'm hoping to be able to hand Excel off eventually, while it's big
it's not difficult.

One major step back also was the loss of the ability to tie a specific
receipt image to a transaction. Now I'm manually organizing the images
weekly during scanning.

[John Wiegley]

>>>>> Russell Adams <RLA...@AdamsInfoServ.Com> writes:

> One major step back also was the loss of the ability to tie a specific
> receipt image to a transaction. Now I'm manually organizing the images
> weekly during scanning.

It should be fairly easily to write a Ledger parser for your Excel file, no,
so that you can still use Ledger for querying?

John

[Russell Adams]

Believe it or not, Excel's table view allows sorting and queries
(narrowing), so I haven't had to export at all. The occasional pivot
table for aggregation, and that's it.

I don't like it, but after coding on TERT for a while, I gave up on
UI's in disgust.

[Simon Michael]

On 8/28/13 9:46 AM, Charles Merriam wrote:
> I guess I looked at documentation as the low hanging fruit, but failed
> to figure out its build system in a day. It is unclear how welcome
> little fixes are, and I get tired of making patches that get ignored.

On 8/28/13 9:49 AM, Craig Earls wrote:
> With respect to patches, if I have ignored any documentation patches I
> am sorry, I completely missed them. As far as building the docs, cd
> into the docs directory and run the gendocs script ../tools/gendocs.sh

Comments like Charles' are a bad sign and should be taken very
seriously, if we want this to be a thriving project. I'm glad Craig replied.

Regarding the ledger 3 manual. The script Craig mentions generates the
info and pdf formats, but not the html format. The html version is what
most folks use, via http://ledger-cli.org/docs.html . I also had severe
problems figuring out the current build system. Here's what I came up
with to build the html docs for ledger-cli.org:

./acprep make doc -- -DBUILD_WEB_DOCS=1

But I see that's not working for me today. Indeed
http://ledger-cli.org/3.0/doc/ledger3.html seems not to have been
regenerated since 2013/1/29! Which is really confusing and a black mark
for me if true. I thought I had been testing it more recently than that.
To fix the official manual on the website, I need a reliable command
that I can use to build the html docs, from whoever installed the
current build system (or anyone).

As for the rest of the website, contributing to this should be easy.
http://ledger-cli.org -> contribute -> About ledger-cli.org describes
the setup and points to the github repo. Any patch or pull request which
you get committed there will appear on the website right away (fixed
yesterday).