Data::Printer in core

[Michiel Beijen]

I was attending FOSDEM last weekend and Paul (cc) explained in his talk
how some
people tend to use Data::Dumper as a tool to print data structures when
debugging, and how that is not great, and how Data::Printer really is
the right tool for that.

Then someone asked if Data::Printer actually is a core module and Paul
responded: no, but I guess that would be great, I would not know if Garu
would be OK with it..?

So I asked Garu (cc) and he responded he would be very happy if
Data::Printer would go in core!

The module has no non-core dependencies. I think it would be a great
fit. ref: https://metacpan.org/pod/Data::Printer

So I ask for the wisdom of the perl5 porters, to consider Data::Printer
to be included in core!


--

Michiel

[James E Keenan]

Before doing this we should take note of the fact that Data::Printer
appears to be sensitive to changes in core, as demonstrated by these
past BBC tickets:

https://github.com/Perl/perl5/issues/13375
https://github.com/Perl/perl5/issues/16096
https://github.com/Perl/perl5/issues/16336

Located via:
https://github.com/Perl/perl5/issues?q=is%3Aissue+sort%3Aupdated-desc+Data%3A%3APrinter+in%3Atitle%2Cbody+is%3Aclosed

That may be an argument *for* inclusion of core (faster detection of
breakage) or it may not -- but we should at least review its breakage
history.

I presume that this module will still be maintained upstream on CPAN by
Breno G. de Oliveira (GARU), correct?

Thank you very much.
Jim Keenan

[Scott Baker]

P5P:

I'm really torn on this.

On one hand I really want a better debug print in Perl core, but on the other hand the syntax and some of the limitations of Data::Printer are less than ideal.

I fully support adding some better debug style printing to core. I could probably even be convinced to support Data::Printer, but I have some concerns. I always approach Perl things from the eyes of a novice (probably because I still am a novice) and I see some simple short comings:

my $x = 3;
my $y = 4;

p($x);        # Ok
p($x, $y);    # Error?
p($x * $y);   # Error?
p([$x, $y]);  # Error?
p("$x - $y"); # Error?
p("debug");   # Error?

I understand that Data::Printer is extremely powerful and does a bunch of amazing things... but for simple novice-style use cases it's going to be very confusing if I can't be used for simple things.

I do like that it:

• Color output is great as it really helps readability
• Auto sorts hash keys to be in alphabetic order
• Aligns all the hash values on the same column for easy readability
• Is really fast
  

- Scottchiefbaker

[Michiel Beijen]

Hi Jim!

On 2/7/24 12:39, James E Keenan wrote:
> I presume that this module will still be maintained upstream on CPAN
> by Breno G. de Oliveira (GARU), correct?

Yes, I guess it would be an upstream-first CPAN module included in perl
core, just as for instance recently Test2 has been added.


The fact it had some BBC issues is because its tight integration in Perl
data structures, the debugger, and so on, on top of its extensive test
suite. In my opinion having it in core would indeed be a good thing.

--

Michiel

[Michael Conrad]

On 2/7/24 13:21, Scott Baker wrote:

P5P:

I'm really torn on this.

On one hand I really want a better debug print in Perl core, but on the other hand the syntax and some of the limitations of Data::Printer are less than ideal.

I fully support adding some better debug style printing to core. I could probably even be convinced to support Data::Printer, but I have some concerns. I always approach Perl things from the eyes of a novice (probably because I still am a novice) and I see some simple short comings:

my $x = 3;
my $y = 4;

p($x);        # Ok
p($x, $y);    # Error?
p($x * $y);   # Error?
p([$x, $y]);  # Error?
p("$x - $y"); # Error?
p("debug");   # Error?

I understand that Data::Printer is extremely powerful and does a bunch of amazing things... but for simple novice-style use cases it's going to be very confusing if I can't be used for simple things.

I do like that it:

• Color output is great as it really helps readability
• Auto sorts hash keys to be in alphabetic order
• Aligns all the hash values on the same column for easy readability
• Is really fast
  

- Scottchiefbaker

I'll second this. I like Data::Printer and use it constantly, but if you want to officially endorse it from Perl core it seems like it ought to be a function that prints a normal expression argument instead of requiring a variable.  Less confusion for new people that way.

Also, I'm not sure how I feel about it loading the config file.  On the one hand, I like the configurability of DDP to custom-tailor it to produce better dumps.  On the other hand, if everyone starts leaving "use Data::Printer" lines in their modules now because it's core, the config files could cause breakage between environments.  It would be nice if there was a stable data-printing function whose API semantics couldn't be changed by a config file.  The other option is training/education for people to use it while debugging and then remove it afterward, but that's less likely when its core.

(Semi-related, the other big debugging feature missing from core perl is Carp::Always.  How on earth are new developers supposed to realize that the way to get an extended stack trace is a CPAN module named "Carp::Always"???)

Maybe what would be nice is a design like Log::Any?  The idea where you have a "log producer" that indicates intent to show something, and a "log consumer" that lets the end user decide how to do it.  For dumping, this could be like a core perl module that does a very simplified version of Data::Printer's semantics (like not dumping the internals of sub-objects, which generally solves most of the infinite-dump problems) and then make an official API where Data::Printer can install itself as a replacement of that function.  Then mention Data::Printer as the recommended upgrade in the documentation of that function, like how IPC::Open3 recommends IPC::Run.  Or, maybe like JSON where JSON loads JSON::XS if it is installed.  Maybe this dump function could even go into builtin:: ?  "show" is a nice verb that I haven't seen many conflicts for in other modules.  Having a new "show" function available by default in perl 5.42 that automatically loads Data::Printer if installed.... would rock!

-Mike

[Darren Duncan]

On 2024-02-08 8:29 a.m., Michael Conrad wrote:
> Maybe what would be nice is a design like Log::Any?  The idea where you have a
> "log producer" that indicates intent to show something, and a "log consumer"
> that lets the end user decide how to do it.  For dumping, this could be like a
> core perl module that does a very simplified version of Data::Printer's
> semantics (like not dumping the internals of sub-objects, which generally solves
> most of the infinite-dump problems) and then make an official API where
> Data::Printer can install itself as a replacement of that function.  Then
> mention Data::Printer as the recommended upgrade in the documentation of that
> function, like how IPC::Open3 recommends IPC::Run.  Or, maybe like JSON where
> JSON loads JSON::XS if it is installed.  Maybe this dump function could even go
> into builtin:: ?  "show" is a nice verb that I haven't seen many conflicts for
> in other modules.  Having a new "show" function available by default in perl
> 5.42 that automatically loads Data::Printer if installed.... would rock!

I feel that this would be a better direction to go in, and that Data::Printer
NOT be added to core.

I feel that the best way forward is for Perl to just add a core hook and
associated "show" or whatever name built-in.

By default this "show" is a no-op, but if the user registers a handler with the
associated hook, then "show" will instead invoke that hook.

This design keeps the Perl core as minimal as possible while maximizing the
amount of choice users have as to the actual format of their debug messages.

I would rather users can pick whether they get the Data::Printer or JSON or
whatever, and also the handler implementations can evolve on CPAN as they should.

This generic support and non-favoritism can also help spawn other alternatives
for CPAN, I know I would make or am making one.

Better yet this hook/show could be generalized somewhat, be somewhat
parameterizable, so it can also serve as a generic logging interface.

So I vote AGAINST adding Data::Printer to core, and doing something more general
and bare bones like this instead.

Also, anything similar like Data::Dumper and such shouldn't be in core either,
and be accessed through the same generic mechanism I proposed.

-- Darren Duncan

[Michiel Beijen]

Hi,

On 09-02-2024 06:09, Darren Duncan wrote:
> I feel that the best way forward is for Perl to just add a core hook
> and associated "show" or whatever name built-in.
>
> By default this "show" is a no-op, but if the user registers a handler
> with the associated hook, then "show" will instead invoke that hook.
>
> This design keeps the Perl core as minimal as possible while
> maximizing the amount of choice users have as to the actual format of
> their debug messages.
>
> I would rather users can pick whether they get the Data::Printer or
> JSON or whatever, and also the handler implementations can evolve on
> CPAN as they should.
>
> This generic support and non-favoritism can also help spawn other
> alternatives for CPAN, I know I would make or am making one.
>
> Better yet this hook/show could be generalized somewhat, be somewhat
> parameterizable, so it can also serve as a generic logging interface.
>
> So I vote AGAINST adding Data::Printer to core, and doing something
> more general and bare bones like this instead.
>
> Also, anything similar like Data::Dumper and such shouldn't be in core
> either, and be accessed through the same generic mechanism I proposed.

Don't get me wrong, but you're voting against including a solution that
exists *today* and is very useful, in favor of a solution that does not
exist because that might be better.

You also suggest removing Data::Dumper from core, which I think is
rather unpractical, not in the last place because core perl uses it in
lots of tests as well as porting scripts, and other modules we ship with
perl also depend on it.

If Data::Dumper would be included in perl, this would not prevent people
from dumping out data using Data::Dumper, stringifying it to JSON, YAML
or however they please. It would just make it a little easier to print
out data using a tool designed specifically for that, unlike
Data::Dumper or JSON.

--

Michiel

[Michiel Beijen]

Hi,

On 07-02-2024 19:21, Scott Baker wrote:

On one hand I really want a better debug print in Perl core, but on the other hand the syntax and some of the limitations of Data::Printer are less than ideal.

I fully support adding some better debug style printing to core. I could probably even be convinced to support Data::Printer, but I have some concerns. I always approach Perl things from the eyes of a novice (probably because I still am a novice) and I see some simple short comings:

my $x = 3;
my $y = 4;

p($x);        # Ok
p($x, $y);    # Error?
p($x * $y);   # Error?
p([$x, $y]);  # Error?
p("$x - $y"); # Error?
p("debug");   # Error?

I understand that Data::Printer is extremely powerful and does a bunch of amazing things... but for simple novice-style use cases it's going to be very confusing if I can't be used for simple things.

I see this usage is not always ideal, it would be better if Data::Printer would try harder to 'print' any data you feed it. But I guess that is fixable.

--

Michiel

[Darren Duncan]

On 2024-02-09 12:35 a.m., Michiel Beijen wrote:
> Don't get me wrong, but you're voting against including a solution that exists
> *today* and is very useful, in favor of a solution that does not exist because
> that might be better.

I'm voting against inclusion for a number of reasons, including:

1. Data::Printer is a rather arbitrary choice to put in core, considering there
are a variety of different ways to do this.

2. Those who want to use it can use it from CPAN.

3. It is not a bootstrapping module meant to form the foundation of other things
and that benefits from being in core.

4. While some growth to core Perl is good, at the same time we still want to
default to having things just on CPAN if there isn't a strong justification to
being in core, and I don't see that here.

5. Others brought up the matters of config files and such.

Yes, Data::Printer exists today, but simply existing and being useful isn't
itself justification to prefer putting it in core.

I see adding this to core as adding core bloat and core maintainer workload
without a strong enough benefit to justify that versus just being on CPAN.

I still vote no to putting Data::Printer in core, its better on CPAN.

-- Darren Duncan

[Darren Duncan]

On further thought, I can think of one good reason to put Data::Printer in core,
which is if we intend for a lot of other core code to be altered to make use of
it, that makes them better than they are now. But unless other things in core
are going to use Data::Printer, if it is only for users, then I don't see a
strong reason to have it in core. That being said, while I'm voting no, its
more of a weak no, and if the actual core maintainers think it is better to have
Data::Printer in core, I will acquiesce. I don't have a strong case against it
that I would push. -- Darren Duncan

[Dave Mitchell]

On Fri, Feb 09, 2024 at 01:02:23PM +0100, Smylers   via perl5-porters wrote:
> There have been Cpan modules added to core Perl in the past which later
> were considered no longer to ideal for including with Perl — and then
> required continuing support because they were there, or effort in
> handling breaking backwards compatibility to remove them (or both, in
> order). All of which takes time and is a distraction from other things
> Porters could be doing.
>
> Bundled modules also ‘distort the market’. There have been plenty of
> times in Perl history where a module is popular, then a new module comes
> along which offers some benefit, and many users switch to that, and it
> becomes the new popular module. (Arguably Data::Printer did just this
> for debugging over Data::Dumper, possibly via some other options in
> between.) But it's much harder for a new module to gain mindshare from
> an existing module if that module is bundled with Perl.

+1

The general guideline (note: not strict rule) these days is that new
things should only be bundled in core because they are needed as part of
the build chain. Or maybe because they're needed to support an important
new core feature.

Users and perl packagers and OS packagers all still have the
option to install the package if it isn't bundled.

Bundling something can, sometimes unfairly, endorse that distribution
over competing or even future distributions.

If the distribution becomes under- or un-maintained, then p5p ends up
assuming the responsibility to maintain it, until such time as it gets
unbundled.

Unbundling something from core is hard and stressful and takes effort
(not technically, but politically).

It was generally agreed that the large-scale bundling that took place
some time around the 5.6.0 era was a mistake.

--
"Procrastination grows to fill the available time"
-- Mitchell's corollary to Parkinson's Law

[Michael Conrad]

On 2/9/24 04:20, Darren Duncan wrote:
> On further thought, I can think of one good reason to put
> Data::Printer in core, which is if we intend for a lot of other core
> code to be altered to make use of it

For my own code, I usually try to remove Data::Printer before committing
because of the "API affected by a config file" problem.  I also only
dump out data structures when I am debugging, or suspect a bug in the
future and want to have details in the log file.  I can't really see a
case where I would choose to use Data::Printer in a CPAN module even
aside from the API problem; that would be sort of like predicting other
people's bugs and proactively dumping things at them that I thought
would be useful to help them debug?  So... seems unlikely to be a thing
core modules would need either.

[Michael Conrad]

On 2/8/24 11:29, Michael Conrad wrote:
> Maybe what would be nice is a design like Log::Any?  The idea where
> you have a "log producer" that indicates intent to show something, and
> a "log consumer" that lets the end user decide how to do it.  For
> dumping, this could be like a core perl module that does a very
> simplified version of Data::Printer's semantics (like not dumping the
> internals of sub-objects, which generally solves most of the
> infinite-dump problems) and then make an official API where
> Data::Printer can install itself as a replacement of that function. 
> Then mention Data::Printer as the recommended upgrade in the
> documentation of that function, like how IPC::Open3 recommends
> IPC::Run.  Or, maybe like JSON where JSON loads JSON::XS if it is
> installed.  Maybe this dump function could even go into builtin:: ? 
> "show" is a nice verb that I haven't seen many conflicts for in other
> modules.  Having a new "show" function available by default in perl
> 5.42 that automatically loads Data::Printer if installed.... would rock!

Well, it turns out there is prior art!

https://metacpan.org/pod/Data::Show

Anyway, fleshing out the idea a little, what if there is a new core
module named Devel::Show.  (or CORE::Show, bikeshedding for later)  It
has two public functions: data_to_string and data_to_tty.  Each of these
is simply a wrapper around a call to $Devel::Show::data_to_string or
$Devel::Show::data_to_tty which are initialized to a simple default
implementation, but easily overridden.  The default implementation is a
bare-bones Data::Dump type thing with the semantics of dumping
structures up to a certain max size and dumping one level deep of
internals of an object.  There is also a configuration variable
$Devel::Show::preferred_module which could be changed by startup code or
configured with /etc/perl or PERL5OPT or so on.  On the first call to
one of the public functions, it tries to load the configured module, and
then calls $data_to_X which would have been set to the chosen module's
implementation.

The documentation for Devel::Show would explain to end users how to
configure their favorite dumper, using Data::Printer as the prime example.

The 'show' builtin would accept parameters exactly as Damian's
Data::Show, and simply run the equivalent of

sub show {
  say -t \*STDOUT
    ? Devel::Show::data_to_tty(...)
    : Devel::Show::data_to_string(...);
  1 # always return true, for "show $x and ..." type of code
}

By making the to-string functions separate from the 'show' builtin it
lets people opt to use those same functions for logging systems.  By
separating the TTY case, it lets code intelligently decide whether or
not to use color escapes or ascii art.  By putting the overridable
functions in scalar variables it lets you more cleanly
{ local $Devel::Show::data_to_string= ... }
without exports and compilation preventing your override from taking
place.  It also makes them easy to wrap without fiddling with the symbol
table.

I guess this is turning into an elevator pitch.

-Mike

[Darren Duncan]

On 2024-02-09 9:52 a.m., Michael Conrad wrote:
> Well, it turns out there is prior art!
>
> https://metacpan.org/pod/Data::Show
>
> Anyway, fleshing out the idea a little, what if there is a new core module named
> Devel::Show.  (or CORE::Show, bikeshedding for later)  It has two public
> functions: data_to_string and data_to_tty.  Each of these is simply a wrapper
> around a call to $Devel::Show::data_to_string or $Devel::Show::data_to_tty which
> are initialized to a simple default implementation, but easily overridden.  The
> default implementation is a bare-bones Data::Dump type thing with the semantics
> of dumping structures up to a certain max size and dumping one level deep of
> internals of an object.  There is also a configuration variable
> $Devel::Show::preferred_module which could be changed by startup code or
> configured with /etc/perl or PERL5OPT or so on.  On the first call to one of the
> public functions, it tries to load the configured module, and then calls
> $data_to_X which would have been set to the chosen module's implementation.

What you're saying here sounds similar to what I said and that you argued
against because it was something that didn't exist. Did you change your mind?
-- Darren Duncan

[Darren Duncan]

Thank you both Smylers and Dave.

You have laid out here (with more in Smyler's message that were omitted here)
essentially the key reasons that I was arguing for voting NO to putting
Data::Printer in core, including that it would distort the market.

-- Darren Duncan

[Michael Conrad]

(check the last names.  I guess I need to upgrade my signature to "-Mike C")

I agreed with most of your ideas and incorporated some into my extended
idea, though, think it should have a useful default rather than no-op if
it's going to be a builtin, and while I'd love an optimized logging
infrastructure in core, that's a collection of bikesheds the size of
Tokyo.  I think narrowing the scope to "these are the user's preferences
for how to render structures for human viewing" is enough to be useful,
and can be later integrated into logging.

-Mike C

[Darren Duncan]

On 2024-02-09 1:23 p.m., Michael Conrad wrote:
> On 2/9/24 15:25, Darren Duncan wrote:
>> What you're saying here sounds similar to what I said and that you argued
>> against because it was something that didn't exist.  Did you change your mind?
>

> (check the last names.  I guess I need to upgrade my signature to "-Mike C")

UGH! I'm sorry, that slipped past me, I confused you with the thread starter
Michael. I retract my above comment. -- Darren Duncan

[Darren Duncan]

On 2024-02-09 3:46 p.m., Robert Rothenberg wrote:
> Including modules in core is basically a way of appeasing people who are unable
> unwilling to install CPAN modules. We have things like perlbrew, plenv,
> local::lib, cpanm/cpm, Carton/Carmel etc. to make it easier to have alternative
> versions of Perl and alternative/local module installs and yet some people or
> their system administrators won't. Likely these people will be using older
> versions of Perl and won't upgrade to a version that has newer modules anyhow.

Speaking generally, and not to the merits of any specific thing being in core...

I don't see that it is to our benefit to cater further than we already have to
users/administrators that refuse to install things from CPAN, by including
things in core that otherwise wouldn't be there.

Anyone in that situation is already in a crippled situation since that also
means huge swaths of functionality are intentionally avoided, for example the
DBI module and friends are not in core and need to be installed from CPAN so
they're denying themselves a lot of databases.

And if administrators are installing things from CPAN on their users' behalf to
get beyond core Perl, they can do so arbitrarily for anything else on CPAN also.

-- Darren Duncan

[Michael Conrad]

On 2/9/24 18:46, Robert Rothenberg wrote:
> After reading and thinking about the arguments here, I've changed my
> mind slightly. On it's own something like Data::Printer is nice, but
> what would be really useful for novices and seasoned developers alike
> is a REPL. Something like Data::Printer is fine along with of a REPL
> that uses it, and could be included in core.

Technically "perl -de 1" is already a REPL.  And while I'm fully aware
of other cpan REPL options, I mostly develop without them, usually
exploring things with "perl -E" or writing small doodles in Test2 if I
think there's a chance it'll become testable functionality.  These are
very different from where I usually use Data::Printer; I'm usually half
way down a multi-hundred line module and curious about a variable, and
type "use DDP; &p([$foo->bar]);" into the middle, then re-run my unit
test.  I'm proposing to replace that with "show $foo->bar" so long as
it's running in the scope of v5.42 or whatever version it becomes
standard in.

But, the concept of a configured user preference of how to dump things
could certainly be useful for a REPL, and would make a great default for
the perl debugger too!

-Mike C.

[James E Keenan]

I agree with Dave's argument, particularly the part about things should
only be bundled in core if they're needed as part of the build chain or
in support of new core features.

If someone were to look over the modules currently shipped with core
(both those maintained upstream on CPAN in the cpan/ directory and those
maintained by P5P in the dist/ directory), you would see many familiar
modules -- but also many about which you would ask, "WTF is *that*?"

Hence, I don't support inclusion of Data::Printer in core.

(I'll just add in passing that for at least a decade Data::Dump has been
my go-to Data::Dumper substitute -- but I've never felt it should be
shipped with core.)

[G.W. Haywood]

Hi there,

On Mon, 12 Feb 2024, Smylers   via perl5-porters wrote:

> ... it's still better to make it easy for users to install the
> actual modules they want than for P5P to predict ...

+1e2

--

73,
Ged.