cperl-mode and fontlock

[Harry Putnam]

(I've cross posted to emacs-devel on the off chance this is strictly a
cvs-emacs problem)
Running recent cvs-emacs on Gentoo linux

Does every one have trouble with syntax highlight in cperl-mode or is
it a local phenomena for me?

I do have `(global-font-lock-mode t)' set in site-start.el

And this in ~/.emacs
===== * ===== * ===== * =====

;;; Make syntax highlight in shell mode show better colors...
;;; No more dark glaring blue on dark terminal
(eval-after-load 'ansi-color
'(progn
(setq ansi-color-names-vector ["gray50" "salmon" "palegreen"
"lightgoldenrod" "SkyBlue2"
"plum3" "turquoise" "white"])
(setq ansi-color-map (ansi-color-make-color-map))))
===== * ===== * ===== * =====

What I see in X, is a general loss of font-locking as I work along.
(Haven't tested in console mode)

As I enter code... syn-hilite appears to be working just as expected
but as time wears on and some of the code scrolls off the screen,
Eventually when viewed later syn-hilite has disappeared in all but a
few items.

Just scrolling code off visible area and back doesn't appear to do it
... I'm not sure exactly what does.

If I return to edit some already entered code that has lost its
syn-hilite, the hilite returns the instant I make any
modification. But then later I see its left again.

Calling M-x `font-lock-fontify-buffer' does not seem to re-instate the
syntax highlight where it has already gone away either.

And its not the kind of thing you see when you've forgotten a closing
quote. In that case the whole section turns a uniform color.
What I'm seeing is return to default face after a while. Instead of
the faces specified by syntax highlight code. And not all items have
turned back to default... some remain hilited.

An example of something that appear to routinely loose thier highlight
are perls' `my' and `our' stipulators. `While' and `if' fall prey after while
too, including coding between the curly braces ({...}).

Some things that stay highlighted are things inside quotes, and things
inside the regular expression operators (//).

Any ideas what to do about it?

[Ilya Zakharevich]

On 2009-09-17, Harry Putnam <rea...@newsguy.com> wrote:
> As I enter code... syn-hilite appears to be working just as expected
> but as time wears on and some of the code scrolls off the screen,
> Eventually when viewed later syn-hilite has disappeared in all but a
> few items.
>
> Just scrolling code off visible area and back doesn't appear to do it
> ... I'm not sure exactly what does.
>
> If I return to edit some already entered code that has lost its
> syn-hilite, the hilite returns the instant I make any
> modification. But then later I see its left again.
>
> Calling M-x `font-lock-fontify-buffer' does not seem to re-instate the
> syntax highlight where it has already gone away either.

^^^^^

What is this "where"? Region of buffers, or buffers as a whole?

Just 2 pieces of how I view similar problems:

a) cperl-mode distributed with RMS Emacs is completely borken. See
"my" versions;

b) I think I have seen things similar to what you describe - about
5-7 years ago.

IIRC, font-locking code is designed (?) to make things as hard to
debug as possible. Everything is run inside try/catch blocks,
and all error messages are thrown away. However, some problems
accumulate; when too many are accumulated, font-locking stops
working in the buffer - until you do something like M-x
normal-mode, which would reset font-locking state.

So the net result was that bugs in cperl-mode would be invisible
until they damage font-lock internal state enough to stop it from
working. (But this was long time ago, and THESE bugs are long gone...)

Hope this helps,
Ilya

[Harry Putnam]

[ALERT - I've attempted cross posting this to devel as
well -ED rea...@newsguy.com]

Ilya Zakharevich <nospam...@ilyaz.org> writes:

> On 2009-09-17, Harry Putnam <rea...@newsguy.com> wrote:
>> As I enter code... syn-hilite appears to be working just as expected
>> but as time wears on and some of the code scrolls off the screen,
>> Eventually when viewed later syn-hilite has disappeared in all but a
>> few items.
>>
>> Just scrolling code off visible area and back doesn't appear to do it
>> ... I'm not sure exactly what does.
>>
>> If I return to edit some already entered code that has lost its
>> syn-hilite, the hilite returns the instant I make any
>> modification. But then later I see its left again.
>>
>> Calling M-x `font-lock-fontify-buffer' does not seem to re-instate the
>> syntax highlight where it has already gone away either.
> ^^^^^
>
> What is this "where"? Region of buffers, or buffers as a whole?

Sorry, even though english is my first language I speak it with near
illiteracy.

`where' was supposed to mean any spots in the code where font-locking
has already stopped working, are not revivied by M-x
font-lock-fontify-buffer. That is, I can see some code with no
font-locking visible in current buffer, but it does not get fixed when
font-lock-fontify-buffer is run.

That isn't quite right either, because some of it does get fixed.
Also with the M-x normal-mode seems to re-font-lock something but not
others. Very confusing and maddening.

If I edit a block where many elements have lost syntax coloring... For
a brief while the syntax coloring is re-established near the cursor
line.

Maybe 2/3 lines in either direction.

Another part to this... not sure if its related is that mic-paren-mode
is now giving false readings at times... I've never noticed that
before.

Any (`) backtick between matching parens (even in comments...) might
cause a false reading it seems

> Just 2 pieces of how I view similar problems:
>
> a) cperl-mode distributed with RMS Emacs is completely borken. See
> "my" versions;

This is the version that came with emacs-23:
(From top of the file)

;;; cperl-mode.el --- Perl code editing commands for Emacs

;; Copyright (C) 1985, 1986, 1987, 1991, 1992, 1993, 1994, 1995, 1996, 1997,
;; 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009
;; Free Software Foundation, Inc.

;; Author: Ilya Zakharevich
;; Bob Olson
;; Maintainer: Ilya Zakharevich <il...@cpan.org>
;; Keywords: languages, Perl

;; This file is part of GNU Emacs.

;; GNU Emacs is free software: you can redistribut...

[...]

Is there something newer or better available?

>
> b) I think I have seen things similar to what you describe - about
> 5-7 years ago.

It appears my version of cperl-mode is not totally outdated eh?.

Any other problems that might cause this? something to do with shadow
paths or whatever its called?

I went to the cperl-mode.el file by calling M-x find-library inside
emacs. So, I guess that means that is the one loading... right.

Any other input would be greatly appreciated... My coding skills are
terrible enough without having syntax coloring problems.

[Harry Putnam]

Ilya Zakharevich <nospam...@ilyaz.org> writes:

> a) cperl-mode distributed with RMS Emacs is completely borken. See
> "my" versions;

Ilya, sorry I don't know enough to know the answer to this but can I
safely ignore the list of warnings I get when byte compiling your most
recent (6.2) version (using emacs-23 from cvs).

Here is the list:

Compiling file /usr/share/emacs/23.1.50/lisp/progmodes/cperl-mode.el at Fri Sep 25 20:39:03 2009
cperl-mode.el:1563:1:Warning: !! The file uses old-style backquotes !!
This functionality has been obsolete for more than 10 years already
and will be removed soon. See (elisp)Backquote in the manual.
cperl-mode.el:1645:42:Warning: value returned from (fboundp (quote
eval-when-compile)) is unused
cperl-mode.el:2140:1:Warning: !! The file uses old-style backquotes !!
This functionality has been obsolete for more than 10 years already
and will be removed soon. See (elisp)Backquote in the manual.
cperl-mode.el:2571:1:Warning: !! The file uses old-style backquotes !!
This functionality has been obsolete for more than 10 years already
and will be removed soon. See (elisp)Backquote in the manual.
cperl-mode.el:2608:11:Warning: assignment to free variable
`unread-command-event'
cperl-mode.el:2672:1:Warning: !! The file uses old-style backquotes !!
This functionality has been obsolete for more than 10 years already
and will be removed soon. See (elisp)Backquote in the manual.
cperl-mode.el:2674:8:Warning: `mapcar' called for effect; use `mapc' or
`dolist' instead
cperl-mode.el:2689:17:Warning: value returned from (fboundp (quote
eval-when-compile)) is unused
cperl-mode.el:3104:1:Warning: !! The file uses old-style backquotes !!
This functionality has been obsolete for more than 10 years already
and will be removed soon. See (elisp)Backquote in the manual.

In cperl-mode:
cperl-mode.el:3388:51:Warning: `vc-header-alist' is an obsolete variable (as
of Emacs 21.1); use `vc-BACKEND-header' instead.
cperl-mode.el:3479:10:Warning: `make-local-hook' is an obsolete function (as
of Emacs 21.1); not necessary any more.

In cperl-electric-brace:
cperl-mode.el:3655:20:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.
cperl-mode.el:3694:57:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.
cperl-mode.el:3669:33:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.
cperl-mode.el:3693:55:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.

In cperl-electric-lbrace:
cperl-mode.el:3735:14:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.
cperl-mode.el:3736:16:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.
cperl-mode.el:3740:16:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.

In cperl-electric-paren:
cperl-mode.el:3761:22:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.
cperl-mode.el:3772:32:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.
cperl-mode.el:3772:26:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.

In cperl-electric-rparen:
cperl-mode.el:3796:20:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.
cperl-mode.el:3806:32:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.
cperl-mode.el:3806:26:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.

In cperl-electric-keyword:
cperl-mode.el:3818:26:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.
cperl-mode.el:3820:28:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.
cperl-mode.el:3874:22:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.

In cperl-electric-pod:
cperl-mode.el:3895:28:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.

In cperl-electric-terminator:
cperl-mode.el:4115:33:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.
cperl-mode.el:4129:31:Warning: `last-command-char' is an obsolete variable (as
of Emacs at least 19.34); use `last-command-event' instead.

In cperl-calculate-indent:
cperl-mode.el:4820:9:Warning: `error' called with 1 args to fill 0 format
field(s)
cperl-mode.el:4810:25:Warning: `error' called with 1 args to fill 0 format
field(s)
cperl-mode.el:4821:10:Warning: `format' called with 1 args to fill 0 format
field(s)

In cperl-find-pods-heres:
cperl-mode.el:5713:60:Warning: assignment to free variable `name'
cperl-mode.el:6290:38:Warning: reference to free variable `name'

In cperl-indent-region:
cperl-mode.el:6902:4:Warning: value returned from (fboundp (quote
imenu-progress-message)) is unused

In cperl-imenu-addback:
cperl-mode.el:7098:11:Warning: `mapcar' called for effect; use `mapc' or
`dolist' instead
cperl-mode.el:7342:1:Warning: !! The file uses old-style backquotes !!
This functionality has been obsolete for more than 10 years already
and will be removed soon. See (elisp)Backquote in the manual.

In cperl-xsub-scan:
cperl-mode.el:8580:24:Warning: `imenu-example--name-and-position' is an
obsolete function (as of Emacs 23.2); use your own function instead.
cperl-mode.el:8578:11:Warning: `imenu-example--name-and-position' is an
obsolete function (as of Emacs 23.2); use your own function instead.

In cperl-write-tags:
cperl-mode.el:8734:12:Warning: `mapcar' called for effect; use `mapc' or
`dolist' instead

In cperl-tags-hier-init:
cperl-mode.el:8858:12:Warning: `mapcar' called for effect; use `mapc' or
`dolist' instead
cperl-mode.el:8867:10:Warning: `mapcar' called for effect; use `mapc' or
`dolist' instead
cperl-mode.el:8867:32:Warning: `mapcar' called for effect; use `mapc' or
`dolist' instead

In cperl-tags-treeify:
cperl-mode.el:8933:6:Warning: `mapcar' called for effect; use `mapc' or
`dolist' instead
cperl-mode.el:8939:6:Warning: `mapcar' called for effect; use `mapc' or
`dolist' instead
cperl-mode.el:8957:6:Warning: `mapcar' called for effect; use `mapc' or
`dolist' instead
cperl-mode.el:8947:17:Warning: `mapcar' called for effect; use `mapc' or
`dolist' instead
cperl-mode.el:8954:55:Warning: `mapcar' called for effect; use `mapc' or
`dolist' instead

In cperl-time-fontification:
cperl-mode.el:10432:11:Warning: `goto-line' used from Lisp code
That command is designed for interactive use only

In cperl-emulate-lazy-lock:
cperl-mode.el:10459:59:Warning: assignment to free variable
`font-lock-cache-position'
cperl-mode.el:10499:10:Warning: `make-variable-buffer-local' should be called
at toplevel

[Harry Putnam]

Ilya Zakharevich <nospam...@ilyaz.org> writes:

> a) cperl-mode distributed with RMS Emacs is completely borken. See
> "my" versions;

Just to let you know... I guess the newest emacs-23 cvs version has a
version that is pretty close to your 6.2 version. Although I notice
quite a hefty byte count difference.

-rw-r--r--+ 1 reader reader 329659 Sep 25 20:56 cperl-mode.el
-rw-r--r--+ 1 reader reader 402912 Sep 25 21:05 cperl-mode.el-ilya

The one with the ilya extension I just downloaded from your repo. And
the emacs-23 from cvs just now too.

As you see the size is different by 73253 bytes.

I found when I load either yours or the cvs one... my syntax highlight
(font-lock) problems disappear.

Your version throws a lot of byte-compile warnings when compiled with
emacs-23 but seems to work just fine.

Thanks for the tip about there being a difference.

But from what I just read on emacs.devel .. the newest version in 23
is synced against your latest.

[Ted Zlatanov]

On Fri, 25 Sep 2009 17:29:45 -0500 Harry Putnam <rea...@newsguy.com> wrote:

HP> It appears my version of cperl-mode is not totally outdated eh?.

On Fri, 4 Sep 2009 00:06:38 +0000 (UTC) Ilya Zakharevich <nospam...@ilyaz.org> wrote:

IZ> On 2009-09-03, Ted Zlatanov <t...@lifelogs.com> wrote:
>>>> That's silly. Can this be resolved with a user option?
>>
IZ> Oups, I meant Lisp coding style, not Perl coding style...
>>
>> Consider js2-mode.el made it into the trunk (there was a long discussion
>> about the nasty hacks it employs), I think you should ask again. I can
>> do it if you prefer.

IZ> I would gladly accept any help in this matter. A lot of thanks in advance.

Ilya, it looks like Chong Yidong has synchronized 6.2 into the Emacs
trunk. Can you follow up on emacs-devel and clear up any remaining
issues? I'd like to make sure the Emacs version of cperl-mode doesn't
get out of sync again, and it's good that Chong (one of the two current
Emacs maintainers) feels strongly enough to synchronize against your
version.

Thanks
Ted

[Ilya Zakharevich]

On 2009-09-26, Harry Putnam <rea...@newsguy.com> wrote:
> Ilya Zakharevich <nospam...@ilyaz.org> writes:
>
>> a) cperl-mode distributed with RMS Emacs is completely borken. See
>> "my" versions;
>
> Ilya, sorry I don't know enough to know the answer to this but can I
> safely ignore the list of warnings I get when byte compiling your most
> recent (6.2) version (using emacs-23 from cvs).

I would not know. The newest version around is about 21. My
workhorse version is v19. ;-)

Yours,
Ilya

[Harry Putnam]

Ilya Zakharevich <nospam...@ilyaz.org> writes:

Where are these version numbers located? On your site I see numbers
like: [...] 6.1, 6.2

Inside I don't see a version number like 19,20,21 etc either.

There it says:
;; $Id: cperl-mode.el,v 6.2 2008/04/14 23:14:52 vera Exp vera $

Is there another place besides:
http://math.berkeley.edu/~ilya/software/emacs/

Where versions with numbers like 19 or 21 are kept?

[Kevin Rodgers]

Harry Putnam wrote:
> Ilya Zakharevich <nospam...@ilyaz.org> writes:
>
>> On 2009-09-26, Harry Putnam <rea...@newsguy.com> wrote:
>>> Ilya Zakharevich <nospam...@ilyaz.org> writes:
>>>
>>>> a) cperl-mode distributed with RMS Emacs is completely borken. See
>>>> "my" versions;
>>> Ilya, sorry I don't know enough to know the answer to this but can I
>>> safely ignore the list of warnings I get when byte compiling your most
>>> recent (6.2) version (using emacs-23 from cvs).
>> I would not know. The newest version around is about 21. My
>> workhorse version is v19. ;-)
>

> Where are these version numbers located? On your site I see numbers
> like: [...] 6.1, 6.2
>
> Inside I don't see a version number like 19,20,21 etc either.
>
> There it says:
> ;; $Id: cperl-mode.el,v 6.2 2008/04/14 23:14:52 vera Exp vera $
>
> Is there another place besides:
> http://math.berkeley.edu/~ilya/software/emacs/
>
> Where versions with numbers like 19 or 21 are kept?

Ilya must be referring to Emacs versions. His workhorse version just turned 13
years old: the last Emacs 19 release (19.34b) was September 6, 1996!

If he's maintaining his cperl-mode for other users, I wonder what percentage
of them are using Emacs 19, ..., 23 (23.1 was released on July 29, 2009).

--
Kevin Rodgers
Denver, Colorado, USA

[Harry Putnam]

Kevin Rodgers <kevin.d...@gmail.com> writes:

> Harry Putnam wrote:
>> Ilya Zakharevich <nospam...@ilyaz.org> writes:
>>
>>> On 2009-09-26, Harry Putnam <rea...@newsguy.com> wrote:
>>>> Ilya Zakharevich <nospam...@ilyaz.org> writes:
>>>>
>>>>> a) cperl-mode distributed with RMS Emacs is completely borken. See
>>>>> "my" versions;
>>>> Ilya, sorry I don't know enough to know the answer to this but can I
>>>> safely ignore the list of warnings I get when byte compiling your most
>>>> recent (6.2) version (using emacs-23 from cvs).
>>> I would not know. The newest version around is about 21. My
>>> workhorse version is v19. ;-)
>>

>> Where are these version numbers located? On your site I see numbers
>> like: [...] 6.1, 6.2
>>
>> Inside I don't see a version number like 19,20,21 etc either.
>>
>> There it says:
>> ;; $Id: cperl-mode.el,v 6.2 2008/04/14 23:14:52 vera Exp vera $
>>
>> Is there another place besides:
>> http://math.berkeley.edu/~ilya/software/emacs/
>>
>> Where versions with numbers like 19 or 21 are kept?
>
> Ilya must be referring to Emacs versions. His workhorse version just turned 13
> years old: the last Emacs 19 release (19.34b) was September 6, 1996!
>
> If he's maintaining his cperl-mode for other users, I wonder what percentage
> of them are using Emacs 19, ..., 23 (23.1 was released on July 29, 2009).

Hehe... it never even occurred to me he might mean emacs versions...

[Ilya Zakharevich]

On 2009-09-25, Harry Putnam <rea...@newsguy.com> wrote:
> Is there something newer or better available?

I already discussed this. If it comes with Emacs, it is broken.

> It appears my version of cperl-mode is not totally outdated eh?.

The question is not about timestamps, but about clue.

Yours,
Ilya

[Ilya Zakharevich]

On 2009-09-28, Ted Zlatanov <t...@lifelogs.com> wrote:
> Ilya, it looks like Chong Yidong has synchronized 6.2 into the Emacs
> trunk.

How much does it differ from what was done a decade ago? That time
a brave guy volunteered and converted cperl-mode to new conventions.
Only years of user experiences had shown that this introduced so many
bugs that the mode is IMO unusable.

(Usability is the question of trust. THE purpose of a mode is to
use faces/paren-match/indentation to make driver errors as much
visible as possible. I know that "my" version has so few false
positives that EVERY irregularity of
highlight/paren-match/indentation deserves to be investigated.
AFAIK, the situation is opposite with the shipped version; the "cry
wolves" effect...)

> Can you follow up on emacs-devel and clear up any remaining issues?

Could not find it on gmane.emacs.devel...

> I'd like to make sure the Emacs version of cperl-mode doesn't
> get out of sync again, and it's good that Chong (one of the two current
> Emacs maintainers) feels strongly enough to synchronize against your
> version.

The only practically viable solution I can see is to aim for identical
byte-code between "my" version and Emacs's one... (At least with some
"define"; the default compile may aim for distinct bytecode, but all
the corresponding places in the .el file should be easily identifyable...)

Thanks,
Ilya

[Harry Putnam]

Ilya Zakharevich <nospam...@ilyaz.org> writes:

ouch

[Ted Zlatanov]

On Fri, 9 Oct 2009 09:04:25 +0000 (UTC) Ilya Zakharevich <nospam...@ilyaz.org> wrote:

IZ> On 2009-09-28, Ted Zlatanov <t...@lifelogs.com> wrote:
>> Ilya, it looks like Chong Yidong has synchronized 6.2 into the Emacs
>> trunk.

IZ> How much does it differ from what was done a decade ago? That time
IZ> a brave guy volunteered and converted cperl-mode to new conventions.
IZ> Only years of user experiences had shown that this introduced so many
IZ> bugs that the mode is IMO unusable.

I don't know, sorry. I am only trying to get you to discuss it on
emacs-devel.

>> Can you follow up on emacs-devel and clear up any remaining issues?

IZ> Could not find it on gmane.emacs.devel...

Here's a reference:

http://thread.gmane.org/gmane.emacs.devel/115635/focus=115643

>> I'd like to make sure the Emacs version of cperl-mode doesn't
>> get out of sync again, and it's good that Chong (one of the two current
>> Emacs maintainers) feels strongly enough to synchronize against your
>> version.

IZ> The only practically viable solution I can see is to aim for identical
IZ> byte-code between "my" version and Emacs's one... (At least with some
IZ> "define"; the default compile may aim for distinct bytecode, but all
IZ> the corresponding places in the .el file should be easily identifyable...)

I don't know if this is practical for everyone, but I hope you can
discuss it on emacs-devel.

Thanks
Ted

[Ilya Zakharevich]

On 2009-10-09, Ted Zlatanov <t...@lifelogs.com> wrote:
> Here's a reference:
>
> http://thread.gmane.org/gmane.emacs.devel/115635/focus=115643

Oups, it does not show when I visit it with slrn...

Hmm, I did not know this:

he is not enthusiastic about accepting patches

Do not see any patches in my mailbox for the last 2 years... Looking
back for 4 years, I see one patch. It was for xemacs, and turned out
to be broken...

Interesting...

Thanks,
Ilya

[Ted Zlatanov]

On Fri, 9 Oct 2009 21:48:53 +0000 (UTC) Ilya Zakharevich <nospam...@ilyaz.org> wrote:

IZ> Hmm, I did not know this:

IZ> he is not enthusiastic about accepting patches

IZ> Do not see any patches in my mailbox for the last 2 years... Looking
IZ> back for 4 years, I see one patch. It was for xemacs, and turned out
IZ> to be broken...

It makes sense to find the reason for the disconnect... Could be a spam
filter or an old e-mail address. Regardless we should discuss the
patches on emacs-devel and what (if any) compromises on both sides are
needed to get your cperl-mode and the Emacs one completely in sync.

Ted