Visual Studio 2019 Edit-and-continue

[Rick C. Hodgin]

[If anyone knows a better place to post this question, please let me
know and I'll take it there.]

In Visual Studio 2019, when you attach to process, suspend execution,
edit code and apply changes using the edit-and-continue feature, it
does not let you detach from the process after those changes have been
made.

Does anyone know why that is?

I can possibly see the need to warn the user that the binary file
on disk will not have been updated, but why can't the altered binary
image running in memory be maintained as-is for the duration of its
life?

Seems an arbitrary limitation.

--
Rick C. Hodgin

[Mr Flibble]

And Satan invented fossils, yes?

/Flibble

--
"Snakes didn't evolve, instead talking snakes with legs changed into
snakes." - Rick C. Hodgin

“You won’t burn in hell. But be nice anyway.” – Ricky Gervais

“I see Atheists are fighting and killing each other again, over who
doesn’t believe in any God the most. Oh, no..wait.. that never happens.” –
Ricky Gervais

"Suppose it's all true, and you walk up to the pearly gates, and are
confronted by God," Bryne asked on his show The Meaning of Life. "What
will Stephen Fry say to him, her, or it?"
"I'd say, bone cancer in children? What's that about?" Fry replied.
"How dare you? How dare you create a world to which there is such misery
that is not our fault. It's not right, it's utterly, utterly evil."
"Why should I respect a capricious, mean-minded, stupid God who creates a
world that is so full of injustice and pain. That's what I would say."

[Öö Tiib]

On Tuesday, 17 September 2019 23:02:03 UTC+3, Rick C. Hodgin wrote:
> [If anyone knows a better place to post this question, please let me
> know and I'll take it there.]

I don't think you will get good (or even much different) answer to that
question from anywhere. It is generally always same.
Q: Why a software product does not allow certain scenario of usage?
A: Because support for that scenario hasn't been implemented.
Q: Why it hasn't been implemented?
A: Because implementers put their effort into other features.
Q: Why?
A: Because the stakeholders prioritized other features as being
of higher importance.
Q: Why?
A: Because they wanted to.

[David Brown]

(Disclaimer - I don't use MSVC, don't use edit-and-continue, and don't
do C or C++ coding on Windows. That may make me ignorant.)

Let's give the developers the benefit of the doubt here - sometimes
features are not implemented because doing so would be impossible, or at
least ridiculously hard. And sometimes it is because there are few
people who want the feature or care about it - it might seem a
marvellous idea to one person but it is not going to be implemented
unless there are many people who would find it useful. And occasionally
it is because they think the feature is a misfeature, directly
counter-productive, or will limit other features or future efforts.


As far as I understand the way Windows works (and I could be wrong
here), this is likely to be /very/ difficult to implement. I don't see
it as being a very important feature - if you are debugging a running
program and make changes to its code, leaving your debugger attached
seems reasonable. (Perhaps MSVS only lets you use one debugger at a
time, in which case it would be more of a hardship.) And it is easy to
see a world of abuse possibilities if there were tools to let you attach
to a running program, change the code, and disappear without a trace.

[Rick C. Hodgin]

If that's all it is I understand. If there's something I don't know
about why it needs to remain attached, that's the thing I'd like to
learn. From writing my own OS, debugger, and tools, I don't see a
reason other than just a decision to keep the in-memory-app aligned
with the on-disk-app, so they aren't out of sync. Seems to be a
policy decision rather than a technical one. And that's fine. I'd
just like to learn if it's otherwise (if it's otherwise).

--
Rick C. Hodgin

[Öö Tiib]

It is basically same what my Q/A tried to say. I did not imply that
developers or stakeholders are bad somehow. It makes sense to
prioritize expensive and rarely used features low since the bang
per buck is low.

> As far as I understand the way Windows works (and I could be wrong
> here), this is likely to be /very/ difficult to implement. I don't see
> it as being a very important feature - if you are debugging a running
> program and make changes to its code, leaving your debugger attached
> seems reasonable. (Perhaps MSVS only lets you use one debugger at a
> time, in which case it would be more of a hardship.) And it is easy to
> see a world of abuse possibilities if there were tools to let you attach
> to a running program, change the code, and disappear without a trace.

I also suspect that edit-and-continue results with quite a hack (not
normal linked binary) that uses features of debugger for to stay
alive and so may be rather tricky to detach from.

[Öö Tiib]

The whole edit and continue is quite rare feature in compiled
languages. Is it because of policy decisions or technical issues?
I see it as technically challenging feature.

[Rick C. Hodgin]

On 9/18/2019 10:54 AM, Öö Tiib wrote:
> I also suspect that edit-and-continue results with quite a hack (not
> normal linked binary) that uses features of debugger for to stay
> alive and so may be rather tricky to detach from.

It's not a quick hack. It is not a traditional binary. If you step
through the edit-and-continue code, you'll see function calls going
to a branching off place, and then going to their destination. This
allows functions to be swapped out as they are changed.

It's a very logical process. It uses some fixed formats like that,
such as an array of effective vtable-like addresses, to know where
to go for each function. Memory addresses for constants can be
changed. Local variable counts and locations on the stack can be
changed if it's not too radical. And code already on the call stack
will still navigate back through the stale versions, only new calls
will go to the new ones.

Edit-and-continue is, by far, the greatest developer asset I have.
In languages I code where I don't have edit-and-continue, it is
sorely missed.

--
Rick C. Hodgin

[Rick C. Hodgin]

On 9/18/2019 11:24 AM, Öö Tiib wrote:
> The whole edit and continue is quite rare feature in compiled
> languages.

Apple added it to GCC back in the mid-2000s. They called it fix-
and-continue, and it never went upstream, but they used it intern-
ally for a while, then stopped.

--
Rick C. Hodgin

[Rick C. Hodgin]

An old reference:

http://www.sourceware.org/ml/gdb/2003-06/msg00500.html

--
Rick C. Hodgin

[Ian Collins]

Indeed. I can also see a it as a good way to make malicious changes to
a binary, so having to leave the debugger attached could be seen as a
security feature.

--
Ian.

[Rick C. Hodgin]

On 9/18/2019 4:17 PM, Ian Collins wrote:
> Indeed.  I can also see a it as a good way to make malicious changes to a
> binary, so having to leave the debugger attached could be seen as a security
> feature.

It makes sense.

It would be difficult to make malicious changes. You'd have to be
hands-on on the machine. Even with remote debugging, you'd have to
be running that service and know the port. In addition, you'd have
to have the source code, or have intimate knowledge of the source
code. And if you have those special abilities, you'd probably be
attach with your own custom debugger, modify as needed, and then
detach.

I wonder how I could search for this information from Microsoft?
Hmmm...

--
Rick C. Hodgin

[Mr Flibble]

[Öö Tiib]

On Wednesday, 18 September 2019 18:30:13 UTC+3, Rick C. Hodgin wrote:
> On 9/18/2019 10:54 AM, Öö Tiib wrote:
> > I also suspect that edit-and-continue results with quite a hack (not
> > normal linked binary) that uses features of debugger for to stay
> > alive and so may be rather tricky to detach from.
>
> It's not a quick hack. It is not a traditional binary. If you step
> through the edit-and-continue code, you'll see function calls going
> to a branching off place, and then going to their destination. This
> allows functions to be swapped out as they are changed.

I meant that the whole "branching off" of changed functions is likely
part of debuggers assets, the original process did not have it.

> It's a very logical process. It uses some fixed formats like that,
> such as an array of effective vtable-like addresses, to know where
> to go for each function. Memory addresses for constants can be
> changed. Local variable counts and locations on the stack can be
> changed if it's not too radical. And code already on the call stack
> will still navigate back through the stale versions, only new calls
> will go to the new ones.

So what does detaching that debugger even mean? The "stale versions"
of functions are still present as part of original executable and
the "vtable-like" hooking mechanism with "branching off places" are
new goods owned by debugger. Avoiding detaching it does not sound
like entirely policy-based decision.

> Edit-and-continue is, by far, the greatest developer asset I have.
> In languages I code where I don't have edit-and-continue, it is
> sorely missed.

But that is totally orthogonal to need of detaching from such
processes.

[Rick C. Hodgin]

On 9/19/2019 2:50 AM, Öö Tiib wrote:
> So what does detaching that debugger even mean?

In Windows, a process launches and has its own allocation, threads,
open handles, etc. It runs in isolation and is protected against
corruption from other processes that may later crash or try to ac-
cess its data by hardware and OS protection mechanisms.

A debugger is like a hand that reaches out to grab a tennis ball.
The hand wraps around the ball and holds onto it. When that hap-
pens, the debugger "attaches" to the process being debugged. It
shares information about it as a debugger debugging another proc-
ess. It has access to the same memory, access to handles, etc.
The debugger typically recognizes compile-time information from
something like a PDB (program database) created at compile + link
time. That PDB allows a correlation between binary location ac-
cess in memory (even through the ABI loader's randomizer) and the
source code which relates to that code.

By making changes to the source code, re-compiling, and applying
the code changes to the running ABI, many common things changed
in source code (adding code, altering code, deleting code, doing
the same for variables, etc.), the ABI can be expanded or con-
tracted as needed by the debugger changes.

When the debugger is done making changes to the running ABI foot-
print in memory, it can detach. This is the same as taking your
hand off that tennis ball and letting it go. The process that
had been being debugged is then released, left again to its own
encapsulated existence.

> The "stale versions"
> of functions are still present as part of original executable and
> the "vtable-like" hooking mechanism with "branching off places" are
> new goods owned by debugger. Avoiding detaching it does not sound
> like entirely policy-based decision.

They are committed into the running process's memory. They're
just not on the disk file, though honestly I cannot see why the
OS + compiler + debugger couldn't coordinate an exchange which
allows the .exe on disk to be updated with the new changes.

I know in VS when you exit a process you've been working on, it
does an auto-link after terminating the process so it can update
the disk image. Seems an unnecessary step IMHO.

>> Edit-and-continue is, by far, the greatest developer asset I have.
>> In languages I code where I don't have edit-and-continue, it is
>> sorely missed.
>
> But that is totally orthogonal to need of detaching from such
> processes.

The advantage of edit-and-continue is the saved contextual state.
You can attach to a running process, suspend its execution where
it is, examine things, change things, inject new things, etc.,
and then resume from where you were. It saves the time of loading
the app, getting back to the place where you were, selecting the
data options you were working with, to get to the point where the
algorithm you're debugging exists.

Using an edit-and-continue ABI reduces code performance by about
20% or so. But the time it saves the developer more than makes
up for that. In addition, most CPUs today on most applications
today running at 80% of their full potential speed is more than
fast enough. Running at 20% of their full potential speed is
probably usable.

I have an old 2007 HP Pavillion desktop computer. It was getting
slow on Windows 7 with a dual-core 2.2 GHz CPU and SATA hard disk.
I upgraded to a quad-core 2.4 GHz CPU, 8 GB of RAM, and an SSD,
and it's like a new machine. I couldn't believe the difference.
That $110 investment resurrected (scavenged DRAM from another
machine, bought CPU on eBay, new SSD) the aging computer and I've
been using it for even Java development in Eclipse without any
performance issues.

In any event, most people I know do not appreciate edit-and-con-
tinue. I find it difficult to code without it. I am 10x or more
productive when I have an edit-and-continue toolset because it
allows me to code as I go, testing bits as I go, which greatly
helps me overcome the issues I code into my code by my dyslexia.

--
Rick C. Hodgin