Package-to-package bindings and package-scope variables

146 views
Skip to first unread message

Michel Levieux

unread,
Sep 16, 2019, 8:27:01 AM9/16/19
to golang-nuts
Hi all,

I'm developing a command that is a utility to move packages around while keeping compatibility with softwares that import those packages. Maybe there are commands and utilities that already do approximately the same thing, but I'm mainly doing this to learn to use the go/ast package and everything. Let me give you a small example:

- Let's say I have a package in my GOPATH that's named 'foo'
- In this package there is another package 'bar', but this package bar has grown a whole lot since the beginning of the project, and might as well be a project on its own.
- What I want is extract package 'bar' from 'foo' without having to go through all other projects importing 'bar' to change import paths
- The process for the command is to first move package 'bar' wherever I tell it to move it and replace anything in previously existing package 'foo/bar' with bindings, so any function's body is replaced with a call to the function in the moved package, any type is transformed into an "alias" of the moved package's corresponding type (TypeA = bar.TypeA), as well as any constant or variable.

The command is not finished yet but I'm running into a certain issue I can't find a solution to:

Whenever the command creates a binding for a given exported variable:

var (
    Anything = 5.0
)

becomes

var (
    Anything = bar.Anything
)

there's a risk that bar.Anything was originally changed during the program's execution and according to the context. If that was the case, any change to bar.Anything will not be applied in foo/bar.Anything, which breaks the whole thing.

Does anyone have an idea how to programmatically make it so that any changes applied to bar.Anything is applied to foo/bar.Anything, as if they really did share the same memory? Of course this would need to be totally transparent to the user of the package.

Maybe this is not possible but if you guys know anything, please tell me.

Thanks in advance!


clement auger

unread,
Sep 16, 2019, 9:10:14 AM9/16/19
to golang-nuts
hi,

it seems you need

then select desired package, get its top scope, loop over public names, generate aliases.

but i m unsure it is such a good idea to keep those shallow copies around, at least for the case you have found out.

Michel Levieux

unread,
Sep 16, 2019, 9:57:54 AM9/16/19
to clement auger, golang-nuts
Hi Clément, thank you for your answer,

Your links are pretty interesting, though maybe I was unclear stating my issue because that is not what I was looking for.
I also think such a tool should be used with parsimony, the goal being that over time, people stop using the old package's location and it can be removed.

I also find it important to specify that my program documents the whole new package as being deprecated, also informing the user where to find the new package.

However, what I'm having trouble with is that a modification of a package scope variable in the newly created package will not result in a modification of the aliased variable of the old package.
I'm using the go/ast and go/parser (along with a little of go/token) packages for the parsing, processing and generation of the different aspects of the packages.

I saw some //go:linkname pragmas that seem to do what I want but I don't wanna do things that are not recommended or unsafe, I want to be sure what I'm doing is the right way to go, or at least that it's not possible with the current state of the language.



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/099b1934-f90f-492a-a620-04c51599a70e%40googlegroups.com.

clement auger

unread,
Sep 16, 2019, 5:09:57 PM9/16/19
to golang-nuts
hi again,

This looks odd use case because when i move a package around,
with all the great tooling provided by the contributors,
import paths are updated automatically after a save.
It fails only if the package name is ambiguous because it
exists twice with different package paths within the workspace.
Its a terribly boring issue when that happens,
but i managed to survive it.

With a bit of research into those tools documentation,
a solution should quickly come up
to recursively update a workspace.

About the deprecation, i think it is strongly dependent
with your development practices and workflow.
If possible, I would not deprecate, but update my dependencies,
and pay my technical debts asap.

As for go.loader, it is a high level API of go.ast, go.parser etc
As such it provides lots of help when you are dealing with
go language loading, parsing, interpreting, traversing and manipulation.
It is tremendously useful when, for example,
cross package reference resolutions comes into your way.

Regarding the linkname pragma, i don't know enough about
the process of compilation to tell you anything relevant to its usage.
However, i tried to "relink" a public var of a package
to another package public var, nothing happened.
More reasonably, this extract (https://gist.github.com/myitcv/e4202a9513bed098262444ed0f4f3625)
of pragma usage within the core tree demonstrates
a pretty specific usage to inject public symbols of C funcs.
e.g. //go:linkname byteIndex strings.IndexByte

Maybe someone else with a deeper knowledge can help you figure that out.

I don't have the feeling that i answered your questions, sorry for that.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

Michel Levieux

unread,
Sep 17, 2019, 4:34:36 AM9/17/19
to golang-nuts
Hi again Clément,

It is indeed an uncommon practice, but I can explain my motivations behind this little project:
Where I work, we have a huge helper package with lots of sub-packages. This package existed long before I started working there, and we (the devs) knew it would be a problem at some point, having this giant package and its sub-packages imported in almost all our projects. Lately I decided that it was too much (we've been adding packages and features to it a lot the last 6 months) and that it seriously needed splitting. So since every single person who develops in the company uses this package, there is a strong probability that many projects are not on my laptop and my IDE can't detect them to update all the relevant import paths. But still, I'd like to be able to improve our codebase without breaking everything, hence the command. It is a really specific case, and I know that, but that's the best solution I've come up with so far.

Maybe I should've asked here about that before, rather than my original question ^^'

Anyways, thank you again for the time you give me, don't hesitate if you have other ideas or questions.

As for the use of go.loader, I think I will switch to it when I really understand go.ast and go.parser, the reason being I wanna understand the most of what is going on under the hood before abstracting it :)

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/d9b2521e-a99a-4caf-be47-6770d3760082%40googlegroups.com.

clement auger

unread,
Sep 17, 2019, 10:38:37 AM9/17/19
to golang-nuts

Michel Levieux

unread,
Sep 18, 2019, 12:47:24 PM9/18/19
to golang-nuts, clement auger
Hi all,

@clement auger  I've looked at your links and they're all great, though the three last ones spoiled me a part of what I still have to do in my own project ;)

Appart from that, I've already (quite successfully it seems) developped the part of the program that is supposed to take care of resolving the different import paths.
In fact, the only thing left to do (in terms of features) in my command, before handling go-code generation, is to handle such cases:

package one

import (
    ...
)

var (
    ExportedOne int
    ExportedTwo string
)

If I move package 'one' to, let say, package 'two', in the context of my project, accessing 'one.ExportedOne' for instance should still work. As I said before, I can't handle cases where ExportedOne or ExportedTwo are mutated during execution, because then, since 'one' and 'two' are as follow:

package one

import (
    ...
)

var (
    ExportedOne = two.ExportedOne
    ExportedTwo = two.ExportedTwo
)

and

package two

import (
    ...
)

var (
    ExportedOne int
    ExportedTwo string
)

any write access to two.ExportedOne will not occur in one.ExportedOne. So "aliasing" variables does the work when those variables are not mutated (but then, they're kind of "false constants"). When they are, the solution is not clear.
The simplest thing I could decide is to just say "Mutating an exported package-scope variable is an edge-case sufficiently considered a bad practice for my command not to handle it". Hence the tool would just inform the user that such or such package cannot be moved because it does not know how to do it and report what variables (exactly) are causing this inability. But this is not ideal, I would like to find a solution to that, even if it is some kind of hack or workaround, as long as it is safe.

I'm sorry I can't share pieces of code from the project for legal reasons, if there is something (that is not code) that can help you understand what I'm trying to accomplish, please let me know and I'll do my best.

PS: the particular issue that is progressively leading me to just say "I can't handle that" is https://github.com/golang/go/issues/18130, however it is quite old now and I'm wondering if anything exists now in the language

Thanks,

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/b6775d1e-d5c4-44dd-9ba7-dba4bf0178d7%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages