Re: [gentoo-user] fetch restriction bypass

[Michael Mol]

On Mon, Apr 30, 2012 at 2:20 PM, james <wire...@tampabay.rr.com> wrote:
> Hello,
>
> OK so I have java that I must use, but it is
> "fetch restricted" becasue of Oracle being
> an a_hole.
>
> However, I do not have time to manually bypass the fetch restrction
> every time the file needs to be updated, as I manage
> too many different gentoo systems.
>
>  FU  ] dev-java/sun-jdk-1.6.0.31 [1.6.0.29]
>
> I need to stay with the sun-jdk so an automated way
> to fix this once is required.
> The license fix (make.conf) does not do the trick:
> ACCEPT_LICENSE="*"
>
> No, I do not want to switch to icetea....
> ideas?

Use a network-mounted distfiles directory on a common file server?
That way, once you've downloaded it once, for any system, the package
is right there for the rest.


--
:wq

[james]

Hello,

OK so I have java that I must use, but it is
"fetch restricted" becasue of Oracle being
an a_hole.

However, I do not have time to manually bypass the fetch restrction
every time the file needs to be updated, as I manage
too many different gentoo systems.

FU ] dev-java/sun-jdk-1.6.0.31 [1.6.0.29]

I need to stay with the sun-jdk so an automated way
to fix this once is required.
The license fix (make.conf) does not do the trick:
ACCEPT_LICENSE="*"

No, I do not want to switch to icetea....
ideas?

James

[Michael Orlitzky]

On 04/30/12 14:20, james wrote:
> Hello,
>
> OK so I have java that I must use, but it is
> "fetch restricted" becasue of Oracle being
> an a_hole.
>
> However, I do not have time to manually bypass the fetch restrction
> every time the file needs to be updated, as I manage
> too many different gentoo systems.

As far as I know, for legal reasons, Gentoo doesn't provide an automated
way to violate the upstream license (no matter how asinine).

You'll have to script something.

[James]

Michael Mol <mikemol <at> gmail.com> writes:


> Use a network-mounted distfiles directory on a common file server?
> That way, once you've downloaded it once, for any system, the package
> is right there for the rest.

Well I do not use NFS or such, but, I do scp the restricted files around.
My environment is such that it is partitions and systems moved around
too frequently (used remotely) to use a dist file system.

So, I'd like to bypass the fetch restrictions all together...
one and for all; any other ideas?


James

[Michael Mol]

Does the ebuild for portage support user-supplied patches?

--
:wq

[Michael Mol]

Patch Portage? Having a local patch like that would depend on whether
or not the Portage ebuild supported particular hooks, but I don't
remember the specifics.

If you're using scp, you might consider using sshfs for your
distfiles. Or, heck, dropbox.


--
:wq

[James]

Michael Orlitzky <michael <at> orlitzky.com> writes:


> You'll have to script something.

OK? Any examples or pseudo code
that outlines how to do this?

Surely, it's been done before?

maybe something in CPAN?

James

[Michael Orlitzky]

On 04/30/12 14:50, Michael Mol wrote:
> On Mon, Apr 30, 2012 at 2:42 PM, James <wire...@tampabay.rr.com> wrote:
>> Michael Mol <mikemol <at> gmail.com> writes:
>>
>>
>>> Use a network-mounted distfiles directory on a common file server?
>>> That way, once you've downloaded it once, for any system, the package
>>> is right there for the rest.
>>
>>
>> Well I do not use NFS or such, but, I do scp the restricted files around.
>> My environment is such that it is partitions and systems moved around
>> too frequently (used remotely) to use a dist file system.
>>
>> So, I'd like to bypass the fetch restrictions all together...
>> one and for all; any other ideas?
>
> Patch Portage? Having a local patch like that would depend on whether
> or not the Portage ebuild supported particular hooks, but I don't
> remember the specifics.

Won't help because the tarball location isn't in the ebuild. You have to
go to the webpage to find it.

You can patch the ebuild every time, but that takes the same amount of
work (on each machine) as wget <foo>.

[Michael Orlitzky]

On 04/30/12 14:44, Michael Mol wrote:
>
> Does the ebuild for portage support user-supplied patches?
>

It doesn't look like it, but you can always hack it with,

post_src_unpack() {
cd "${S}"
epatch_user
}

in your ~/.bashrc.

[Michael Mol]

I was thinking 'skip the fetch restriction check', but if the ebuild
doesn't have the file path to retrieve, that's almost moot. It's
_plausible_ one could calculate the path from the version of the
package being emerged, though, so I suppose it's automateable.

--
:wq

[Michael Orlitzky]

You said you're already using scp to move things around; I think that's
as good as it's going to get if you don't want to share distfiles.

It's not as easy as just bypassing the fetch restriction. Neither the
ebuild nor portage know where the upstream tarball is; the only thing in
the ebuild is a link to the webpage.

If you can settle on one machine to offer up its own distfiles folder,
you might be able to overlay that onto each machine with UnionFS.
Multiple DISTDIRs would also work but don't seem to exist. There was a
patch way back in 2003:

> http://archives.gentoo.org/gentoo-dev/msg_4c28fe3b3ff086d022734f20c3aca9a0.xml

[Neil Bothwick]

On Mon, 30 Apr 2012 15:15:49 -0400, Michael Mol wrote:

> I was thinking 'skip the fetch restriction check', but if the ebuild
> doesn't have the file path to retrieve, that's almost moot. It's
> _plausible_ one could calculate the path from the version of the
> package being emerged, though, so I suppose it's automateable.

Assuming there even is a path on a publicly accessible ftp or http server
and not a file in a location that can only be accessed by PHP or whatever
code running on the server that runs after you sign over your soul.

I'm not sure what the big deal is, so portasge skips emerging one package
because it can't download the distfile. So what? The previous version
worked OK the day before and won't suddenly break because an update is
available. Just download and upgrade when you have the time, casting the
appropriate curses for those that set the licence.


--
Neil Bothwick

Who messed with my anti-paranoia shot?

[Mark Knecht]

I agree. To me it's not much of an issue. However sometime ago when
there was a conversation about how people update their machines I
mentioned that I always do an

emerge -fDuN @world

prior to kicking off the real emerge just to ensure that when the
build finally does start all the files are here and ready. This sort
of issue is precisely why I do that.

I know most people don't like calculating all the dependencies
multiple times but I prefer to do so, get the files and then pretty
much be guaranteed that the build proceeds without much attention from
me.

Cheers,
Mark

[Neil Bothwick]

On Mon, 30 Apr 2012 15:50:18 -0700, Mark Knecht wrote:

> > I'm not sure what the big deal is, so portasge skips emerging one
> > package because it can't download the distfile. So what? The previous
> > version worked OK the day before and won't suddenly break because an
> > update is available. Just download and upgrade when you have the
> > time, casting the appropriate curses for those that set the licence.
> >
>
> I agree. To me it's not much of an issue. However sometime ago when
> there was a conversation about how people update their machines I
> mentioned that I always do an
>
> emerge -fDuN @world

I do something similar, from a cron script that runs emerge --sync and a
couple of other bits.

> prior to kicking off the real emerge just to ensure that when the
> build finally does start all the files are here and ready. This sort
> of issue is precisely why I do that.

This still isn't an issue, as long as you use --keep-going. The emerge
world will proceed to update everything but the one affected package.

> I know most people don't like calculating all the dependencies
> multiple times but I prefer to do so, get the files and then pretty
> much be guaranteed that the build proceeds without much attention from
> me.

It is a useful method of detecting potential problems, but this isn't
really a problem. However, as my script emails me the output of emerge -p
world (which means I actually calculate the dependencies three times, but
I don't care as I'm asleep for the first two) I know about the fetch
restriction as soon as I read my mail, and can decide whether to deal
with it immediately or ignore it.


--
Neil Bothwick

Don't be humble, you're not that great.

[Michael Orlitzky]

On 04/30/2012 02:45 PM, James wrote:
> Michael Orlitzky <michael <at> orlitzky.com> writes:
>
>
>> You'll have to script something.
>

I gave this a serious shot, but it's not easy.

First, you can override the ebuild environment:

$ cat /etc/portage/bashrc
if [ "${EBUILD_PHASE}" == "clean" ] && [ "${PN}" == "sun-jdk" ]; then
...

You can parse out the important stuff from the ebuild. This sets JDK_URI
to the value contained in the ebuild:

eval `"${GREP}" JDK_URI= "${EBUILD}"`

And you can even parse the URL out of the HTML file pretty easily with a
regular expression. But, unfortunately, they're checking for cookies:

$ wget http://download.oracle.com/otn-pub/java/jdk/6u31-b04/jdk-
6u31-linux-x64.bin
...
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: http://download.oracle.com/errors/download-fail-1505220.html

And, the cookies don't get set in a normal HTTP request. So you can't
just `curl $JDK_URI` and save the cookies.

It looks like the URL that sets the cookies is created by that
javascript lightbox code, so you need to be able to evaluate JS, get
that URL, hit the page, and save its cookies before you're allowed to
download the file.

Finally, the cookies are dynamic, and not something like let_me_in=True.
So maybe it's still possible, but scp is looking a lot better right now.

[Michael Orlitzky]

On 04/30/2012 09:40 PM, Michael Orlitzky wrote:
>
> And, the cookies don't get set in a normal HTTP request.

For this to make sense, you probably want to read, "HTML request."

[James]

Michael Orlitzky <michael <at> orlitzky.com> writes:

Ok, I've got to think about all of this feedback
and figure out what to do. I'm leaning towards
manual download, once, and then an automation
script that runs each time I do an update. That
script would check for Fetch restricted packages on each
machine locally, as there cannot be too many that I use,
and then download the latest version via scp from a
(suggested) previous download.


Maybe this is a chance to play with port-knocking before
allowing the local file transfer.... I gotta
think about how I want to do this. My network is
"hard partition" internally, as portions move to different
locations and must be "stand alone" no matter how the
partitions are split. For now, the partitions do not
morph (change in component count).

** note** a partition does not reference a hard drive
scheme but the fact that security and feature enhancements
are achieved by physical and/or software isolation.

Each partition should be fully functional and survivable
from frequent physical separation. Each partition can contain
one or more computational/storage resources and are not
similar in component count.


thanks for all the responses,
James