Official FreeBSD Binary Packages now available for pkgng

[Bryan Drewery]

We are pleased to announce that official binary packages are now
available for pkg, the next generation package management tool for FreeBSD.

Pkg allows you to either use ports with portmaster/portupgrade or to
have binary remote packages without ports.

We have binary packages available for i386 and amd64 on
8.3,8.4,9.1,9.2,10.0 and 11 (head).

Pkg will be the default starting in FreeBSD 10.

The pkg_install suite of tools pkg_create(1), pkg_add(1), and
pkg_info(1) (which ports also use), are deprecated and will be
discontinued in roughly 6 months. A communication regarding the
deprecation of the pkg_install suite of tools will be sent separately in
the future.

If you are currently not using pkg and wish to, run the following as
root. Be sure not to add WITH_PKGNG=yes to your make.conf until after
pkg is installed.

# cd /usr/ports/ports-mgmt/pkg && make install clean
# echo WITH_PKGNG=yes >> /etc/make.conf
# pkg2ng

You can now either continue to use ports with portmaster/portupgrade, as
before or switch to using binary packages only.


To use binary packages:

1. Ensure your pkg(8) is up-to-date. 'pkg -v' should say at least
1.1.4_8. If it does not, first upgrade from ports.
2. Remove any repository-specific configuration from
/usr/local/etc/pkg.conf, such as PACKAGESITE, MIRROR_TYPE, PUBKEY.
If this leaves your pkg.conf empty, just remove it.
3. mkdir -p /usr/local/etc/pkg/repos
4. Create the file /usr/local/etc/pkg/repos/FreeBSD.conf with:
FreeBSD: {
url: "http://pkg.FreeBSD.org/${ABI}/latest",
mirror_type: "srv",
enabled: "yes"
}

* Note that pkg.FreeBSD.org does not have a browsable web page on it and
does not have a DNS A record. This is intended as it is an SRV host.
pkg(8) knows how to properly use it. You can use 'pkg search' to browse
the available packages in the repository.

Mirrors you may use instead of the global pkg.FreeBSD.org:

pkg.eu.FreeBSD.org
pkg.us-east.FreeBSD.org
pkg.us-west.FreeBSD.org

Your system is now ready to use packages!

Refer to the handbook section on pkgng for usage at
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/pkgng-intro.html.
Also see 'man pkg' for examples or 'pkg help'.


Packages are built weekly from a snapshot of the Ports Collection every
Wednesday morning 01:00 UTC. They typically will be available in the
repository after a few days.

Pkg 1.2 will be released in the coming month which will bring many
improvements including officially signed packages. FreeBSD 10's pkg
bootstrap now also supports signed pkg(8) installation.


Regards,
Bryan Drewery
on behalf of portmgr@


_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"

[Eric Camachat]

It doesn't work with our (microsoft) proxy server, see below.

root@basay:/usr/local/etc/pkg/repos # pkg update -f
Updating repository catalogue
pkg: http://pkg.FreeBSD.org/freebsd:10:x86:64/latest/digests.txz: Service
Unavailable
pkg: No digest falling back on legacy catalog format
pkg: http://pkg.FreeBSD.org/freebsd:10:x86:64/latest/repo.txz: Service
Unavailable
root@basay:/usr/local/etc/pkg/repos #

Eric

[Bryan Drewery]

On 10/31/2013 4:06 PM, Adrian Chadd wrote:
> ... I still think the SRV record stuff is a bad idea.
>
> Well, I think it's a great idea - because I plan on supporting it in
> the next HTTP thing I write - but not having an A record is going to
> continue to bite things.

I don't like it either, it's not up to portmgr.

>
> Also, http+pkg:// isn't a defined protocol either and some strict
> proxies may actually reject it. You should go back to the http://
> protocol.

It's not real. It's a client-side thing only. The "pkg+" is stripped
away before the fetch. It's only intended to make people realize they
can't just drop it into firefox and hit enter.

1.2 adds support for it which is coming soon, but it doesn't really
change much.

>
>
> -adrian
>
> (with his HTTP hat on..)
>
>
> On 31 October 2013 14:04, Allan Jude <fre...@allanjude.com> wrote:

> I am guessing the proxy passes the full HTTP request, without doing the
> SRV lookup, and then can't find the A record.
>
> I wonder if the http+pkg:// protocol can solve this, likely will require
> a patch to fetch to implement the logic to do the dns lookup and make
> the proxies request for the real hostname

>
>>
>> _______________________________________________
>> freebsd...@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
> _______________________________________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
>

--
Regards,
Bryan Drewery

[Eric Camachat]

Same result, neither pkg+http:// nor http+pkg:// worked with proxy server.

Eric

[Bryan Drewery]

On 10/31/2013 4:25 PM, Eric Camachat wrote:
> Same result, neither pkg+http:// nor http+pkg:// worked with proxy server.
>

Top-posting kills babies

pkg+http is NOT supported in 1.1 and as I said, changes nothing.

--
Regards,
Bryan Drewery

[Bryan Drewery]

On 10/31/2013 3:47 PM, Eric Camachat wrote:
> It doesn't work with our (microsoft) proxy server, see below.
>
> root@basay:/usr/local/etc/pkg/repos # pkg update -f
> Updating repository catalogue
> pkg: http://pkg.FreeBSD.org/freebsd:10:x86:64/latest/digests.txz: Service
> Unavailable
> pkg: No digest falling back on legacy catalog format
> pkg: http://pkg.FreeBSD.org/freebsd:10:x86:64/latest/repo.txz: Service
> Unavailable
> root@basay:/usr/local/etc/pkg/repos #

I somewhat doubt this is a DNS or SRV issue. The pkg(8) client will do
the DNS lookup and then contact the real server directly. It's more
likely your proxy is just blocking the requests. Perhaps ask your
administrator to add an exception for *.freebsd.org:80 :)

--
Regards,
Bryan Drewery

[Eric Camachat]

browsing www.freebsd.org worked fine.
tried pkg.freebsd.org it got below.
Our DNS server can resolve proxy server only.
Only proxy server can resolve internet sites, this is how our company force
all traffic went through proxy server.

Eric

Network Error (dns_server_failure)

Your request could not be processed because an error occurred contacting
the DNS server. The DNS server may be temporarily unavailable, or there
could be a network problem.
If problem persists, please open a ticket with Motorola help desk; and copy
and paste this page in ticket.

Date/Time: 2013-10-31 22:11:37 Request: GET http://pkg.freebsd.org/ Error:
(dns_server_failure) Proxy Name: proxy
Proxy IP: xxx.xxx.xxx.xxx Client IP: zzz.zzz.zzz.zzz
Referer URL:

[Freddie Cash]

On Thu, Oct 31, 2013 at 3:15 PM, Eric Camachat <eric.c...@gmail.com>wrote:

> browsing www.freebsd.org worked fine.
> tried pkg.freebsd.org it got below.
> Our DNS server can resolve proxy server only.
> Only proxy server can resolve internet sites, this is how our company force
> all traffic went through proxy server.
>
> Eric
>
> Network Error (dns_server_failure)
>
> Your request could not be processed because an error occurred contacting
> the DNS server. The DNS server may be temporarily unavailable, or there
> could be a network problem.
> If problem persists, please open a ticket with Motorola help desk; and copy
> and paste this page in ticket.
>
> Date/Time: 2013-10-31 22:11:37 Request: GET http://pkg.freebsd.org/ Error:
> (dns_server_failure) Proxy Name: proxy
> Proxy IP: xxx.xxx.xxx.xxx Client IP: zzz.zzz.zzz.zzz
> Referer URL:
>

​So, then manually specific a specific pkg mirror and by-pass the DNS SRV
record resolution step.​


--
Freddie Cash
fjw...@gmail.com

[Eric van Gyzen]

On 10/31/2013 05:21 PM, Freddie Cash wrote:
>> >tried pkg.freebsd.org it got below.
>> >Our DNS server can resolve proxy server only.
>> >Only proxy server can resolve internet sites, this is how our company force
>> >all traffic went through proxy server.
>> >
>> >Eric
>> >
>> >Network Error (dns_server_failure)
>> >
>> > Your request could not be processed because an error occurred contacting
>> >the DNS server. The DNS server may be temporarily unavailable, or there
>> >could be a network problem.
>> >If problem persists, please open a ticket with Motorola help desk; and copy
>> >and paste this page in ticket.
>> >

>> >Date/Time: 2013-10-31 22:11:37 Request: GEThttp://pkg.freebsd.org/ Error:

>> >(dns_server_failure) Proxy Name: proxy
>> >Proxy IP: xxx.xxx.xxx.xxx Client IP: zzz.zzz.zzz.zzz
>> >Referer URL:
>> >
> ​So, then manually specific a specific pkg mirror and by-pass the DNS SRV
> record resolution step.​

This kind of proxy configuration is not uncommon. It would be awesome
if this would Just Work. It would remove an impediment to adoption,
which is especially important in the kind of environments that have this
kind of proxy configuration.

Simply adding the mirrors' A (and AAAA) records to pkg.freebsd.org might
suffice.

Alternatively, running an HTTP-redirection service on a host named
pkg.freebsd.org would offer as much flexibility as the SRV records, if
not more. However, it would require maintenance of yet another central
service.

Eric

[Matthias Andree]

Matthew Seaman <mat...@FreeBSD.org> schrieb:

>On 02/11/2013 01:55, Eric van Gyzen wrote:
>> This kind of proxy configuration is not uncommon. It would be
>awesome
>> if this would Just Work. It would remove an impediment to adoption,
>> which is especially important in the kind of environments that have
>this
>> kind of proxy configuration.
>>
>> Simply adding the mirrors' A (and AAAA) records to pkg.freebsd.org
>might
>> suffice.
>

>You seem hung up on the idea that pkg.freebsd.org should resolve to a
>list of IP addresses. It doesn't and for very good reasons.
>Admittedly, using eg. 'http://' as the URL scheme for PACKAGESITE URLs
>was an error -- it contravenes RFC 2616 -- which is why we will be
>switching to a new 'pkg+http://' (or 'pkg+https://', 'pkg+ftp://',
>etc.)
>set of URL schemes with pkg-1.2.x
>
>There certainly are all of the necessary A and AAAA records in the DNS
>for the real servers that host the repositories.
>
>If I understand what you're complaining about is that you see behavious
>like the following:
>
> * You download package foo-1.2.3.txz from pkg.freebsd.org
>
> * Internally, that gets resolved to an HTTP request to eg.
> pkg0.isc.freebsd.org
>
> * Your web proxy caches this package
>
> * On another host, you also want to download foo-1.2.3.txz
>
> * This time the SRV record gets resolved to a different mirror,
> say pkg1.nyi.freebsd.org
>
> * Your proxy has no way of knowing that foo-1.2.3.txz from pkg1.nyi
> is exactly the same file as foo-1.2.3.txz from pkg0.isc so it
> downloads the whole package all over again.
>
>Yes, this is certainly undesirable behaviour. I need to run some tests
>to determine if this is actually what does happen in practice. If so,
>I've an idea about how this problem might be addressed, but it will
>require some changes to the repository configuration.
>
>In the mean time, I suggest just choosing which ever of the
>pkg.freebsd.org repositories is closest to you and using it directly --
>eg.
>
>cat <<EOF > /usr/local/etc/pkg/repos/myrepo.conf
>pkg0.isc {
> url: http://pkg0.isc.freebsd.org/${ABI}/latest
> enabled: yes
> mirror_type: none
>}
>EOF
>
>Obviously, substitute which ever one of
>
> pkg0.isc.freebsd.org (US West)
> pkg1.nyi.freebsd.org (US East)
> pkg0.bme.freebsd.org (Europe)
>
>is appropriate. And be prepared to deal with that specific mirror
>being
>down or replaced by some other server.

>
>> Alternatively, running an HTTP-redirection service on a host named
>> pkg.freebsd.org would offer as much flexibility as the SRV records,
>if
>> not more. However, it would require maintenance of yet another
>central
>> service.
>

>This is already supported in pkg when using the HTTP mirror type. This
>would entail significantly more administrative effort and hardware
>requirement to maintain and keep consistent in the specific case of
>pkg.freebsd.org which is exactly why the SRV mirror type was selected.
>
> Cheers,
>
> Matthew
>
>
>--
>Dr Matthew J Seaman MA, D.Phil.
>PGP: http://www.infracaninophile.co.uk/pgpkey

I understand from Eric's pist that the issue is that through his limiting proxies, the SRV are not available at all so he does not even get to the point where he could get the pkgN.nyi.freebsd.org name back.

[Eric van Gyzen]

On 11/02/2013 07:28 AM, Matthew Seaman wrote:
> On 02/11/2013 11:37, Kurt Jaeger wrote:
>> Hi!

>>
>>> On 02/11/2013 10:15, Matthias Andree wrote:
>>>> I understand from Eric's pist that the issue is that through his
>>>> limiting proxies, the SRV are not available at all so he does not even
>>>> get to the point where he could get the pkgN.nyi.freebsd.org

>>>> <http://pkgN.nyi.freebsd.org> name back.
>>>
>>> That doesn't make sense. All the DNS SRV lookups on pkg.freebsd.org are
>>> done internally to pkg(8),
>>
>> ... which only works, if the DNS server queried answers SRV queries
>> with SRV values.
>>
>> Which is not always true, especially in heavily firewalled environments.
>
> I feel no obligation to do anything to encourage people that
> deliberately break the DNS. They've made their bed, and now they have
> to lie in it.

Eric Camachat didn't break the DNS: his network administrator did.
Matthew, you're right: that doesn't make sense. But people do it,
often for security, either real or perceived. In this kind of
environment, many other things are typically equally broken. I imagine
Eric needs all the encouragement he can get.

Yes, he can reconfigure pkg to use a specific mirror. I only suggest
that it could be made to work without that manual step (and the research
necessary to determine that step).

Lest anyone think I'm complaining: I am very impressed with pkg, and I
appreciate all the technical and non-technical effort that Bryan,
Baptiste, and many others spent on making it real. Instead of a
complaint, consider this a feature request. That is, after all, the
expected response to a feature announcement. :)

Eric (van Gyzen)

[Kurt Jaeger]

Hi!

> On 02/11/2013 10:15, Matthias Andree wrote:

> > I understand from Eric's pist that the issue is that through his
> > limiting proxies, the SRV are not available at all so he does not even
> > get to the point where he could get the pkgN.nyi.freebsd.org

> > <http://pkgN.nyi.freebsd.org> name back.
>
> That doesn't make sense. All the DNS SRV lookups on pkg.freebsd.org are
> done internally to pkg(8),

.. which only works, if the DNS server queried answers SRV queries
with SRV values.

Which is not always true, especially in heavily firewalled environments.

--
p...@opsec.eu +49 171 3101372 7 years to go !

[Bernhard Fröhlich]

Am 02.11.2013 11:51 schrieb "Matthew Seaman" <mat...@freebsd.org>:
>
> On 02/11/2013 10:15, Matthias Andree wrote:

> > I understand from Eric's pist that the issue is that through his
> > limiting proxies, the SRV are not available at all so he does not even
> > get to the point where he could get the pkgN.nyi.freebsd.org

> > <http://pkgN.nyi.freebsd.org> name back.
>
> That doesn't make sense. All the DNS SRV lookups on pkg.freebsd.org are

> done internally to pkg(8), which then issues an HTTP GET to the specific
> mirror selected by its internal algorithms. The web cache won't see
> literal 'pkg.freebsd.org' anywhere in the HTTP traffic -- as far as it
> is concerned, it's a simple HTTP request to a specific mirror
> 'pkg1.nyi.freebsd.org', and can be cached using the usual processes.
>
> What makes it cache unfriendly is that as far as the web cache is
> concerned each of the different mirrors appears to be completely
> independent of the others. So at the moment the chance of getting a
> cache hit is reduced by a factor of three because of the traffic
> distribution across the three mirrors.

Just to add another viewpoint. The redports backendmachines are put into an
IPv6 private address space without default router and without a dns server.
The only internet connection that they have is via an squid proxy.
This setup works fine now that libfetch supports http proxies also for
https urls. This all works based on the assumption that no direct dns
lookups are required on the machines itself but all dns stuff is done on
the proxy.

Your description makes me believe that this won't work for pkgng. So it's
not that people in the real world break their network setups but we also
use that in our own FreeBSD infrastructure.

[Walter Hurry]

On Sat, 02 Nov 2013 09:01:56 -0500, Eric van Gyzen wrote:

> On 11/02/2013 07:28 AM, Matthew Seaman wrote:

>> I feel no obligation to do anything to encourage people that
>> deliberately break the DNS. They've made their bed, and now they have
>> to lie in it.
>
> Eric Camachat didn't break the DNS: his network administrator did.
> Matthew, you're right: that doesn't make sense. But people do it,
> often for security, either real or perceived. In this kind of
> environment, many other things are typically equally broken. I imagine
> Eric needs all the encouragement he can get.
>

When Matthew mentioned "people that deliberately break the DNS", I don't
think he meant Eric personally; I think he was referring to Eric's
*organisation*.

[Adrian Chadd]

On 2 November 2013 05:28, Matthew Seaman <mat...@freebsd.org> wrote:

> I feel no obligation to do anything to encourage people that
> deliberately break the DNS. They've made their bed, and now they have
> to lie in it.
>

Holy, holy crap.

* We (as FreeBSD) are not big enough to dictate the direction that
technology takes. In this instance, the direction that DNS SRV
adoption should be;
* This design is inherently not cachable, and as you add more CDN
nodes, it will become less cachable;
* And as far as I know, you haven't approached any cache vendors (eg
Squid) which may have the infrastructure to _handle_ this (which
Squid-2.x does, and I think Squid-3.x should be growing soon if it
hasn't already.)



You've removed the possibility of _standards_ and _well accepted_ HTTP
caching techniques without also deploying technology extensions in
popular open source projects to cope. You're using a DNS feature which
isn't well adopted/supported and you haven't provided a fallback
legacy, well tested path.

In short, you've taken the least supported paths, glued it into the
least HTTP caching scalable paths and not created a suitable fallback.

I hate to say it, but pushing the CDN logic into pkgng is a cute but
stupid idea for this deployment.

Please reconsider this choice before it becomes more widely deployed
and you/others have "moved onto other things", leaving it to others to
clean up.



-adrian

[Mark Felder]

On Nov 2, 2013, at 11:54 AM, Adrian Chadd <adr...@freebsd.org> wrote:

> You're using a DNS feature which
> isn't well adopted/supported and you haven't provided a fallback
> legacy, well tested path.

But SRV has been widely deployed since� before 2000? It�s literally the backbone of Active Directory deployments. Here�s a list of things that his company�s network design probably breaks:

* Office 365 (cloud Exchange hosting by Microsoft; requires you use SRV records to get your company�s clients pointed to their cloud infrastructure)
* LDAP
* SIP
* XMPP
* CALDAV / CARDDAV
* SMTP, IMAP, and POP clients should also obey published SRV records. Not sure how many clients really do, though.
* Teamspeak 3 doesn�t force you to use SRV, but you can use only SRV records
* Minecraft
* Last I knew IRCv4 specs are slated to include SRV as a core feature

I can�t speak for the caching issues, but SRV is pretty active and only getting more popular because things like �round robin DNS� are a horrible, ugly, unreliable hack and things like Anycast or Geo-DNS isn�t always feasible.

-0.02c

[Matthias Andree]

Am 02.11.2013 11:50, schrieb Matthew Seaman:
> On 02/11/2013 10:15, Matthias Andree wrote:

>> I understand from Eric's pist that the issue is that through his
>> limiting proxies, the SRV are not available at all so he does not even
>> get to the point where he could get the pkgN.nyi.freebsd.org

>> <http://pkgN.nyi.freebsd.org> name back.
>
> That doesn't make sense. All the DNS SRV lookups on pkg.freebsd.org are
> done internally to pkg(8), which then issues an HTTP GET to the specific
> mirror selected by its internal algorithms. The web cache won't see
> literal 'pkg.freebsd.org' anywhere in the HTTP traffic -- as far as it
> is concerned, it's a simple HTTP request to a specific mirror
> 'pkg1.nyi.freebsd.org', and can be cached using the usual processes.
>
> What makes it cache unfriendly is that as far as the web cache is
> concerned each of the different mirrors appears to be completely
> independent of the others. So at the moment the chance of getting a
> cache hit is reduced by a factor of three because of the traffic
> distribution across the three mirrors.

I think it does make sense - if the end user is behind a site where he
must use a proxy because his end user's computer does not resolve any
external addresses, then SRV is not getting you anywhere and you need a
HTTP(S)-based redirector.

[Adrian Chadd]

On 2 November 2013 10:44, Mark Felder <fe...@freebsd.org> wrote:

> But SRV has been widely deployed since… before 2000? It’s literally the backbone of Active Directory deployments. Here’s a list of things that his company’s network design probably breaks:
>
> * Office 365 (cloud Exchange hosting by Microsoft; requires you use SRV records to get your company’s clients pointed to their cloud infrastructure)

> * LDAP
> * SIP
> * XMPP
> * CALDAV / CARDDAV
> * SMTP, IMAP, and POP clients should also obey published SRV records. Not sure how many clients really do, though.

> * Teamspeak 3 doesn’t force you to use SRV, but you can use only SRV records

> * Minecraft
> * Last I knew IRCv4 specs are slated to include SRV as a core feature

Wonderful.

> I can’t speak for the caching issues, but SRV is pretty active and only getting more popular because things like “round robin DNS” are a horrible, ugly, unreliable hack and things like Anycast or Geo-DNS isn’t always feasible.

I can speak for the caching issues. It's a non-starter.

I'd rather see patches to Squid and such that support more automated
SRV handling (if it doesn't already do it; I haven't checked lately!)
and make things work correctly with caching. With a fallback, of
course, to A records.

A lot of HTTP infrastructure lives on anycast DNS, HTTP redirects and
geoip records. Saying it's broken and not feasible is nonsense.

Also - "all" you have to do is require all the servers in your farm to
handle requests for 'pkg.freebsd.org' rather than
'somethinguniqueperhost.freebsd.org' and then teach pkgng to actually
issue requests for that, and caching will mostly just work again.
Right now you're having SRV return a set of named aliases to issue
requests to and this set of hostnames is what's breaking effective
caching.

Sheesh!



-adrian

[Mark Felder]

On Nov 2, 2013, at 3:27 PM, Adrian Chadd <adr...@freebsd.org> wrote:

> A lot of HTTP infrastructure lives on anycast DNS, HTTP redirects and
> geoip records. Saying it's broken and not feasible is nonsense.

More specifically what I was referring to was the fact that traditionally HTTP failover with round-robin A records is very unreliable; every client can act differently. You really need to be doing anycast as well to ensure those records are always available which adds additional architecture complexity that the project may not have the resources to throw at. GeoDNS also adds a layer of complexity, but as it turns out there are members of the project with extensive experience running it. SRV would give us very simple, cheap, reliable failover. It seems we do have some blockers, though.

The good news is that we fully control the client. Hopefully we can just work around these issues.

[RW]

On Sat, 02 Nov 2013 12:28:06 +0000
Matthew Seaman wrote:


> > Which is not always true, especially in heavily firewalled
> > environments.
>

> I feel no obligation to do anything to encourage people that
> deliberately break the DNS. They've made their bed, and now they have
> to lie in it.

In other words, one more reason to choose Linux. You can only afford to
say "no soup for you" if you're the only soup-nazi in town.

I think there's an important distinction between broken dns and local
dns. If someone wants to provide controlled web access through a web
cache without giving general internet access then I don't see why they
shouldn't. This doesn't affect admins running servers, it affect
people trying to install FreeBSD on the locked-down part of the
network - typically the desktop machines of developers.

It also seems to be a fundamentally bad idea for a client that knows
it's connected to a proxy to be choosing the server in the first place.