Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Disable ASPM on various devices

216 views
Skip to first unread message

Matthew Garrett

unread,
Nov 11, 2011, 11:10:01 AM11/11/11
to
If the firmware has given us control of PCIe capabilities then it's valid
for an operating system to configure ASPM more aggressively than the
firmware did. A small number of devices object to this and exhibit various
failure modes. Windows provides a mechanism to disable ASPM in the driver,
indicated by the Needs=PciASPMOptOut statement in the .inf file. Trawling
through Windows drivers has indicated the following set of hardware that
disables ASPM in Windows but doesn't currently disable it in Linux. It makes
sense for us to mimic Windows in this situation.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Matthew Garrett

unread,
Nov 11, 2011, 11:10:01 AM11/11/11
to
http://www.atheros.cz/atheros-inf-file.php?inf=209&chipset=45&system=6
indicates that ASPM is disabled on all L1E hardware. Duplicate for sanity.

Signed-off-by: Matthew Garrett <m...@redhat.com>
Cc: net...@vger.kernel.org
Cc: jcli...@gmail.com
Cc: chris...@gmail.com
---
drivers/net/ethernet/atheros/atl1e/atl1e_main.c | 3 +++
1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/drivers/net/ethernet/atheros/atl1e/atl1e_main.c b/drivers/net/ethernet/atheros/atl1e/atl1e_main.c
index 95483bc..fc74dd1 100644
--- a/drivers/net/ethernet/atheros/atl1e/atl1e_main.c
+++ b/drivers/net/ethernet/atheros/atl1e/atl1e_main.c
@@ -2264,6 +2264,9 @@ static int __devinit atl1e_probe(struct pci_dev *pdev,

int err = 0;

+ pci_disable_link_state(pdev, PCIE_LINK_STATE_L0S | PCIE_LINK_STATE_L1 |
+ PCIE_LINK_STATE_CLKPM);
+
err = pci_enable_device(pdev);
if (err) {
dev_err(&pdev->dev, "cannot enable PCI device\n");
--
1.7.7.1

Matthew Garrett

unread,
Nov 11, 2011, 11:10:01 AM11/11/11
to
The Windows driver .inf disables ASPM on all cciss devices. Do the same.

Signed-off-by: Matthew Garrett <m...@redhat.com>
Cc: mike....@hp.com
Cc: iss_sto...@hp.com
Cc: ax...@kernel.dk
---
drivers/block/cciss.c | 4 ++++
1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/drivers/block/cciss.c b/drivers/block/cciss.c
index 486f94e..7e43ad3 100644
--- a/drivers/block/cciss.c
+++ b/drivers/block/cciss.c
@@ -4319,6 +4319,10 @@ static int __devinit cciss_pci_init(ctlr_info_t *h)
dev_warn(&h->pdev->dev, "controller appears to be disabled\n");
return -ENODEV;
}
+
+ pci_disable_link_state(h->pdev, PCIE_LINK_STATE_L0S |
+ PCIE_LINK_STATE_L1 | PCIE_LINK_STATE_CLKPM);
+
err = pci_enable_device(h->pdev);
if (err) {
dev_warn(&h->pdev->dev, "Unable to Enable PCI device\n");
--
1.7.7.1

Matthew Garrett

unread,
Nov 11, 2011, 11:10:01 AM11/11/11
to
http://driveragent.com/archive/30421/7-0-14 indicates that ASPM is
disabled on the 250 and 260. Duplicate for sanity.

Signed-off-by: Matthew Garrett <m...@redhat.com>
Cc: net...@vger.kernel.org
Cc: cool...@cooldavid.org
---
drivers/net/ethernet/jme.c | 3 +++
1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/drivers/net/ethernet/jme.c b/drivers/net/ethernet/jme.c
index 7becff1..b28a873 100644
--- a/drivers/net/ethernet/jme.c
+++ b/drivers/net/ethernet/jme.c
@@ -2860,6 +2860,9 @@ jme_init_one(struct pci_dev *pdev,
/*
* set up PCI device basics
*/
+ pci_disable_link_state(pdev, PCIE_LINK_STATE_L0S | PCIE_LINK_STATE_L1 |
+ PCIE_LINK_STATE_CLKPM);
+
rc = pci_enable_device(pdev);
if (rc) {
pr_err("Cannot enable PCI device\n");
--
1.7.7.1

Matthew Garrett

unread,
Nov 11, 2011, 11:10:02 AM11/11/11
to
The Windows driver disables ASPM support for various chipsets supported
by atl1c. This adds the same set of logic to the Linux driver. ASPM is
disabled on l1c, l2c, l2cb and l2cb2 devices except for those in Toshiba
or Lenovo devices. Data taken from
http://www.atheros.cz/atheros-inf-file.php?inf=199&chipset=51&system=6

Signed-off-by: Matthew Garrett <m...@redhat.com>
Cc: net...@vger.kernel.org
Cc: jcli...@gmail.com
Cc: chris...@gmail.com
---
drivers/net/ethernet/atheros/atl1c/atl1c_main.c | 15 +++++++++++++++
1 files changed, 15 insertions(+), 0 deletions(-)

diff --git a/drivers/net/ethernet/atheros/atl1c/atl1c_main.c b/drivers/net/ethernet/atheros/atl1c/atl1c_main.c
index 02c7ed8..d91dabd 100644
--- a/drivers/net/ethernet/atheros/atl1c/atl1c_main.c
+++ b/drivers/net/ethernet/atheros/atl1c/atl1c_main.c
@@ -19,6 +19,7 @@
* Temple Place - Suite 330, Boston, MA 02111-1307, USA.
*/

+#include <linux/pci-aspm.h>
#include "atl1c.h"

#define ATL1C_DRV_VERSION "1.0.1.0-NAPI"
@@ -2652,6 +2653,20 @@ static int __devinit atl1c_probe(struct pci_dev *pdev,

int err = 0;

+ switch (pdev->device) {
+ case PCI_DEVICE_ID_ATTANSIC_L1C:
+ case PCI_DEVICE_ID_ATTANSIC_L2C:
+ case PCI_DEVICE_ID_ATHEROS_L2C_B:
+ case PCI_DEVICE_ID_ATHEROS_L2C_B2:
+ if (pdev->subsystem_vendor == PCI_VENDOR_ID_TOSHIBA ||
+ pdev->subsystem_vendor == PCI_VENDOR_ID_LENOVO)
+ break;
+ pci_disable_link_state(pdev, PCIE_LINK_STATE_L0S |
+ PCIE_LINK_STATE_L1 |
+ PCIE_LINK_STATE_CLKPM);
+ break;
+ }
+
/* enable device (incl. PCI PM wakeup and hotplug setup) */
err = pci_enable_device_mem(pdev);
if (err) {
--
1.7.7.1

Matthew Garrett

unread,
Nov 11, 2011, 11:10:02 AM11/11/11
to
The Windows driver .inf disables ASPM on hpsa devices. Do the same.

Signed-off-by: Matthew Garrett <m...@redhat.com>
Cc: faisal...@intel.com
Cc: linux...@vger.kernel.org
---
drivers/infiniband/hw/nes/nes.c | 3 +++
1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/drivers/infiniband/hw/nes/nes.c b/drivers/infiniband/hw/nes/nes.c
index 5965b3d..2737b7a 100644
--- a/drivers/infiniband/hw/nes/nes.c
+++ b/drivers/infiniband/hw/nes/nes.c
@@ -463,6 +463,9 @@ static int __devinit nes_probe(struct pci_dev *pcidev, const struct pci_device_i
printk(KERN_INFO PFX "NetEffect RNIC driver v%s loading. (%s)\n",
DRV_VERSION, pci_name(pcidev));

+ pci_disable_link_state(pcidev, PCIE_LINK_STATE_L0S |
+ PCIE_LINK_STATE_L1 | PCIE_LINK_STATE_CLKPM);
+
ret = pci_enable_device(pcidev);
if (ret) {
printk(KERN_ERR PFX "Unable to enable PCI device. (%s)\n", pci_name(pcidev));
--
1.7.7.1

Matthew Garrett

unread,
Nov 11, 2011, 11:10:03 AM11/11/11
to
The Windows driver .inf disables ASPM on hpsa devices. Do the same.

Signed-off-by: Matthew Garrett <m...@redhat.com>
Cc: scam...@beardog.cce.hp.com
Cc: iss_sto...@hp.com
Cc: linux...@vger.kernel.org
---
drivers/scsi/hpsa.c | 4 ++++
1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/drivers/scsi/hpsa.c b/drivers/scsi/hpsa.c
index e76107b..fc2f4c4 100644
--- a/drivers/scsi/hpsa.c
+++ b/drivers/scsi/hpsa.c
@@ -3922,6 +3922,10 @@ static int __devinit hpsa_pci_init(struct ctlr_info *h)
dev_warn(&h->pdev->dev, "controller appears to be disabled\n");
return -ENODEV;
}
+
+ pci_disable_link_state(h->pdev, PCIE_LINK_STATE_L0S |
+ PCIE_LINK_STATE_L1 | PCIE_LINK_STATE_CLKPM);
+
err = pci_enable_device(h->pdev);
if (err) {
dev_warn(&h->pdev->dev, "unable to enable PCI device\n");
--
1.7.7.1

Matthew Garrett

unread,
Nov 11, 2011, 11:20:01 AM11/11/11
to
The Windows driver .inf disables ASPM on hpsa devices. Do the same.

Signed-off-by: Matthew Garrett <m...@redhat.com>
Cc: scam...@beardog.cce.hp.com
Cc: iss_sto...@hp.com
Cc: linux...@vger.kernel.org
---
drivers/scsi/hpsa.c | 5 +++++
1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/drivers/scsi/hpsa.c b/drivers/scsi/hpsa.c
index e76107b..865d452 100644
--- a/drivers/scsi/hpsa.c
+++ b/drivers/scsi/hpsa.c
@@ -23,6 +23,7 @@
#include <linux/interrupt.h>
#include <linux/types.h>
#include <linux/pci.h>
+#include <linux/pci-aspm.h>
#include <linux/kernel.h>
#include <linux/slab.h>
#include <linux/delay.h>
@@ -3922,6 +3923,10 @@ static int __devinit hpsa_pci_init(struct ctlr_info *h)

Matthew Garrett

unread,
Nov 11, 2011, 11:20:02 AM11/11/11
to
If the firmware has given us control of PCIe capabilities then it's
valid for an operating system to configure ASPM more aggressively than
the firmware did. A small number of devices object to this and exhibit
various failure modes. Windows provides a mechanism to disable ASPM in
the driver, indicated by the Needs=PciASPMOptOut statement in the .inf
file. Trawling through Windows drivers has indicated the following set
of hardware that disables ASPM in Windows but doesn't currently disable
it in Linux. It makes sense for us to mimic Windows in this situation.

(V2: send the version that actually builds)

Matthew Garrett

unread,
Nov 11, 2011, 11:20:02 AM11/11/11
to
http://www.atheros.cz/atheros-inf-file.php?inf=209&chipset=45&system=6
indicates that ASPM is disabled on all L1E hardware. Duplicate for sanity.

Signed-off-by: Matthew Garrett <m...@redhat.com>
drivers/net/ethernet/atheros/atl1e/atl1e_main.c | 3 +++
1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/drivers/net/ethernet/atheros/atl1e/atl1e_main.c b/drivers/net/ethernet/atheros/atl1e/atl1e_main.c
index 95483bc..fc74dd1 100644
--- a/drivers/net/ethernet/atheros/atl1e/atl1e_main.c
+++ b/drivers/net/ethernet/atheros/atl1e/atl1e_main.c
@@ -2264,6 +2264,9 @@ static int __devinit atl1e_probe(struct pci_dev *pdev,

int err = 0;

+ pci_disable_link_state(pdev, PCIE_LINK_STATE_L0S | PCIE_LINK_STATE_L1 |
+ PCIE_LINK_STATE_CLKPM);
+
err = pci_enable_device(pdev);
if (err) {
dev_err(&pdev->dev, "cannot enable PCI device\n");
--
1.7.7.1

Matthew Garrett

unread,
Nov 11, 2011, 11:20:02 AM11/11/11
to
The Windows driver .inf disables ASPM on all cciss devices. Do the same.

Signed-off-by: Matthew Garrett <m...@redhat.com>
drivers/block/cciss.c | 5 +++++
1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/drivers/block/cciss.c b/drivers/block/cciss.c
index 486f94e..92d8a2f 100644
--- a/drivers/block/cciss.c
+++ b/drivers/block/cciss.c
@@ -24,6 +24,7 @@
#include <linux/interrupt.h>
#include <linux/types.h>
#include <linux/pci.h>
+#include <linux/pci-aspm.h>
#include <linux/kernel.h>
#include <linux/slab.h>
#include <linux/delay.h>
@@ -4319,6 +4320,10 @@ static int __devinit cciss_pci_init(ctlr_info_t *h)
dev_warn(&h->pdev->dev, "controller appears to be disabled\n");
return -ENODEV;
}
+
+ pci_disable_link_state(h->pdev, PCIE_LINK_STATE_L0S |
+ PCIE_LINK_STATE_L1 | PCIE_LINK_STATE_CLKPM);
+
err = pci_enable_device(h->pdev);
if (err) {
dev_warn(&h->pdev->dev, "Unable to Enable PCI device\n");
--
1.7.7.1

Matthew Garrett

unread,
Nov 11, 2011, 11:20:02 AM11/11/11
to
The Windows driver .inf disables ASPM on hpsa devices. Do the same.

Signed-off-by: Matthew Garrett <m...@redhat.com>
drivers/infiniband/hw/nes/nes.c | 4 ++++
1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/drivers/infiniband/hw/nes/nes.c b/drivers/infiniband/hw/nes/nes.c
index 5965b3d..ea320f5 100644
--- a/drivers/infiniband/hw/nes/nes.c
+++ b/drivers/infiniband/hw/nes/nes.c
@@ -45,6 +45,7 @@
#include <linux/if_arp.h>
#include <linux/highmem.h>
#include <linux/slab.h>
+#include <linux/pci-aspm.h>
#include <asm/io.h>
#include <asm/irq.h>
#include <asm/byteorder.h>
@@ -463,6 +464,9 @@ static int __devinit nes_probe(struct pci_dev *pcidev, const struct pci_device_i
printk(KERN_INFO PFX "NetEffect RNIC driver v%s loading. (%s)\n",
DRV_VERSION, pci_name(pcidev));

+ pci_disable_link_state(pcidev, PCIE_LINK_STATE_L0S |
+ PCIE_LINK_STATE_L1 | PCIE_LINK_STATE_CLKPM);
+
ret = pci_enable_device(pcidev);
if (ret) {
printk(KERN_ERR PFX "Unable to enable PCI device. (%s)\n", pci_name(pcidev));
--
1.7.7.1

Matthew Garrett

unread,
Nov 11, 2011, 11:20:03 AM11/11/11
to
http://driveragent.com/archive/30421/7-0-14 indicates that ASPM is
disabled on the 250 and 260. Duplicate for sanity.

Signed-off-by: Matthew Garrett <m...@redhat.com>
drivers/net/ethernet/jme.c | 4 ++++
1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/drivers/net/ethernet/jme.c b/drivers/net/ethernet/jme.c
index 7becff1..88e7ce1 100644
--- a/drivers/net/ethernet/jme.c
+++ b/drivers/net/ethernet/jme.c
@@ -27,6 +27,7 @@
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/pci.h>
+#include <linux/pci-aspm.h>
#include <linux/netdevice.h>
#include <linux/etherdevice.h>
#include <linux/ethtool.h>
@@ -2860,6 +2861,9 @@ jme_init_one(struct pci_dev *pdev,
/*
* set up PCI device basics
*/
+ pci_disable_link_state(pdev, PCIE_LINK_STATE_L0S | PCIE_LINK_STATE_L1 |
+ PCIE_LINK_STATE_CLKPM);
+
rc = pci_enable_device(pdev);
if (rc) {
pr_err("Cannot enable PCI device\n");
--
1.7.7.1

Matthew Garrett

unread,
Nov 11, 2011, 11:20:03 AM11/11/11
to
The Windows driver disables ASPM support for various chipsets supported
by atl1c. This adds the same set of logic to the Linux driver. ASPM is
disabled on l1c, l2c, l2cb and l2cb2 devices except for those in Toshiba
or Lenovo devices. Data taken from
http://www.atheros.cz/atheros-inf-file.php?inf=199&chipset=51&system=6

Signed-off-by: Matthew Garrett <m...@redhat.com>
Cc: net...@vger.kernel.org
Cc: jcli...@gmail.com
Cc: chris...@gmail.com
---
drivers/net/ethernet/atheros/atl1c/atl1c_main.c | 15 +++++++++++++++
1 files changed, 15 insertions(+), 0 deletions(-)

diff --git a/drivers/net/ethernet/atheros/atl1c/atl1c_main.c b/drivers/net/ethernet/atheros/atl1c/atl1c_main.c
index 02c7ed8..d91dabd 100644
--- a/drivers/net/ethernet/atheros/atl1c/atl1c_main.c
+++ b/drivers/net/ethernet/atheros/atl1c/atl1c_main.c
@@ -19,6 +19,7 @@
* Temple Place - Suite 330, Boston, MA 02111-1307, USA.
*/

+#include <linux/pci-aspm.h>
#include "atl1c.h"

#define ATL1C_DRV_VERSION "1.0.1.0-NAPI"
@@ -2652,6 +2653,20 @@ static int __devinit atl1c_probe(struct pci_dev *pdev,

int err = 0;

+ switch (pdev->device) {
+ case PCI_DEVICE_ID_ATTANSIC_L1C:
+ case PCI_DEVICE_ID_ATTANSIC_L2C:
+ case PCI_DEVICE_ID_ATHEROS_L2C_B:
+ case PCI_DEVICE_ID_ATHEROS_L2C_B2:
+ if (pdev->subsystem_vendor == PCI_VENDOR_ID_TOSHIBA ||
+ pdev->subsystem_vendor == PCI_VENDOR_ID_LENOVO)
+ break;
+ pci_disable_link_state(pdev, PCIE_LINK_STATE_L0S |
+ PCIE_LINK_STATE_L1 |
+ PCIE_LINK_STATE_CLKPM);
+ break;
+ }
+
/* enable device (incl. PCI PM wakeup and hotplug setup) */
err = pci_enable_device_mem(pdev);
if (err) {
--
1.7.7.1

Mike Miller

unread,
Nov 11, 2011, 11:30:01 AM11/11/11
to
Acked-by: Mike Miller <mike....@hp.com>

Mike Miller

unread,
Nov 11, 2011, 11:30:01 AM11/11/11
to
On 11/11/2011 10:14 AM, Matthew Garrett wrote:
Acked-by: Mike Miller <mike....@hp.com>

Roland Dreier

unread,
Nov 11, 2011, 12:40:02 PM11/11/11
to
On Fri, Nov 11, 2011 at 8:05 AM, Matthew Garrett <m...@redhat.com> wrote:
> The Windows driver .inf disables ASPM on hpsa devices. Do the same.

This isn't a patch to the hpsa driver... cut and paste in the description,
or misfire on the patch?

Faisal, any info on whether this is needed/desired for nes devices
based on Intel's knowledge of the PCIe core used?

- R.

Matthew Garrett

unread,
Nov 11, 2011, 2:30:01 PM11/11/11
to
On Fri, Nov 11, 2011 at 09:31:44AM -0800, Roland Dreier wrote:
> On Fri, Nov 11, 2011 at 8:05 AM, Matthew Garrett <m...@redhat.com> wrote:
> > The Windows driver .inf disables ASPM on hpsa devices. Do the same.
>
> This isn't a patch to the hpsa driver... cut and paste in the description,
> or misfire on the patch?

Sorry, yes, cut and paste in the description. It should say nes.

--
Matthew Garrett | mj...@srcf.ucam.org

Roland Dreier

unread,
Nov 11, 2011, 2:40:02 PM11/11/11
to
> Sorry, yes, cut and paste in the description. It should say nes.

OK. Just got a vacation message from Faisal that he's out til
next week, let's see if Intel has a comment on this.

Thanks,
Roland

Jens Axboe

unread,
Nov 11, 2011, 4:10:01 PM11/11/11
to
Applied for 3.2, thanks.

--
Jens Axboe

David Miller

unread,
Nov 14, 2011, 12:40:02 AM11/14/11
to
From: Matthew Garrett <m...@redhat.com>
Date: Fri, 11 Nov 2011 11:14:20 -0500

> http://www.atheros.cz/atheros-inf-file.php?inf=209&chipset=45&system=6
> indicates that ASPM is disabled on all L1E hardware. Duplicate for sanity.
>
> Signed-off-by: Matthew Garrett <m...@redhat.com>

Applied.

David Miller

unread,
Nov 14, 2011, 12:40:02 AM11/14/11
to

I'm reverting all of these.

You do all of that work to research the inf files and whatnot, then
you don't even build test the patches you submit?

Forget it, for something as potentially fragile as this stuff might
be, I simply cannot trust you if you can't even do due diligence
wrt. the build validation.

David Miller

unread,
Nov 14, 2011, 12:40:02 AM11/14/11
to
From: Matthew Garrett <m...@redhat.com>
Date: Fri, 11 Nov 2011 11:14:21 -0500

> http://driveragent.com/archive/30421/7-0-14 indicates that ASPM is
> disabled on the 250 and 260. Duplicate for sanity.
>
> Signed-off-by: Matthew Garrett <m...@redhat.com>

Applied.

David Miller

unread,
Nov 14, 2011, 12:40:02 AM11/14/11
to
From: David Miller <da...@davemloft.net>
Date: Mon, 14 Nov 2011 00:31:13 -0500 (EST)

> From: Matthew Garrett <m...@redhat.com>
> Date: Fri, 11 Nov 2011 11:14:20 -0500
>
>> http://www.atheros.cz/atheros-inf-file.php?inf=209&chipset=45&system=6
>> indicates that ASPM is disabled on all L1E hardware. Duplicate for sanity.
>>
>> Signed-off-by: Matthew Garrett <m...@redhat.com>
>
> Applied.

Ummm, -ENOBUILDTESTING?!??!

drivers/net/ethernet/atheros/atl1e/atl1e_main.c: In function ‘atl1e_probe’:
drivers/net/ethernet/atheros/atl1e/atl1e_main.c:2267:2: error: implicit declaration of function ‘pci_disable_link_state’ [-Werror=implicit-function-declaration]
drivers/net/ethernet/atheros/atl1e/atl1e_main.c:2267:31: error: ‘PCIE_LINK_STATE_L0S’ undeclared (first use in this function)
drivers/net/ethernet/atheros/atl1e/atl1e_main.c:2267:31: note: each undeclared identifier is reported only once for each function it appears in
drivers/net/ethernet/atheros/atl1e/atl1e_main.c:2267:53: error: ‘PCIE_LINK_STATE_L1’ undeclared (first use in this function)
drivers/net/ethernet/atheros/atl1e/atl1e_main.c:2268:11: error: ‘PCIE_LINK_STATE_CLKPM’ undeclared (first use in this function)

David Miller

unread,
Nov 14, 2011, 12:40:02 AM11/14/11
to
From: Matthew Garrett <m...@redhat.com>
Date: Fri, 11 Nov 2011 11:14:19 -0500

> The Windows driver disables ASPM support for various chipsets supported
> by atl1c. This adds the same set of logic to the Linux driver. ASPM is
> disabled on l1c, l2c, l2cb and l2cb2 devices except for those in Toshiba
> or Lenovo devices. Data taken from
> http://www.atheros.cz/atheros-inf-file.php?inf=199&chipset=51&system=6
>
> Signed-off-by: Matthew Garrett <m...@redhat.com>

Applied.

Matthew Garrett

unread,
Nov 14, 2011, 10:50:02 AM11/14/11
to
Sorry, I screwed up there - it looks like I tested atl1c twice. Still
inexcusable, and apologies for the wasted time.

--
Matthew Garrett | mj...@srcf.ucam.org
0 new messages