The bug seems to be quite sensitive to the polynomials involved. Deleting any subset of factors seems to not trigger the bug, and multiplying through by the denominator causes it to disappear as well (though other examples still trigger the bug even when all coefficients are integral).
When I try factoring in gp, I don't get an error (thus why I'm writing to sage-devel rather than Pari).
If anyone has ideas on how to diagnose this, or reproduce it in a format more easily reportable to the Pari developers, I'd be happy to hear it. In the meantime, calling squarefree_decomposition and then factoring each of the resulting terms seems to be a functional workaround for my purposes.
RAM: 256GB in 16 Hynix DDR3 DIMMs (the bug occurs when only a small amount of memory is in use)
sage: R.<x> = QQ[]
sage: f = (x - 1)^8 * (x^2 - 2/3*x + 1)^2 * (x^2 + 2/3*x + 1)^2 * (x^4 + 3*x^3 + 37/9*x^2 + 3*x + 1) * (x^4 + 1/3*x^3 - x^2 + 1/3*x + 1)^2 * (x^16 + 2*x^14 + 211/81*x^12 + 28/9*x^10 + 269/81*x^8 + 28/9*x^6 + 211/81*x^4 + 2*x^2 + 1)^2
sage: g = (x - 1)^32 * (x^2 + 5/3*x + 1)^16
sage: factor(f)
(x - 1)^8 * (x^2 - 2/3*x + 1)^2 * (x^2 + 2/3*x + 1)^2 * (x^4 + 3*x^3 + 37/9*x^2 + 3*x + 1) * (x^4 + 1/3*x^3 - x^2 + 1/3*x + 1)^2 * (x^16 + 2*x^14 + 211/81*x^12 + 28/9*x^10 + 269/81*x^8 + 28/9*x^6 + 211/81*x^4 + 2*x^2 + 1)^2
sage: factor(g)
---------------------------------------------------------------------------
PariError Traceback (most recent call last)
<ipython-input-5-41d64ee9c816> in <module>()
----> 1 factor(g)
/home/sage/sage-8.7/local/lib/python2.7/site-packages/sage/arith/misc.pyc in factor(n, proof, int_, algorithm, verbose, **kwds)
2563 except TypeError:
2564 # Maybe the factor() method doesn't have a proof option
-> 2565 return m(**kwds)
2566
2567
/home/sage/sage-8.7/local/lib/python2.7/site-packages/sage/rings/polynomial/polynomial_element.pyx in sage.rings.polynomial.polynomial_element.Polynomial.factor (build/cythonized/sage/rings/polynomial/polynomial_element.c:35617)()
4252 R = self._parent.base_ring()
4253 if hasattr(R, '_factor_univariate_polynomial'):
-> 4254 return R._factor_univariate_polynomial(self, **kwargs)
4255
4256 G = None
/home/sage/sage-8.7/local/lib/python2.7/site-packages/sage/rings/rational_field.pyc in _factor_univariate_polynomial(self, f)
1348
1349 """
-> 1350 G = list(f._pari_with_name().factor())
1351
1352 # normalize the leading coefficients
cypari2/gen.pyx in cypari2.gen.Gen.factor()
cypari2/handle_error.pyx in cypari2.handle_error._pari_err_handle()
PariError: bug in gerepile, significant pointers lost, please report