How to build both ngx_pagespeed and mod_pagespeed from source?

[Ho Kuen Leung]

I want to make some changes to mod_pagespeed (change some logging message)

But I am using ngx_pagespeed with Nginx

I want to build a docker image with Nginx and ngx_pagespeed and my modified mod_pagespeed

I have read the wiki and both google groups for instructions

• https://github.com/pagespeed/ngx_pagespeed/wiki/Building-PSOL-From-Source
  
• https://groups.google.com/forum/#!searchin/mod-pagespeed-discuss/build$20from$20source|sort:relevance/mod-pagespeed-discuss/Nmy4hnIiVW4/ZKX0p2cvAwAJ
  

And tried with this code in my Dockerfile

The instruction is incomplete but I fail at `python build/gyp_chromium --depth=.`

with error message:

```

Updating projects from gyp files...

gyp: /root/mod_pagespeed/src/third_party/chromium/src/build/common.gypi not found (cwd: /root/mod_pagespeed/src) while reading includes of /root/mod_pagespeed/src/build/common.gypi while reading includes of /root/mod_pagespeed/src/build/all.gyp

```

RUN \
  set -x \
    && \
  apt-fast update && \
  apt-fast install -y --no-install-recommends \
    git \
    python \
    && \
  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*


# === Install pip === #
RUN \
set -x \
  && \
  DEBIAN_FRONTEND=noninteractive \
  apt-fast update \
  && \
  apt-fast install -y --no-install-recommends \
    python-dev \
  && \
  rm -rf /var/lib/apt/lists/* \
  && \
  curl https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py \
    && \
    python /tmp/get-pip.py \
    && \
    rm /tmp/get-pip.py \
  && \
  pip install -U pip setuptools
# === Install pip === #


# === Download module for Google Page Speed
#
# Ref:
# https://modpagespeed.com/doc/build_ngx_pagespeed_from_source
RUN \
  NPS_VERSION="1.12.34.2"; \
  PSOL_URL="https://dl.google.com/dl/page-speed/psol/${NPS_VERSION}.tar.gz"; \
  set -x \
    && \
  mkdir -p ~/bin \
    && \
    git clone https://chromium.googlesource.com/external/gyp.git ~/bin/tools/gyp \
    && \
    cd ~/bin/tools/gyp \
    && \
    python setup.py install \
    && \
    git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git ~/bin/depot_tools \
    && \
    mkdir ~/mod_pagespeed \
    && \
    cd ~/mod_pagespeed \
    && \
    git clone https://github.com/pagespeed/mod_pagespeed.git src \
    && \
    ~/bin/depot_tools/gclient config https://github.com/pagespeed/mod_pagespeed.git --unmanaged --name=src \
    && \
    cd ./src \
    && \
    git checkout ${BRANCH} \
    && \
    cd .. \
    && \
    ~/bin/depot_tools/gclient sync --force --jobs=1 \
    && \
    cd ~/mod_pagespeed/src/ \
    && \
    python build/gyp_chromium --depth=. \
    && \
    make BUILDTYPE=Release mod_pagespeed_test pagespeed_automatic_test \
    && \
    cd ~/mod_pagespeed/src/pagespeed/ \
    && \
    make BUILDTYPE=Release all \
    && \
    cd ~ \
    && \
    git clone https://github.com/pagespeed/ngx_pagespeed.git \
    && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
# === Download module for Google Page Speed

[Otto van der Schaaf]

See https://github.com/pagespeed/mod_pagespeed/wiki/Development

The flow looks like this these days:

# set this to your ngx_pagespeed fork's url
git clone YOUR_NGX_PAGESPEED_FORK
cd ngx_pagespeed
git submodule update --init --recursive

# edit .git/config to change the repo url for mod_pagespeed to be
# the one for your fork

scripts/build_ngx_pagespeed.sh --devel

(note: mod_pagespeed is included as a submodule, and is pinned at a specific revision though, so you may have to update it if you want to build 1.12.34.2 stable.)

The wiki you point out is for older versions, I'll add a note to it and cross link the wiki above.

Otto

--
You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.
Visit this group at https://groups.google.com/group/ngx-pagespeed-discuss.
For more options, visit https://groups.google.com/d/optout.

[Ho Kuen Leung]

Thanks for your answer.

I encounter more errors after that:

error: no available version of php-cgi php5-cgi
error: no available version of libapache2-mod-php libapache2-mod-php5
error: no available version of php-mbstring libapache2-mod-php5

Any more dependencies should be installed in addition to those specified?

[Otto van der Schaaf]

The script should install the dependencies, I wonder what went wrong. 

What os/distro versions did you run "scripts/build_ngx_pagespeed.sh --devel" on when you observed these errors?

To get an idea of what is needed you could take a look at 

- https://github.com/pagespeed/ngx_pagespeed/blob/master/scripts/build_ngx_pagespeed.sh#L484 (ngx_pagespeed)

- https://github.com/pagespeed/mod_pagespeed/blob/master/install/ubuntu/install_required_packages.sh (mod_pagespeed, ubuntu)

- https://github.com/pagespeed/mod_pagespeed/blob/master/install/centos/install_required_packages.sh (mod_pagespeed, centos) 

Otto

--

[Ho Kuen Leung]

The base image is https://github.com/phusion/baseimage-docker/releases/tag/0.9.22

Actual OS details:

```

$ lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 16.04.2 LTS

Release: 16.04

Codename: xenial

```

But even I manually installed all the missing packages it's reported

I will get this error complaining missing libtool:

```

Step 11/22 : RUN set -x     &&   cd /usr/local/src/ngx_pagespeed     &&     scripts/build_ngx_pagespeed.sh --devel

 ---> Running in 7d5f9720e614

+ cd /usr/local/src/ngx_pagespeed

+ scripts/build_ngx_pagespeed.sh --devel

Detected that we're running in an existing ngx_pagespeed checkout.

Detected debian-based distro.

Operating system dependencies are all set.

Downloading dependencies...

Switching submodules over to git protocol.

/usr/local/src/ngx_pagespeed/testing-dependencies/mod_pagespeed /usr/local/src/ngx_pagespeed

Reading package lists...

Building dependency tree...

Reading state information...

libev-dev is already the newest version (1:4.22-1).

libtool is already the newest version (2.4.6-0.1).

libpcre3-dev is already the newest version (2:8.38-3.1).

libapache2-mod-php is already the newest version (1:7.0+35ubuntu6).

apache2 is already the newest version (2.4.18-2ubuntu3.4).

psmisc is already the newest version (22.21-2.1build1).

curl is already the newest version (7.47.0-1ubuntu2.2).

memcached is already the newest version (1.4.25-2ubuntu1.2).

libssl-dev is already the newest version (1.0.2g-1ubuntu4.8).

php-cgi is already the newest version (1:7.0+35ubuntu6).

uuid-dev is already the newest version (2.27.1-6ubuntu3.3).

ssl-cert is already the newest version (1.0.37).

autoconf is already the newest version (2.69-9).

devscripts is already the newest version (2.16.2ubuntu3).

gperf is already the newest version (3.0.4-2).

fakeroot is already the newest version (1.20.2-1ubuntu1).

rsync is already the newest version (3.1.1-3ubuntu1).

php-mbstring is already the newest version (1:7.0+35ubuntu6).

wget is already the newest version (1.17.1-1ubuntu1.2).

subversion is already the newest version (1.9.3-2ubuntu1).

g++ is already the newest version (4:5.3.1-1ubuntu1).

g++ set to manually installed.

valgrind is already the newest version (1:3.11.0-1ubuntu4.2).

language-pack-tr-base is already the newest version (1:16.04+20160627).

openjdk-7-jre is already the newest version (7u95-2.6.4-3).

openjdk-7-jre set to manually installed.

net-tools is already the newest version (1.60-26ubuntu1).

redis-server is already the newest version (2:3.0.6-1).

git is already the newest version (1:2.7.4-0ubuntu1.1).

zlib1g-dev is already the newest version (1:1.2.8.dfsg-2ubuntu4.1).

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

buildconf: checking installation...

buildconf: python version 2.7.12 (ok)

buildconf: autoconf version 2.69 (ok)

buildconf: libtool not found.

           You need libtool version 1.4 or newer installed

           to build APR from SVN.

Error: Failure running 'install/build_development_apache.sh 2.2 prefork', exiting.

```

[Otto van der Schaaf]

Could you try this?:

sudo apt-get install libtool-bin

Otto

--

[Ho Kuen Leung]

It runs now

But some script got sudo and will generate error "command not found"

https://github.com/pagespeed/mod_pagespeed/blob/master/install/build_psol.sh#L56

Otto van der Schaaf於 2017年8月7日星期一 UTC+8下午10時59分43秒寫道：

[Ho Kuen Leung]

Should I fork it and remove all sudo myself?

[Otto van der Schaaf]

If all else fails, cloning may work for you. 

But alternatively, if the changes you need are generally useful, pull requests are welcome too!

One thing you could try to avoid needing sudo is passing --no-deps-check, see

https://github.com/pagespeed/ngx_pagespeed/blob/master/scripts/build_ngx_pagespeed.sh#L60

Otto

On Tue, Aug 15, 2017 at 9:22 AM Ho Kuen Leung <pikac...@gmail.com> wrote:

Should I fork it and remove all sudo myself?

--

[Ho Kuen Leung]

I passed in --no-deps-check but it seems still trying to install packages

I guess it's due to

https://github.com/pagespeed/ngx_pagespeed/blob/master/scripts/build_ngx_pagespeed.sh#L637

is always trying to install dependencies, and does not pass --skip_deps when --no-deps-check is input

Log:

+ cd /usr/local/src/ngx_pagespeed

+ scripts/build_ngx_pagespeed.sh --psol-from-source --assume-yes --no-deps-check

Detected that we're running in an existing ngx_pagespeed checkout.

Not checking whether operating system dependencies are installed.

/usr/local/src/ngx_pagespeed/testing-dependencies/mod_pagespeed /usr/local/src/ngx_pagespeed

Installing required packages...

[ 7:34:32] sudo install/install_required_packages.sh --additional_dev_packages >> log/install_deps.log

End of log/install_deps.log:

[ 7:34:32] sudo install/install_required_packages.sh --additional_dev_packages

install/shell_utils.sh: line 190: sudo: command not found

[ 7:34:32] Completed with exit status 127

[Otto van der Schaaf]

I filed https://github.com/pagespeed/ngx_pagespeed/issues/1459 for tracking that build_ngx_pagespeed.sh cascades the --no-deps-check argument to subscripts that get called.

Otto

--