For novices, what is the absolute simplest way of getting things set up?

[Thomas Wright]

Hi All,

I tried a few months ago to get a fixmystreet instance up and running for my local community. I ended up getting very bogged down trying to understand how to get things running with docker and mixing up the development and production environments, I left things stuck with a docker stack running in portainer that just crashes at load banging my head against a wall trying to get things running.
I run a little home automation server with plex and a few docker containers, and am used to tinkering with the likes of home assistant.

I'm coming back to try again with fixmystreet but I'd like a bit of help. Can you please treat me like a big dummy who has little experience running a linux server.

I have a separate beelink x86 machine I've bought just for fixmystreet. Ideally I'd like to be running this via docker so I have the option a few of my own containers down the line. I've got a domain ready and a DDNS pointing to my home IP, and will push that domain through a reverse proxy running on my other server.

With this setup can anyone help me out with what commands I should be running to get everything set up so that I can just get to tinkering with the high level yaml files and the branding files?

Assume I'm starting with a fresh install of ubuntu server and I've got my user accounts setup.

[Thomas Wright]

Okay I managed to get an instance up and running, and now I'm trying to brute force my way to getting a working set of config files.

In any case, here are the steps I took to get a docker version of this up and running on my machine running ubuntu server

Install ubuntu server

https://ubuntu.com/server/docs/installation

https://www.zdnet.com/article/how-to-install-ubuntu-server-in-less-than-30-minutes/

As part of the install you optionally decide to create a separate storage volume, I created  ssa en ext4 format and with the mountpoint ssd for my fixmystreet install.

At the end make sure to include installing OpenSSH, I didn't install any of the optional extra features.

Logging into the server

After installing, you can switch over to using your normal machine and SSH into your server.

E.g. when on your server run the following command to print out your machine IP address 

hostname -I

Which should show your machine's IP address, something like 192.168.x.x

I use a windows machine normally, and use Powershell to SSH into my linux boxes

So I hit the windows button, and type "Powershell" and open that

In the PowerShell window type:

ssh user...@192.168.x.x

Where username is the user you set up as part of the ubuntu install and the IP address is the one you looked up. It should ask for your password and then let you in.

Install Git

https://git-scm.com/book/en/v2/Getting-Started-Installing-Git

Installed docker

https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository

Install docker-compose

(Step 1 already completed above)

https://docs.docker.com/compose/install/linux/#install-using-the-repository

Installed Portainer Community Edition (Optional)

Portainer makes it a lot easier for me to interact with the fixmystreet containers to restart them as necessary, although I couldn't get the actual installation to work from Portainer

Important - make sure to change the port portainer runs on from 8000 to something else as this clashes with the default port of FixMyStreet which is also 8000

https://docs.portainer.io/start/install-ce/server/docker/linux

Once installed you should be able to access Portainer on the port you specified, probably 9000

In your browser, go to 192.168.x.x:9000 (or whatever your server IP is) 

You should be prompted to set up a new account.

Once logged in, bookmark the page as Portainer.

Fork the fixmystreet repository

Even if you don't plan on making any changes, you'll probably end up tinkering with the cobrand templates or some other config files so it's worth forking the official repository so you can keep a backup on github of your work.

If you don't already have one, set up a github account https://github.com/signup

Go to the official repo: https://github.com/mysociety/fixmystreet

Click the Fork button in the top right and follow the prompts until you've completed the fork

Keep note of the URL of your forked repo for the next step.

Clone the repository to your server

Cloning will copy the repository to your local machine.

First create a folder for your installation.

Double check where you'll install by looking up your storage volumes and mountpoints with this command:

sudo lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL

E.g. for me that pinted out:

NAME                      FSTYPE        SIZE MOUNTPOINT        LABEL
loop0                     squashfs     63.5M /snap/core20/2015
loop1                     squashfs    111.9M /snap/lxd/24322
loop2                     squashfs       87M /snap/lxd/27948
loop3                     squashfs     63.9M /snap/core20/2182
loop4                     squashfs     40.9M /snap/snapd/20290
loop5                     squashfs     39.1M /snap/snapd/21184
sda                       ext4        931.5G /ssd
nvme0n1                               238.5G
├─nvme0n1p1               vfat            1G /boot/efi
├─nvme0n1p2               ext4            2G /boot
└─nvme0n1p3               LVM2_member 235.4G
  └─ubuntu--vg-ubuntu--lv ext4          100G /

and I can see my largest mountpoint is /ssd - the volume I'd set up earlier for the install.

So I run the command

cd /ssd

Which opens that directory.

Then make a new directory for your install, something like "fixmystreet"

mkdir fixmystreet

Then open that directory

cd fixmystreet

Now you're in the right spot, you can clone your repository here, you'll need the URL of the repository you forked. E.g. in my case that is https://github.com/ThomasHFWright/fixmystreet

Then run the command to clone it using git:

git clone https://github.com/yourusername/yourrepositoryname

Now you have the files, just initiate git to track any changes so you can pull them down as you make any on your local machine

git init

Now, when you make any changes to your repository you can pull the changes to your server using the command:

git pull

Start fixmystreet

Now you've cloned the repo, you can go ahead and build the stack of containers using docker-compose

In the same directory as you cloned into use the following command

docker-compose up

It should build the containers and start everything up with the default settings and passwords. Note that we expect to delete the containers created, this step is just to confirm things work.

After some time new messages will stop appearing, and you can hit ctrl z to stop viewing the docker-compose log.

Head to 192.168.x.x:8000 (or whatever your server IP is) and hopefully you should see the garish default view of fixmystreet running!

Clone the repository to your local machine

I run windows on my machine, so I installed git on it and use VS code to edit my repository

Install Git

With git you can clone your repository to your main computer, make changes and push them to github ready to pull them to your server.

https://git-scm.com/download/win

I used the 64-bit Git for Windows Setup standalone installer.

Install VSCode

VSCode is a great program for editing code/config files and integrates well with Git to push changes to Github.

https://code.visualstudio.com/download

Clone your repository to your windows machine

Set up a new folder somewhere on your windows machine and then copy the path to that folder.

Open PowerShell.

type "cd " and then right click into powershell which will paste the folder path, should look like this:

cd path/to/new/folder

That will open the folder, and then you're ready to clone the repository again:

git clone https://github.com/yourusername/yourrepositoryname

Load your repository in VSCode

Open up the folder you created in VSCode and things should work automatically, I think you'll get asked to install the source control and Github plugins, go ahead and do both.

You can use the Source control menu to push changes to Github, which will ask you to login to github when you do that the first time.

Removing credentials

Make note of the .gitignore file

This tells git not to push certain files to github - which is important for files which have things like your usernames and passwords in them. The docker-compose.yml and conf/general.yml files are both examples of files that by default have credentials in them you probably don't want to share.

I've used a docker-compose.override.yml file to move credentials I don't want shared on github to a .env file that contains my usernames and passwords.

Using an override file like this means I can leave the "docker-compose.yml" file alone and pull in any changes to that file from the main fixmystreet repository without any conflicts.

Check out https://github.com/ThomasHFWright/fixmystreet/blob/master/docker-compose.override.yml

version: '3'

services:

  fixmystreet:

    environment:

      POSTGRES_PASSWORD: '${POSTGRES_PASSWORD}'

      SUPERUSER_EMAIL: '${SUPERUSER_EMAIL}'

      SUPERUSER_PASSWORD: '${SUPERUSER_PASSWORD}'

    volumes:

      - ${CORE_PATH}/conf/general.yml:/var/www/fixmystreet/fixmystreet/conf/general.yml

  postgres:

    environment:

      POSTGRES_PASSWORD: '${POSTGRES_PASSWORD}'

I've added several environment attributes stored in a separate .env file. These attributes are in the format ${ATTRIBUTE_NAME}

This is also where you can tell docker to read your files in your install folder, rather than the files in the docker image. This is the "volume" mapping above to the general.yml file. I've not mapped any other files yet, but I will have to in order to update to a new cobrand, otherwise the docker containers will have no idea I've added a new cobrand that's not in the standard docker package.

Then my ".env" file looks like this - the .env file is in the same base directory as docker-compose.yml

POSTGRES_PASSWORD='secretpasswordgoeshere'

SUPERUSER_EMAIL='mye...@mydomain.com'

SUPERUSER_PASSWORD='secretpasswordgoeshere'

CORE_PATH='/path/to/my/install/directory'

Since I have my secrets in the .env file, I added that to my .gitignore Installation section, and also commented out the line removing the docker-compose.override.yml. I also added the general.yml file since it contains my email credentials. Like so:

# Installation

/local

/local-*

.prove

.vagrant

.carton

META.yml

Makefile

blib/

inc/

_Inline/

*.pyc

#docker-compose.override.yml

.env

/conf/general.yml

Note that since these files are not tracked by git, they don't get pushed to Github, and so don't get pulled down to your server when you've made changes to them.

This means that changes to .env and /conf/general.yml have to be made directly on the server, or copied across manually.

I set up a Samba service on my server so that I could add my install directory as a network drive in windows and edit the files directly, but it's possible to edit the files in a SSH session with a text editor such as nano (nano needs installing with sudo apt install nano, don't use vim - it's impossible)

To install Samba: https://ubuntu.com/tutorials/install-and-configure-samba#1-overview

Updating config files and making changes

Make changes to your files in VSCode, and then when happy with those, push those to Github.

I'm not going to go into too much detail with git as I'm a novice myself.

It's usually sensible to create a new "branch" when working on a new feature and then "merge" that into the master branch once you're done working on it. Then you pull that "branch" down to your server and test it out.

If you mess up you can just discard the branch and reload from the master branch if all else fails.

You can create a new branch in VSCode by clicking on the "master" text in the Source Control menu on the left side of VSCode, then click "Create new branch" and give your branch a name.

Then you push that branch to github in the VSCode UI.

On your server, git pull to load the new branch

git pull

Then git switch to switch to that branch

git switch branch-name-goes-here

In future, to load updates use git pull.

Once you've loaded new changes, in Portainer go to your fixmystreet stack, select all four containers and then stop them. Wait for all four to stop, then start them again.

If the fixmystreet_fixmystreet_1 container refuses to start, you've probably made some formatting error in a yml file, but you can check the logs in Portainer.

Access fom the internet

I've not mentioned getting anything set up so that fixmystreet can be accessed from the internet.

For this I have the following setup:

Port forwarding

On my router I forward port 80 and 443 to my reverse proxy, I point both these ports to my reverse proxy IP and port.

In a simple case you could just forward both ports to port 8000 for fixmystreet.

DDNS

A DDNS service running in your network will tell your domain host to point your domain at your home IP address. Normally home IP addresses change pretty regularly, so this is required so that it keeps things up to date and correctly pointing to the right place.
I use Cloudflare as my domain host which adds some nice extra features.

My router has the option of setting up the DDNS service, though I run this on my Synology NAS instead.

Reverse Proxy (optional)

A reverse proxy is useful on your home network if you expose multiple services to the internet but don't want to open tons of ports on your router.

E.g. let's say you have a domain myhome.com and you want to allow access to both Portainer and your fixmystreet instance from the internet.
You could allow subdomains to be used and the reverse proxy points to the right place, such as portainer.myhome.com and fixmystreet.myhome.com will both point to your public IP (from your DDNS service) and your reverse proxy service runs within your network to point portainer.myhome.com internally to 192.168.x.x:9000 and fixmystreet.myhome.com to 192.168.x.x:8000.

I don't think many routers have the option of setting up a reverse proxy typically, I run mine on my Synology NAS.

Anyway, there's still lots more for me to run through so don't take this as a complete comprehensive guide, it's been my experience so far - and I haven't even started on the cobrand, getting email working correctly, or the mapping side of things!

[I.TServiceNow]

Hi Did yo have a chance to change any of the cobranding and customisation? I'm still trying to figure out how to customize my site.

[Thomas Wright]

Hey there,

Yeah it took me a bit of time to figure out the cobrand stuff, but I have it mostly working now.

Check out my fork on github to see the details. I'm using docker so this will be different if you've done a direct install.

I needed to set up docker to look at my custom cobrand files

My docker-compose.override.yml here https://github.com/ThomasHFWright/fixmystreet/blob/master/docker-compose.override.yml

has this line under the fixmystreet volumes:

./fixmybsresort-cobrand:/var/www/fixmystreet/cobrand

So that tells docker to look at the fixmybsresort-cobrand directory when fixmystreet is looking for the cobrand directory. You can name this whatever you want, even just "cobrand".

Then here is the contents of that directory for me:

https://github.com/ThomasHFWright/fixmystreet/tree/master/fixmybsresort-cobrand

In the structure there you'll see directorys called "fixmybsresort" - you need to rename these to whatever you called your cobrand in the ALLOWED_COBRANDS section of your   conf/general.yml-docker file (or conf/general.yml file if you overrode the file name in a docker-compose.override.yml file like I did)

After making any cobrand changes, I restart my fixmystreet docker container and that usually updates. Depending one what I've edited I sometimes need to delete the container fixmystreet container and run docker compose up again so that the css creation scripts run.

There is some documentation I found at one time about the minimum files you need to create for the cobrand to work, I can't find that right now but my example is pretty minimal. 

I think the absolute bare minimum is to create a base.scss which does a bunch of imports of other files, and also a  _colours.scss file to override the garish standard base colours:

https://github.com/ThomasHFWright/fixmystreet/blob/master/fixmybsresort-cobrand/web/cobrands/fixmybsresort/base.scss

With the import lines at the top:

@import "../sass/h5bp";
@import "./_colours";
@import "../sass/mixins";
@import "../sass/base";

https://github.com/ThomasHFWright/fixmystreet/blob/master/fixmybsresort-cobrand/web/cobrands/fixmybsresort/_colours.scss

That should get you going with a starting point to override other individual files like the site name and favicon.

Good luck with it!

[Richard Perry]

Hi Thomas, 

Very informative post on docker install and configuration, thanks.
We are just investigating suitability of fixmystreet for our community charity and would really any updates you may have since these posts.
I've looked at your GIT repo and looks like you stopped 5 months ago.

We would rather not tread this path if it is not going to work - what are your thoughts?

Can I discuss direct?

https://qpc.org.uk and then contact form may be best if we can't share email details direct here.

Thanks,

Richard

[Thomas Wright]

Hi Richard,

I haven't given up on the project, but work stuff has taken over my life the last few months.
I still have my staging site up and running https://fixmybsresort.pt/ with a few things I still want to fix such as some links not redirectly correctly which I think is a header issue with my reverse proxy, and I also never got emails sending properly.

None of the issues would suggest to me this won't work with a docker setup, though probably if you do have a dedicated server for this it'd likely be easier to not use docker and just install directly.

Feel free to clone my repo and work from there, it's a bit closer tuned to getting docker setup and you'll have the basic cobrand I set up here and you can remove files that I added to revert to the "default" test cobrand.

https://qpc.org.uk/ isn't working for me , getting a Forbidden error.

Thanks,

Thomas

[Matthew Somerville]

Hi,

Thanks for providing your input, Thomas, good to hear how you've been getting on.

I haven't seen anyone mention the development Docker in this thread so far. For *development*, using Docker, we have a special page about this at https://fixmystreet.org/install/docker-develop/ - I have just run the example `docker/compose-dev up` in a clean installation and it ran to completion and the server was running, so hopefully that is still set up to work okay. This includes a database, CSS compiler, and mailhog (for seeing sent emails). You can edit the files directly and see your changes instantly. This should be easier to develop with than trying to run a production Docker instance.

You can also run the code directly on a Mac or Linux or WSL device, as described at https://fixmystreet.org/install/manual-install/

This is how we at mySociety run https://www.fixmystreet.com for example.

ATB,

Matthew

[Richard Perry]

Hi Matthew,

 

Thanks for your input.

I have just received a message back from Chris Edwards @mySociety, who is kindly arranging a call with one of the fixmystreet dev team to see how you can help.

 

Great to hear that thedocker (dev) is working as intended, and will form the basis for our discussion with the dev when we talk.

We will endeavour to get installed and oeprational later this week (on Azure).

 

My personal preference was the EC2 AMI, https://fixmystreet.org/install/ami/, but this appears to no longer exist in the AWS public AMI registry.

Consquently, and after review of the google groups and questions/response history on various threads, it gave cause for concern on clarity of code offered /assistance and as to whether this would in fact be viable codebase for us to build on.

 

Can you confirm of the AMI is still an option?

If not, may I suggest it may be worth removing from the website as an option?

 

Great guidance on self install, thanks.

 

Richard

--
You received this message because you are subscribed to a topic in the Google Groups "FixMyStreet" group.
To unsubscribe from this topic, visit https://groups.google.com/a/mysociety.org/d/topic/fixmystreet/GE995L1jxcg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fixmystreet...@mysociety.org.
To view this discussion visit https://groups.google.com/a/mysociety.org/d/msgid/fixmystreet/CAKs8PY2JqscTmWxs0_Z%2B%2BKCiswDNQ_zLcAwMPy85T_HGcBLPpg%40mail.gmail.com.

Virus-free.www.avg.com

[Matthew Somerville]

Hi,

> it gave cause for concern on clarity of code offered /assistance and as to whether this would in fact be viable codebase for us to build on.

FixMyStreet is an open source project, created by a small registered charity. It is used for fixmystreet.com, international installs such as FixaMinGata in Sweden, and other uses as well, and is under active development. I cannot say whether that makes it a viable codebase for you to build on, but we and others have managed to do so. However, that doesn't mean anyone will have the resources to support people trying to install the software beyond the documentation that is already provided.

On the AMI - no-one has removed the AMI as far as I'm aware, so I'm not sure what's happened to that. We do link to the public builds repository that was used to generate the AMI, from the page you link to, but this is also quite old, and we haven't had the resources to maintain and keep it up to date; but the codebase itself is up to date and people are running it fine. Happy to accept a pull request updating the AMI page to update about its current availability, and/or next time we do a proper release, which is overdue, we can look and see if it's easy to regenerate at that point.

 

ATB,

Matthew

[Richard Perry]

Thanks Matthew,

100% understand.

I very much admire providing the code as open-source, and thank you for it.

I was not being critical, and apologies if that appeared the case.

I have taken the time to read through most if not all public facing info on fixmystreet (and Pro) and mySociety as well as technical docs and this google group and can see the value.
The benefit of open source is, one hopes, also community collaboration in order to move the codebase forward for the good of all.

I can see others have also asked about the AMI availability earlier this year, with the question remaining unanswered.

It is great you've provided clarity in this chat, but it appears that the most up to date and recommended build to use are manual or docker (dev) and others are no longer available or older / possibly unsupported code bases.

ATB,

Richard

[Paul Bullivant]

Hello,

Continuing on in this thread created by Richard. Here are the steps we have followed:

I created a new Ubuntu VM in Azure running ubuntu server 24.04.

Once I had connected via SSH command line to the server I followed the steps here:
https://fixmystreet.org/install/manual-install/

All was well and I was able to follow steps 1-3 and get the database setup and I was able to log onto the database.

I then ran step 4 i.e. ran script/setup

This failed.

I then copied the example general.yml file to set up my configuration and I edited the database credentials in the general.yml to match the database I had previously installed.

I then ran script/setup again but it is still faling with the following error:

Installing modules using /FixMyStreet/fixmystreet/cpanfile (deployment mode)
! Configure failed for Term-ReadLine-Gnu-1.45. See /root/.cpanm/work/1731661329.84886/build.log for details.
! Installing the dependencies failed: Module 'Term::ReadLine::Gnu' is not installed
! Bailing out the installation for /FixMyStreet/fixmystreet/.
Installing modules failed

On checking the log I have:

Checking if you have Template 2.29 ... Yes (2.29)
==> Found dependencies: Term::ReadLine::Gnu
Searching Term::ReadLine::Gnu on mirror index /FixMyStreet/fixmystreet/local/cache/modules/02packages.details.txt ...
--> Working on Term::ReadLine::Gnu
Fetching http://cpan.metacpan.org/authors/id/H/HA/HAYASHI/Term-ReadLine-Gnu-1.45.tar.gz
-> OK
Unpacking Term-ReadLine-Gnu-1.45.tar.gz
Copying Term-ReadLine-Gnu-1.45.tar.gz to /FixMyStreet/fixmystreet/local/cache/authors/id/H/HA/HAYASHI/Term-ReadLine-Gnu>Entering Term-ReadLine-Gnu-1.45
Checking configure dependencies from META.json
Checking if you have ExtUtils::MakeMaker 7.20 ... Yes (7.70)
Configuring Term-ReadLine-Gnu-1.45
Running Makefile.PL
rlver.c:3:10: fatal error: readline/readline.h: No such file or directory
    3 | #include <readline/readline.h>
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Could not compile rlver.c.

Can anyone help me get past this sticking point.

Thanks

Paul

[Matthew Somerville]

Hi,

Thanks for raising this.

It appears that libreadline-dev was missing from our conf/packages.generic file (but was present in our conf/packages and conf/packages.docker files), which is the file used by the manual install instructions.

That would cause the error you raise, due to that package not being installed.

I've now added it to that file, and included a comment in the documentation so hopefully this won't happen again in future.

If you update to the latest code, and re-run step 2 (to install the system packages), that should then add the missing package, and you can then hopefully proceed.

ATB,
Matthew

[Paul Bullivant]

Thank you.  That worked and I was able to run both

script/setup

and

bin/update-all-reports

I am now trying to set up the webserver.

I installed nginx using:

   sudo apt update

   sudo apt install nginx

   sudo ufw allow 'Nginx Full'    (this to allow 80 and 443 through)

I then copied nginx.conf.example from the fixmystreet/conf folder to /etc/nginx/sites-available

and renamed the conf file to fms.

So my sites-available folder now has:

    default

    fms

I'm not sure what to do next.  I'm aware that both default and fms configurations are listenting on port 80 so that looks like a conflict to me if no host header is specified.

Also, I'm not sure how to run either FMS app or the WebServer site as a daemon service.  The script sysvinit.example referenced in https://fixmystreet.org/install/manual-install/ does not come with easy instructions on how to use it.

Running

     ./sysvinit.example

returns:

     Usage: /etc/init.d/fixmystreet {start|stop|reload|restart|force-reload}

and running

     /etc/init.d/fixmystreet {start|stop|reload|restart|force-reload}

returns

     -bash: /etc/init.d/fixmystreet: No such file or directory

Please advise on how I can run both FMS application and the webserver as daemon services

Thanks

Paul

[Matthew Somerville]

Hi,

There are many ways to install and run different web servers; example configs are provided, but it is assumed that people following the manual steps will know how to run the server/web server they choose. There is also an install script at https://fixmystreet.org/install/install-script/ that sets up a basic server for you. You do not necessarily have to run that script, given what you have already done, but you could look at what the script does to see how it sets up nginx, the application server, and so on.

For example, https://github.com/mysociety/fixmystreet/blob/master/bin/site-specific-install.sh runs 'add_website_to_nginx', a function contained in the https://github.com/mysociety/commonlib/blob/master/bin/install-site.sh file. This appears to do basically what you say you have already done, copying the example file across and tweaking its contents for the particular setup, so that's good :)

 

The setting up of the application server is the 'install_sysvinit_script' function in the same install-site.sh script - we also provide a systemd example, which I think is the more modern way of running services. If setting it up manually, you would copy/tweak the example file into /etc/init.d/ (sysvinit) or /etc/systemd/system/ (systemd) and then run the relevant commands to enable the service there.

All a service basically does is run a script in the background, so if you wanted to just test the service, the example nginx config file expects the application to be running on port 9000, so you could manually run `script/server --port 9000` and see if your nginx then proxies to that. (I think nginx just defaults to the first matching server if there's no Host, but there should always be a Host header.) If e.g. "curl http://localhost:9000/" then worked on the server, but didn't work when visiting the web page, then the issue would be in the nginx config somewhere, and so on.

ATB,

Matthew