Firefox Bookmark Syncing

As you may already have heard, the bookmark syncing service Xmarks is shutting down on 1st of May 2018. This is really a pity since the service supported multiple operating system platforms as well as different browsers via corresponding extensions, plus there was not any costs involved.

After checking different (fee-based) alternatives, I examined again the built-in syncing services that are included in today's browsers: However, since I do not want to store my private data at Google -- for reasons --, I checked the syncing service that is provided by the new Firefox Quantum. Although, Mozilla seems to be slightly less evil than Google, I still had a bad feeling about storing my bookmarks at their server, so I did some further research. After a while, I stumbled over the SyncServer, which is basically the backend of the syncing service used in Firefox. I decided to setup a custom syncing backend that can be used with Firefox. The bad news about this approach is that there is no Debian package available and Mozilla's documentation leaves room for improvement, so expect a path of trial and tribulation ;-)

Before discussing the installation of the private bookmark syncing service, a short warning: though, the syncing server itself can be installed on your private server, the general user authentication, that is required to use the syncing service, is another matter. In fact there seems to be ways to install also a custom authentication server, but this is even less documented, so I will skip this part and simply rely on Mozilla's service for authentication, which is some sort of compromise, but okay with me.

Installation of Mozilla's SyncServer

There are a couple of steps necessary to install the syncing service: First, some packages from the Debian repositories must be installed to be able to obtain the server's source code and to build it.

apt-get install python-dev git-core python-virtualenv g++

Then, the source code can be cloned from the corresponding github repository to the /opt directory and be built:

cd /opt
git clone https://github.com/mozilla-services/syncserver
cd syncserver
make build

A separate system user is created that should own the service in the following:

useradd -s /usr/sbin/nologin -r -M syncserver
chown syncserver:www-data -R /opt/syncserver
chmod 750 -R /opt/syncserver

To check the successful installation of the syncing server do:

make test

Configuration

The main idea of the configuration is to run the syncing server via the provided gunicorn server and prefix it with an nginx server to obtain a better stability and performance. In the end, the nginx server will provide the syncing service on port 8080, using https to secure the connection.

The main configuration of the syncing server is done via the /opt/syncserver/syncserver.ini file. Adapt the configuration, for example, as follows:

[server:main]
use = egg:gunicorn
host = 127.0.0.1
port = 8081
workers = 2
timeout = 60

Also set the public_url to https://sync.example.com:8080/ and sqluri to sqlite:////opt/syncserver/syncserver.db. It is also recommended to set a unique secret to secure the server. At least on first start, also the allow_new_users variable must be set to true, otherwise it will not be possible to setup a new account on the server.

For a short test of the new configuration, the server can be simply launched as follows:

make serve

Systemd

Since the syncing server should run automatically a systemd service is configured. Create the /etc/systemd/system/syncserver.service file with the following content:

[Unit]
Description=syncserver daemon
Requires=gunicorn.socket
After=network.target

[Service]
PIDFile=/run/gunicorn/pid
User=syncserver
Group=www-data
RuntimeDirectory=gunicorn
WorkingDirectory=/opt/syncserver
ExecStart=/opt/syncserver/local/bin/gunicorn --pid /run/gunicorn/pid   \
          --bind unix:/run/gunicorn/socket --paste syncserver.ini
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

The service will run using the previously created user.

Additionally, prepare the corresponding socket file /etc/systemd/system/gunicorn.socket:

[Unit]
Description=gunicorn socket

[Socket]
ListenStream=/run/gunicorn/socket

[Install]
WantedBy=sockets.target

Besides, an allowance for the creation of the corresponding run directory must be created in /etc/tmpfiles.d/gunicorn.conf, containing:

d /run/gunicorn 0755 syncserver www-data -

Ensure that gunicorn subdirectory is existing:

mkdir /run/gunicorn
chown syncserver:www-data -R /run/gunicorn
chmod 750 -R /run/gunicorn

Finally, the new systemd gunicorn socket can be enabled and started:

systemctl enable gunicorn.socket
systemctl start gunicorn.socket

Gunicorn itself should automatically start after requesting the socket:

curl --unix-socket /run/gunicorn/socket http

Nginx

To delegate the gunicorn-based syncing server to the outside world nginx is used, which simply can be installed from the Debian repositories:

apt install nginx

Then, create a nginx configuration file /etc/nginx/sites-available/syncserver with the following content:

server {
    listen 8080 ssl;
    server_name sync.example.com;

    # use letsencrypt to get ssl certificates
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_redirect off;
        proxy_read_timeout 120;
        proxy_connect_timeout 10;
        proxy_pass http://unix:/run/gunicorn/socket;
    }
}

The nginx configuration passes the gunicorn socket to the external port 8080. The configuration uses a free Let's Encrypt SSL certificate to protect the communication via https.

Link the created syncing server nginx page to the corresponding directory to enable the site (and disable the default page, if desired).

ln -s /etc/nginx/sites-available/syncserver /etc/nginx/sites-enabled/syncserver
# rm /etc/nginx/sites-enabled/default

Enable and start the nginx server:

systemctl enable nginx
systemctl start nginx

The syncing server should now be running and can be tested by simply accessing the corresponding webpage:

curl https://sync.example.com:8080/token/1.0/sync/1.5

Configure Firefox

Finally, the Firefox configuration can be adapted to use the private syncing server. To this end, open the configuration page via about:config and search for tokenserver. Change the value of identity.sync.tokenserver.uri to https://sync.example.com:8080/token/1.0/sync/1.5. That's basically it!

Now, you can click the upper right main menu and select Sign in to Sync. As stated above, in the current configuration you need to have a Mozilla account for authentication i.e. if you do not have such account yet, create it now on Firefox Account. After the confirmation of the account you can log in using the chosen credentials and start the syncing process. Et voilà -- your bookmarks should be synced to your private syncing server.

Debugging

When the the syncing is not working as expected, you can access the Firefox logs via about:sync-log. Happy debugging ;-)

Some Last Words

Do not forget to set allow_new_users to false, if you do not want to store other people's bookmarks. Additionally, do not forget to adapt your firewall to allow access to the used port.