indefero/doc/syncmonotone.mdtext

# Plugin SyncMonotone by Thomas Keller (me@thomaskeller.biz)

The SyncMonotone plugin allow the direct creation and synchronisation of
monotone repositories with the InDefero database. It has been built to
work together with monotone's "super server" usher, which is used to control
several repositories at once, acts as proxy and single entrance.

## Prerequisites

* a unixoid operating system
* monotone >= 0.99.1
* for a proxy setup with usher:
    * boost headers (for usher compilation)
    * a current version of usher
    * a daemonizer, like supervise

## Installation of monotone

If you install monotone from a distribution package, ensure you do not
install and / or activate the server component. We just need a plain
client installation which usually consists only of the `mtn` binary and
a few docs.

If you install monotone from source (<http://monotone.ca/downloads.php>),
please follow the `INSTALL` document which comes with the software.
It contains detailed instructions, including all needed dependencies.

## Choose your indefero (IDF) setup

The monotone plugin can be used in several different ways:

1. One database for everything. This is the easiest setup and of possible
   use in case you do not want indefero to manage the access to your project.
   Your `idf.php` should look like this:

            $ cat idf.php
            ...
            $cfg['mtn_path'] = 'mtn';
            $cfg['mtn_opts'] = array('--no-workspace', '--no-standard-rcfiles');
            $cfg['mtn_repositories'] = '/home/monotone/all_projects.mtn';
            $cfg['mtn_remote_url'] = 'ssh://monotone@my.server.com:~all_projects.mtn';
            $cfg['mtn_db_access'] = 'local';
            ...

   Pro:
      * easy to setup and to manage

   Con:
      * you need to give committers SSH access to your machine
      * database lock problem: the database from which
        indefero reads its data might be locked in case a user
        syncs at the very moment via SSH

2. One database for every project. Similar to the above setup, but this
   time you use the '%s' placeholder which is replaced with the short name
   of the indefero project:

            $ cat idf.php
            ...
            $cfg['mtn_path'] = 'mtn';
            $cfg['mtn_opts'] = array('--no-workspace', '--no-standard-rcfiles');
            $cfg['mtn_repositories'] = '/home/monotone/%s.mtn';
            $cfg['mtn_remote_url'] = 'ssh://monotone@my.server.com:~%s.mtn';
            $cfg['mtn_db_access'] = 'local';
            ...

   The same pro's and con's apply. Additionally you have to be careful about
   not giving people physical read/write access of another project's database.

   Furthermore, if you do not want to use `ssh`, but `netsync` transport,
   each project's database must be served over a separate port.

3. One database for every project, all managed with usher. This is the
   recommended setup for a mid-size forge setup. The remaining part of this
   document will describe the process to set this up in detail.

   Pro:
      * access rights can be granted per project and are automatically
        managed by indefero, just like the user's public monotone keys
      * no database locking issues
      * one public server running on the one well-known port

   Con:
      * harder to setup

## Installation and configuration of usher

1. Clone usher's monotone repository:

            $ mtn clone "mtn://monotone.ca?net.venge.monotone.contrib.usher"

2. Compile usher:

            $ autoreconf -i
            $ ./configure && make
            $ sudo make install

   This installs the usher binary in $prefix/bin.

3. Create a new usher user:

            $ adduser --system --disabled-login --home /var/lib/usher usher

4. Create the basic usher setup:

            $ cd /var/lib/usher
            $ mkdir projects logs
            $ cat > usher.conf
             userpass "admin" "<secret-password>"
            adminaddr "127.0.0.1:12345"
               logdir "log"
            ^D
            $ chmod 600 usher.conf

    **ATTENTION:** Do _not_ configure a default monotone server key in `usher.conf`,
    otherwise the individual server key that IDF creates for each project is
    not used and this could cause problems.

    Your indefero www user needs later write access to `usher.conf` and
    `projects/`. There are two ways of setting this up:

    * Make the usher user the web user, for example via Apache's `suexec`.
      This is however a bit clumsy.
    * Preferred: Use Access Control Lists (ACLs), like this:

            #
            # Linux
            #
            $ setfacl -m u:www:rw usher.conf
            $ setfacl -m d:u:www:rwx projects/
            $ setfacl -m d:u:usher:rwx projects/
            #
            # FreeBSD
            #
            $ setfacl -m user:www:rw::allow usher.conf
            $ setfacl -m user:www:rwxp:fd:allow projects/
            $ setfacl -m user:usher:rwxp:fd:allow projects/
            #
            # Mac OS X
            #
            chmod +a '_www allow read,write' usher.conf
            chmod +a '_www allow read,write,delete,add_file,add_subdirectory,file_inherit,directory_inherit' projects/
            chmod +a 'usher allow read,write,delete,add_file,add_subdirectory,file_inherit,directory_inherit' projects/

      In each example's last line, `usher` is the user which is executing
      the usher instance. **It is very important to add this line, otherwise
      usher won't be able to read and write into the initial file system
      setup IDF creates!**

5. Wrap a daemonizer around usher, for example supervise from daemontools
   (<http://cr.yp.to/damontools.html>):

            $ cat > run
            #!/bin/sh
            cd /var/lib/usher
            exec 2>&1
            exec \
                setuidgid usher \
                usher usher.conf
            ^D

   The service can now be started through supervise:

            $ supervise /var/lib/usher

## Configuration of indefero

Based on the above setup, the configuration in `src/IDF/conf/idf.php` should
look like this:

        $ cat idf.php
        ...
        $cfg['mtn_path'] = 'mtn';
        $cfg['mtn_opts'] = array('--no-workspace', '--no-standard-rcfiles');
        $cfg['mtn_repositories'] = '/var/lib/usher/projects/%s/';
        $cfg['mtn_remote_url'] = 'mtn://my.server.com/%s';
        $cfg['mtn_db_access'] = 'remote';
        $cfg['mtn_usher_conf'] = '/var/lib/usher/usher.conf';
        ...

The `%s` placeholders are automatically replaced by the name of the
indefero project. The plugin assumes that every project is separated
by a distinct server name in the monotone URL (hence the use of `/%s`),
so if a user calls

        $ mtn sync mtn://my.server.com/project1

then the database / repository of the indefero `project1` is used.
Note that 'mtn_remote_url' is also used as internal URI to query the data
for indefero's source view, so it *must* be a valid host!

Usher also allows the identification of a project repository by hostname,
which would allow an URL template like `mtn://%s.my.server.com`, however
the plugin does not write out the configuration which is needed for this
yet.

For even more advanced setups, usher can also be used to forward sync
requests to other remote servers for load balancing, please consult the
README file for more information.

## Security and remote access

Indefero distinguishs between public and private projects and so does
the monotone plugin.

Public projects can be pulled by everybody and pushed by team members
or additional invited people. Remote command execution is enabled, but
only for read-only commands.

Remote commands can be helpful for a user or a 3rd party tool (like
[mtn-browse](http://mtn-browse.sourceforge.net) or
[guitone](http://guitone.thomaskeller.biz)) to browse the database
contents remotely without having to pull everything in first instance.

Private projects on the other hand can only be synced by team members
or additional invited people. Remote command execution is disabled
by default. If you want to enable that, simply put the keys of the users
you want to give access to in your project's `remote-automate-permissions`
file. In the future this plugin might handle this file just as it handles
`read-permissions` and `write-permissions`.

## Notifications

If you have successfully set up your monotone instance, you probably want
to notify 3rd party systems for incoming changes or simply mirror them
somewhere else for backup purposes.  The monotone source tree already comes
with [many example scripts and hooks](http://code.monotone.ca/p/monotone/source/tree/h:net.venge.monotone/contrib)
which serve these purposes, after only little additional configuration.

The usher/indefero-controlled setup automatically looks for *.lua files
in a directory called `hooks.d` right under the project's base directory
(configured via $cfg['mtn_repositories']) and this is the ideal place to
put or link these additional lua sources.

## Custom project configurations and templates

If a new project is created in IDF, the SyncMonotone plugin creates a new
configuration tree for the project into the project's configuration directory,
determined by `$cfg['mtn_repositories']`. IDF ships with the minimum set of
files for this configuration tree and sets up everything automatically for you.

Even more, most of the configuration files from the newly created tree are only
symlinked to the original configuration directory which is configurable via
`$cfg['mtn_confdir']` and defaults to `src/IDF/Plugin/SyncMonotone/`. This has
the advantage that your standard IDF setup automatically receives updates to
existing (symlinked) configuration files as soon as you update to a newer
version.

You could, however, also choose to place the directory tree somewhere else
and adapt the contents of the individual files yourself, so these changes get
automatically applied to all new projects you create. You could even go so far
and add new files to the tree and let them be processed automatically just
as the basic files! All you need to do is to copy your files and / or directories
underknees your `$cfg['mtn_confdir']` and add their relative paths to
`$cfg['mtn_confdir_extra']`.

By convention, all entries which end with a slash are considered directories,
so mkdir(1) is issued for these entries, all files which do not end up with
".in" are considered to be static script files which are just symlinked from
the basic configuration dir and all entries ending on ".in" are considered
configuration files or templates, which are copied over to the project's
configuration tree and which get some basic project-specific values replaced.

The following placeholders are currently recognized and replaced for these files:

 * %%PROJECT%% - the name of the created project
 * %%MTNPOSTPUSH%% - the absolute path to the `mtn-post-push` script
 * %%MTNCLIENTKEY%% - the public key hash of the key which is used by IDF
   to authenticate remote stdio access

Thats it - I hope you find it useful :)

## Q&A

### After I created a new project, IDF throws an exception and tells me that it couldn't save the membership data with a cryptic error message. Whats wrong?

Multiple issues could cause that. If you've set up usher, make sure the usher
can fork your database at all and look out for specific errors in the log file
of your project. If you stumble upon permission issues, ensure that the user
who runs the usher can access all files in your project's configuration directory,
including symlinked files.

### I pushed a branch to my server, but it does not show up in IDF. Whats wrong?

Check if the heads of your branch are not suspended, i.e. do not carry a
`suspend` certificate.  This usually hides the branch and all of its history
from monotone's eyes and therefor also from indefero.  You can either choose
to "unsuspend" the branch simply by committing and pushing another head or
by letting monotone ignore all suspend certs.  For the latter, its usually
enough to add `--ignore-suspend-certs` to the list of options in `$cfg['mtn_opts']`.

### I want to display another default branch when I click the "Source" tab. How can I do that?

Let the forge admin know the new master branch for your project.  He is able
to change that quickly.  Depending on the backend / server setup this might
also require some changes in the usher configuration, but only if usher
recognizes and proxies your database on a branch name level.