Commit 39086203 authored by Lars Kruse's avatar Lars Kruse
Browse files

feat(matrix_chat): initial integration

The integration of a Matrix chat is supposed to provide users with an
instant chat communication tool in addition to the currently supported
mail/forum based communication.

See: #696
parent d1bc0460
......@@ -20,6 +20,7 @@ Documentation for users can be found on `stadtgestalten <https://stadtgestalten.
configuration/assets
database/index
mail_setup
matrix_chat
authentication/index
upgrading
releases/index
# Matrix Chat Integration
[Matrix](https://matrix.org/) is a chat communication protocol.
It can be used for real-time communication (with end-to-end encryption) between grouprise users
and external Matrix users.
Matrix supports [federation](https://en.wikipedia.org/wiki/Federation_(information_technology)).
Thus grouprise users may use either their existing Matrix account (hosted by an external provider)
or they can use a new Matrix account, which is based on their grouprise account.
## Usage
The Matrix integration of grouprise provides the following features (compared with a separate
Matrix homeserver):
* the grouprise user account (name and password) can be used for logging into the Matrix homeserver
(there is no need for a separate Matrix account)
* a local instance of the [Element](https://github.com/vector-im/element-web) web client is
provided
* a public and a private Matrix room is automatically created for each grouprise group
* all members of a grouprise group are automatically invited to the group's rooms
* new content (articles, events, discussions, ...) are mentioned (with a link) in their
corresponding Matrix rooms (internal messages are announced in the private room, only)
* the group overview page in grouprise provides links to the group's Matrix rooms
Additionally the usual benefits of the Matrix system are available:
* users from external homeservers can join the public rooms and can be invited to the private rooms
* new rooms and communities can be created freely on the homeserver
* direct communication between Matrix users (local and external) is possible
* any [Matrix client](https://matrix.org/clients/) (e.g. on mobile devices) can be used
The web client is accessible via `https://YOUR_DOMAIN/stadt/chat`.
## Setup
The following steps are based on the [deb-based deployment](deployment/deb).
Warning: currently (January 2021) the required Debian packages are not part of a stable Debian
release (as of now: Debian Buster).
Thus you may need to wait for the next Debian release (*Bullseye*), if you prefer a setup of
packages with proper security support.
1. add the `buster-backports` repository to your apt sources file
(for the `matrix-synapse` package)
1. install `python3-django-cas-server` from the Debian *testing* repository
The required version of `python3-django-cas-server` will be part of the *Debian Bullseye* in 2021.
Hint: do not forget to remove the *testing* repository from your sources list afterwards.
1. install the matrix integration package for grouprise: `apt install grouprise-matrix`
1. answer the configuration questions during package installation:
* matrix-synapse:
* server: the name of your grouprise domain (e.g. `example.org`)
* element-web-installer:
* webserver configuration: *none* (configuration is done by `grouprise-matrix`)
* default matrix server: the name of your grouprise domain (e.g. `example.org`)
* grouprise-matrix:
* webserver configuration: *nginx*
1. create the postgresql database connection for matrix-synapse:
```
CREATE USER grouprise_matrix WITH password 'YOUR_SECRET_RANDOM_PASSWORD';
CREATE DATABASE grouprise_matrix ENCODING 'UTF8' LC_COLLATE='C' LC_CTYPE='C' template=template0 OWNER grouprise_matrix;
```
1. configure this database connection in `/etc/matrix-synapse/conf.d/grouprise-matrix.yaml`
1. start matrix-synapse: `service matrix-synapse start`
1. generate an administrative access token for matrix-synapse to be used by grouprise: `GROUPRISE_USER=root grouprisectl matrix_register_grouprise_bot`. The resulting access token needs to be configured in `/etc/grouprise/settings.py`:
```python
GROUPRISE["MATRIX_CHAT"] = {
'ENABLED': True,
'BOT_USERNAME': 'grouprise-bot',
'BOT_ACCESS_TOKEN': '_YOUR_BOT_ACCESS_TOKEN_',
}
```
1. Run `grouprisectl matrix_chat_manage create-rooms` and `grouprisectl matrix_chat_manage invite-room-members` in order to populate the Matrix rooms for all groups.
In order to apply all settings properly, it is (for now) necessary to go run the configuration of
the `grouprise-matrix` package manually again:
```shell
dpkg-reconfigure --unseen-only grouprise-matrix
```
## Configuration settings
The following configuration settings are available below the `GROUPRISE["MATRIX_CHAT"]` dictionary
in your grouprise settings file (e.g. `/etc/grouprise/settings.py`):
* `DOMAIN`: The Matrix domain to be used. Defaults to the grourise domain.
* `BOT_USERNAME`: The local name of the Matrix bot used by grouprise. Defaults to `grouprise-bot`.
* `BOT_ACCESS_TOKEN`: The access token of the Matrix bot used by grouprise. To be generated via
`GROUPRISE_USER=root grouprisectl matrix_register_grouprise_bot`. In case of manual account
creation, it can be retrieved from the Matrix grouprise database via
`select token from access_tokens where user_id='@USERNAME:MATRIX_DOMAIN';`.
* `ADMIN_API_URL`: An API URL of the Matrix instance, which accepts Synapse admin requests
(e.g. `/_synapse/admin`). Defaults to `http://localhost:8008`.
import sys
from django.contrib.sites.models import Site
from django.core.management.base import BaseCommand
import cas_server.models as models
class Command(BaseCommand):
args = ""
help = "Enable or disable CAS authentication for the attached matrix server"
def add_arguments(self, parser):
default_app_url = "https://{}:8448/".format(Site.objects.get_current().domain)
parser.add_argument("action", choices=("add", "remove"))
parser.add_argument("label", type=str)
parser.add_argument("--app-url", type=str, default=default_app_url)
def get_cas_service_pattern(self, label):
patterns = models.ServicePattern.objects.filter(name=label)
try:
return patterns[0]
except IndexError:
return None
def handle(self, *args, **options):
action = options["action"]
label = options["label"]
app_url = options["app_url"]
if action == "add":
existing = self.get_cas_service_pattern(label)
if existing is None:
new_obj = models.ServicePattern(pattern=app_url, name=label)
new_obj.save()
self.stdout.write(
self.style.SUCCESS("Added service pattern for authentication")
)
elif existing.pattern == app_url:
self.stdout.write(
self.style.NOTICE("Keeping existing service pattern unchanged")
)
else:
existing.pattern = app_url
existing.save()
self.stdout.write(
self.style.SUCCESS("Modified existing service pattern")
)
elif action == "remove":
existing = self.get_cas_service_pattern(label)
if existing is None:
self.stdout.write(
self.style.NOTICE("No matching service pattern found")
)
else:
existing.delete()
self.stdout.write(
self.style.SUCCESS("Removed existing service pattern")
)
else:
self.stderr.write(
self.style.ERROR("Invalid action requested: {}".format(action))
)
sys.exit(1)
import asyncio
import sys
from django.core.management.base import BaseCommand
from grouprise.features.groups.models import Group
from grouprise.features.matrix_chat.matrix_bot import MatrixBot
class Command(BaseCommand):
args = ""
help = "Synchronize rooms and other state from grouprise to matrix"
def add_arguments(self, parser):
parser.add_argument("action", choices=("create-rooms", "invite-room-members"))
def handle(self, *args, **options):
action = options["action"]
if action == "create-rooms":
async def create_group_rooms(groups):
async with MatrixBot() as bot:
for group in groups:
async for updated_room in bot.synchronize_rooms_of_group(group):
self.stdout.write(
self.style.NOTICE(
f"Created or updated room '{updated_room}'"
)
)
asyncio.run(create_group_rooms(list(Group.objects.all())))
elif action == "invite-room-members":
async def invite_to_group_rooms(groups):
async with MatrixBot() as bot:
for group in groups:
async for (
room,
gestalt,
) in bot.send_invitations_to_group_members(group):
self.stdout.write(
self.style.NOTICE(
f"Invited '{gestalt}' to room '{room}'"
)
)
asyncio.run(invite_to_group_rooms(list(Group.objects.all())))
else:
self.stderr.write(
self.style.ERROR("Invalid action requested: {}".format(action))
)
sys.exit(1)
import json
import urllib.request
from grouprise.features.matrix_chat.settings import MATRIX_SETTINGS
class MatrixAdmin:
"""execute administrative requests (not provided by the "nio" package)
The "nio" package covers all aspects of client-server requests.
But some tasks may require access to the admin API of an matrix-synapse server (usually below
/_synapse/admin/).
"""
def __init__(self, auth_token, api_url=MATRIX_SETTINGS.ADMIN_API_URL):
self.api_url = api_url
self.auth_token = auth_token
def _submit_http_request(self, path, method, data=None):
if method in {"PUT", "POST"}:
if data is not None:
payload = json.dumps(data).encode()
else:
payload = b"{}"
else:
payload = None
url = "{}/{}".format(self.api_url.rstrip("/"), path.lstrip("/"))
request = urllib.request.Request(
url,
method=method,
headers={"Authorization": "Bearer {}".format(self.auth_token)},
data=payload,
)
with urllib.request.urlopen(request) as req:
response = req.read()
return json.loads(response)
def request_get(self, path):
return self._submit_http_request(path, method="GET")
def request_put(self, path, data=None):
return self._submit_http_request(path, method="PUT", data=data)
def request_post(self, path, data=None):
return self._submit_http_request(path, method="POST", data=data)
import logging
import markdown
import nio
from grouprise.core.templatetags.defaultfilters import full_url
from .models import (
MatrixChatGestaltSettings,
MatrixChatGroupRoom,
MatrixChatGroupRoomInvitations,
)
from .settings import MATRIX_SETTINGS
logger = logging.getLogger(__name__)
class MatrixError(Exception):
""" an error occurred while communicating with the matrix server """
class MatrixBot:
def __init__(self):
matrix_url = f"https://{MATRIX_SETTINGS.DOMAIN}"
bot_matrix_id = f"@{MATRIX_SETTINGS.BOT_USERNAME}:{MATRIX_SETTINGS.DOMAIN}"
logger.info(f"Connecting to {matrix_url} as '{MATRIX_SETTINGS.BOT_USERNAME}'")
self.client = nio.AsyncClient(matrix_url, bot_matrix_id)
# maybe we should use "self.client.login()" instead?
self.client.access_token = MATRIX_SETTINGS.BOT_ACCESS_TOKEN
async def sync(self, set_presence="online"):
sync_result = await self.client.sync(set_presence=set_presence)
if isinstance(sync_result, nio.responses.SyncError):
raise MatrixError(
f"Failed to synchronize state with homeserver: {sync_result}"
)
if not self.client.logged_in:
raise MatrixError(
"Failed to login. There is a problem with the matrix server or the configured "
"account settings are invalid."
)
async def __aenter__(self):
await self.sync()
return self
async def __aexit__(self, exc_type, exc_value, traceback):
await self.client.close()
return False
async def send_text(self, room, body, msgtype="m.notice", parser="markdown"):
msg = {"body": body, "msgtype": msgtype}
if parser == "markdown":
msg["format"] = "org.matrix.custom.html"
msg["formatted_body"] = markdown.markdown(body)
try:
response = await self.client.room_send(room.room_id, "m.room.message", msg)
except nio.exceptions.ProtocolError as exc:
raise MatrixError(f"Failed to send message: {exc}")
else:
if not isinstance(response, nio.responses.RoomSendResponse):
raise MatrixError(f"Failed to send message: {response}")
async def synchronize_rooms_of_group(self, group):
""" create rooms for the group and synchronize their avatar with the grouprise avatar """
for is_private in (False, True):
try:
room, created = await self._get_or_create_room(group, is_private)
except MatrixError as exc:
logger.error(f"Failed to synchronize room: {exc}")
continue
avatar_changed = False
if group.avatar:
grouprise_avatar_url = full_url(group.avatar_64.url)
response = await self.client.room_get_state_event(
room.room_id, "m.room.avatar"
)
if isinstance(response, nio.responses.RoomGetStateEventResponse):
current_matrix_avatar_url = response.content.get("url")
elif isinstance(response, nio.responses.RoomGetStateEventError) and (
response.status_code == "M_NOT_FOUND"
):
current_matrix_avatar_url = None
else:
logger.warning(
f"Failed to retrieve current avatar of group '{group}': {response}"
)
continue
if current_matrix_avatar_url != grouprise_avatar_url:
response = await self.client.room_put_state(
room.room_id, "m.room.avatar", {"url": grouprise_avatar_url}
)
if isinstance(response, nio.responses.RoomPutStateResponse):
avatar_changed = True
else:
logger.warning(
f"Failed to update avatar for room '{room}': {response}"
)
if created or avatar_changed:
yield room
async def _get_or_create_room(self, group, is_private):
try:
room = MatrixChatGroupRoom.objects.get(group=group, is_private=is_private)
return room, False
except MatrixChatGroupRoom.DoesNotExist:
pass
# the room does not exist: we need to create it
group_url = full_url(group.get_absolute_url())
if is_private:
suffix = "-private"
room_title = "{} (private)".format(group.name)
room_description = "{} - members only".format(group_url)
else:
suffix = ""
room_title = group.name
room_description = group_url
group_name_local = (group.slug or group.name) + suffix
preset = (
nio.api.RoomPreset.private_chat
if is_private
else nio.api.RoomPreset.public_chat
)
try:
response = await self.client.room_create(
name=room_title, topic=room_description, preset=preset
)
except nio.exceptions.ProtocolError as exc:
raise MatrixError(f"Failed to create room '{group_name_local}': {exc}")
if not isinstance(response, nio.responses.RoomCreateResponse):
raise MatrixError(
f"Create room requested for '{group_name_local}' was rejected: {response}"
)
# store the room
room = MatrixChatGroupRoom.objects.create(
group=group, is_private=is_private, room_id=response.room_id
)
# try to attach the canonical alias (optional)
room_alias = f"#{group_name_local}:{MATRIX_SETTINGS.DOMAIN}"
try:
response = await self.client.room_put_state(
room.room_id,
"m.room.canonical_alias",
{"alias": room_alias},
)
except nio.exceptions.ProtocolError as exc:
logger.warning(f"Failed to assign alias ({room_alias}) to room: {exc}")
else:
if not isinstance(response, nio.responses.RoomPutStateResponse):
logger.warning(
f"Refused to assign alias ({room_alias}) to room: {response}"
)
# respond with success, even though the alias assignment may have failed
return room, True
async def send_invitations_to_group_members(self, group):
for room in MatrixChatGroupRoom.objects.filter(group=group):
invited_members = [
invite.gestalt.id
for invite in room.invitations.distinct("gestalt").select_related(
"gestalt"
)
]
for gestalt in group.members.exclude(id__in=invited_members):
gestalt_matrix_id = MatrixChatGestaltSettings.get_matrix_id(gestalt)
try:
result = await self.client.room_invite(
room.room_id, gestalt_matrix_id
)
except nio.exceptions.ProtocolError as exc:
logger.warning(f"Failed to invite {gestalt} into {room}: {exc}")
else:
if isinstance(result, nio.responses.RoomInviteResponse) or (
isinstance(result, nio.responses.RoomInviteError)
and (result.status_code == "M_FORBIDDEN")
):
# "forbidden" is used for "is already in the room" - we remember this
MatrixChatGroupRoomInvitations.objects.create(
room=room, gestalt=gestalt
)
yield room, gestalt
else:
logger.warning(
f"Invite request for {gestalt} into {room} was rejected: {result}"
)
# Generated by Django 2.2.17 on 2021-01-17 00:41
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
initial = True
dependencies = [
("gestalten", "0014_auto_20190815_0926"),
("groups", "0024_group_tags"),
]
operations = [
migrations.CreateModel(
name="MatrixChatGroupRoom",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
("room_id", models.CharField(max_length=256)),
("is_private", models.BooleanField()),
(
"group",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
related_name="matrix_rooms",
to="groups.Group",
),
),
],
),
migrations.CreateModel(
name="MatrixChatGestaltSettings",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
("matrix_id_override", models.CharField(blank=True, max_length=256)),
(
"gestalt",
models.OneToOneField(
on_delete=django.db.models.deletion.CASCADE,
related_name="matrix_chat_settings",
to="gestalten.Gestalt",
),
),
],
),
migrations.CreateModel(
name="MatrixChatGroupRoomInvitations",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
("time_invited", models.DateTimeField(auto_now=True)),
(
"gestalt",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
related_name="+",
to="gestalten.Gestalt",
),
),
(
"room",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
related_name="invitations",
to="matrix_chat.MatrixChatGroupRoom",
),
),
],
),
]
from django.db import models
from django.core.exceptions import ObjectDoesNotExist
from grouprise.features.matrix_chat.settings import MATRIX_SETTINGS
class MatrixChatGroupRoom(models.Model):
group = models.ForeignKey(
"groups.Group", related_name="matrix_rooms", on_delete=models.CASCADE
)
room_id = models.CharField(max_length=256)
is_private = models.BooleanField()
def get_client_url(self):
return f"/stadt/chat/#/room/{self.room_id}"
def __str__(self):
if self.is_private:
return "MatrixChatGroupRoom<{}, private, {}>".format(
self.group.name, self.room_id
)
else:
return "MatrixChatGroupRoom<{}, public, {}>".format(
self.group.name, self.room_id
)
class MatrixChatGroupRoomInvitations(models.Model):
room = models.ForeignKey(