From 5d5489b766ffa41012b96cff2b47c72f0245f3c2 Mon Sep 17 00:00:00 2001 From: Half-Shot Date: Mon, 9 Sep 2019 12:31:35 +0100 Subject: [PATCH 1/8] Draft development guide --- docs/development_guide.md | 106 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 106 insertions(+) create mode 100644 docs/development_guide.md diff --git a/docs/development_guide.md b/docs/development_guide.md new file mode 100644 index 000000000..c4d4e7fc1 --- /dev/null +++ b/docs/development_guide.md @@ -0,0 +1,106 @@ +Development Guide +----------------- + +This guide is for those that want to help hack on the Slack bridge, and need some instruction on how +to set it up. This is **not** a guide for setting up an instance for production usage. + +For information on how to contribute issues and or pull requests, please read [CONTRIBUTING](../CONTRIBUTING.md). + +## Setting up your environment + +This section explains how to setup Synapse, Riot and the Slack bridge for local development. + +### Prereqs + +Ensure at the minimum you have installed: + +- NodeJS ( [nvm](https://github.com/nvm-sh/nvm) is a good tool for this) +- Docker (optional, but it will make your life easier) + +For the sake of making this easier to follow, we will create a directory in the home directory +called `slack-bridge-env`: + +```bash +mkdir ~/slack-bridge-env +cd ~/slack-bridge-env +``` + +### Setting up Synapse + +Largely to setup Synapse, you can follow https://hub.docker.com/r/matrixdotorg/synapse and just ensure your data directory points to `~/slack-bridge-env/synapse`. +You will want to make sure your server name is something that routes to your local box. I tend to use the devbox hostname, but `localhost` is also sufficent. +You should enable registration in the `homeserver.yaml` to create a testing user. +You will need to add an appservice registration file in the future, but it is not imprtant for now. + +### Setting up Riot + +Setting up Riot is also quite straightforward: + +Create a new config called `~/slack-bridge-env/riot-config.json` with the contents needed to set defaults to your local homeserver. + +As an example: + +```json +{ + "default_server_config": { + "m.homeserver": { + "base_url": "http://localhost:8008", + "server_name": "localhost" + } + }, + "disable_custom_urls": false, + "disable_guests": false, + "disable_login_language_selector": false, + "disable_3pid_login": false, + "brand": "Riot", + "defaultCountryCode": "GB", + "showLabsSettings": false, + "default_federate": true, + "default_theme": "light" +} +``` + +Finally, run `docker run -v /home/will/git/scalar-env/riot-config.json:/app/config.json -p 8080:80 vectorim/riot-web` to start your Riot instance. You should be able to register a new user on your local synapse instance through Riot. + +### Setting up the bridge postgres + +Setting up the postgres instance for the bridge is also quite easy. + +```bash +docker run -d --name slackpg -p 59999:5432 -e POSTGRES_PASSWORD=pass postgres +``` + +You should also create a new database in preparation for using the bridge + +```bash +sudo apt install postgresql-client # Ensure you have psql installed +psql -h localhost -U postgres -W postgres +``` + +And then on the postgres shell + +```sql +CREATE DATABASE slack +``` + +### Setting up the bridge + +Clone the bridge repo to a place of your choice, again for simplicity we will +use the env directory. Make sure you clone your **fork** and not the upstream repo. + +```bash +git clone git@github.com:your-github-username/matrix-appservice-slack.git +cd matrix-appservice-slack +git remote add matrix-org git@github.com:matrix-org/matrix-appservice-slack.git +git checkout matrix-org/develop # Always base your changes off matrix-org/develop +npm i +npm run build +``` + +The steps above should make sure you are running the latest development version +and have built the typescript. To ensure that it's all working, you can run +`npm test` which will run the unit and integration tests. + +You can follow the instructions in the [README](../README.md) to generate and update the registration file, as well as creating a testing Slack app and workspace. You should enable RTM support in the config, as Slack will not be able to push events to your local bridge. + +Make sure that you copy the generated registration file to `~/slack-bridge-env/synapse` and add an entry for it in the `homeserver.yaml` before starting the synapse container. \ No newline at end of file From 38c888c14a638ac8d7e652b8a0cb4cf347bc707a Mon Sep 17 00:00:00 2001 From: Half-Shot Date: Mon, 9 Sep 2019 13:15:13 +0100 Subject: [PATCH 2/8] Add section about making changes --- docs/development_guide.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/docs/development_guide.md b/docs/development_guide.md index c4d4e7fc1..ee9534c4c 100644 --- a/docs/development_guide.md +++ b/docs/development_guide.md @@ -103,4 +103,19 @@ and have built the typescript. To ensure that it's all working, you can run You can follow the instructions in the [README](../README.md) to generate and update the registration file, as well as creating a testing Slack app and workspace. You should enable RTM support in the config, as Slack will not be able to push events to your local bridge. -Make sure that you copy the generated registration file to `~/slack-bridge-env/synapse` and add an entry for it in the `homeserver.yaml` before starting the synapse container. \ No newline at end of file +Make sure that you copy the generated registration file to `~/slack-bridge-env/synapse` and add an entry for it in the `homeserver.yaml` before starting the synapse container. + +## Making changes + +Whenever you want to make changes to the codebase, you must: + +```bash +git fetch matrix-org +git checkout matrix-org/develop +git checkout -b yourfeaturname +npm i +``` + +You should always work within the `src` directory. It is helpful to have a code editor setup with linting enabled so you can see mistakes as you work. Always remember to run `npm run build` before commiting or testing so that you know the latest changes work. + +Before commiting your work, ensure that the tests pass locally with `npm test`. If you are making changes to packages, ensure that `package-lock.json` is included in the commit. From 53e9df3286297a0f26e85ca5a5c97046b6c1941e Mon Sep 17 00:00:00 2001 From: Half-Shot Date: Mon, 9 Sep 2019 16:15:35 +0100 Subject: [PATCH 3/8] Commit newsfile and fix verification --- .buildkite/pipeline.yml | 2 +- changelog.d/215.doc | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) create mode 100644 changelog.d/215.doc diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml index 9b2f3fa92..598fb94c4 100644 --- a/.buildkite/pipeline.yml +++ b/.buildkite/pipeline.yml @@ -37,7 +37,7 @@ steps: branches: "!master !develop !release-*" command: - "python3 -m pip install towncrier" - - "python3 -m towncrier.check" + - "python3 -m towncrier.check --compare-with=origin/develop" plugins: - docker#v3.0.1: image: "python:3.6" diff --git a/changelog.d/215.doc b/changelog.d/215.doc new file mode 100644 index 000000000..95d1d2fae --- /dev/null +++ b/changelog.d/215.doc @@ -0,0 +1 @@ +Added docs/development_guide.md to aid new users hacking on the bridge. \ No newline at end of file From bfddaa4195fa14d12005693a1d9ce424bc8edab5 Mon Sep 17 00:00:00 2001 From: Half-Shot Date: Thu, 12 Sep 2019 11:44:58 +0100 Subject: [PATCH 4/8] Reworded guide, improved line width, ruined diff --- docs/development_guide.md | 103 ++++++++++++++++++++++++++++---------- 1 file changed, 76 insertions(+), 27 deletions(-) diff --git a/docs/development_guide.md b/docs/development_guide.md index ee9534c4c..a34149c42 100644 --- a/docs/development_guide.md +++ b/docs/development_guide.md @@ -1,42 +1,84 @@ Development Guide ----------------- -This guide is for those that want to help hack on the Slack bridge, and need some instruction on how -to set it up. This is **not** a guide for setting up an instance for production usage. +This guide is for those that want to help hack on the Slack bridge, and +need some instruction on how to set it up. This is **not** a guide for +setting up an instance for production usage. -For information on how to contribute issues and or pull requests, please read [CONTRIBUTING](../CONTRIBUTING.md). +For information on how to contribute issues and or pull requests, please +read [CONTRIBUTING](../CONTRIBUTING.md). ## Setting up your environment -This section explains how to setup Synapse, Riot and the Slack bridge for local development. +This section explains how to setup Synapse, Riot and the Slack bridge +for local development. -### Prereqs +### 0. Prerequisites Ensure at the minimum you have installed: -- NodeJS ( [nvm](https://github.com/nvm-sh/nvm) is a good tool for this) -- Docker (optional, but it will make your life easier) +- NodeJS ([nvm](https://github.com/nvm-sh/nvm) is a good tool for this) +- Docker +- `psql` shell utility for accessing the database + - On Debian/Ubuntu based systems you can install `postgresql-client` + - On a Mac, this is `libpg` -For the sake of making this easier to follow, we will create a directory in the home directory -called `slack-bridge-env`: +Docker is used here to reduce the number of requirements on the host system, +and will allow you to setup your environment faster. + +For the sake of making this easier to follow, we will create a directory in +our home directory called `slack-bridge-env`: ```bash mkdir ~/slack-bridge-env cd ~/slack-bridge-env ``` -### Setting up Synapse +### 1. Setting up Synapse + +Synapse is the reference implementation of a Matrix homeserver. You may use other +homeserver implementations that support the AS API, but for the sake of simplicity +this is how to setup Synapse. + +To generate your config: + +```bash +docker run -it --rm \ + -v ~/slack-bridge-env/synapse:/data \ + -e SYNAPSE_SERVER_NAME=localhost \ + -e SYNAPSE_REPORT_STATS=no \ + matrixdotorg/synapse:v1.3.1-py3 generate +``` + +Open the generated `homeserver.yaml` file for editing. +Find and uncomment the `enable_registration` field, set it to `true`: + +```yaml +enable_registration: true +``` + +You will need to add an appservice registration file in the future, +but it is not important for now. + + +And then to run the homeserver: + +```bash +docker run -d --name synapse \ + -v ~/slack-bridge-env/synapse:/data \ + -p 8008:8008 \ + matrixdotorg/synapse:v1.3.1-py3 +``` + +These instructions are based off those given in https://hub.docker.com/r/matrixdotorg/synapse. -Largely to setup Synapse, you can follow https://hub.docker.com/r/matrixdotorg/synapse and just ensure your data directory points to `~/slack-bridge-env/synapse`. -You will want to make sure your server name is something that routes to your local box. I tend to use the devbox hostname, but `localhost` is also sufficent. -You should enable registration in the `homeserver.yaml` to create a testing user. -You will need to add an appservice registration file in the future, but it is not imprtant for now. -### Setting up Riot +### 2. Setting up Riot Setting up Riot is also quite straightforward: -Create a new config called `~/slack-bridge-env/riot-config.json` with the contents needed to set defaults to your local homeserver. +Create a new config called `~/slack-bridge-env/riot-config.json` with the contents +needed to set defaults to your local homeserver. As an example: @@ -60,11 +102,13 @@ As an example: } ``` -Finally, run `docker run -v /home/will/git/scalar-env/riot-config.json:/app/config.json -p 8080:80 vectorim/riot-web` to start your Riot instance. You should be able to register a new user on your local synapse instance through Riot. +Finally, run `docker run -v /home/will/git/scalar-env/riot-config.json:/app/config.json -p 8080:80 vectorim/riot-web` +to start your Riot instance. You should be able to register a new user on your +local synapse instance through Riot. -### Setting up the bridge postgres +### 3. Setting up the bridge with PostgreSQL -Setting up the postgres instance for the bridge is also quite easy. +You can setup PostgreSQL in docker. ```bash docker run -d --name slackpg -p 59999:5432 -e POSTGRES_PASSWORD=pass postgres @@ -73,17 +117,16 @@ docker run -d --name slackpg -p 59999:5432 -e POSTGRES_PASSWORD=pass postgres You should also create a new database in preparation for using the bridge ```bash -sudo apt install postgresql-client # Ensure you have psql installed psql -h localhost -U postgres -W postgres ``` -And then on the postgres shell +And then on the postgres shell: ```sql CREATE DATABASE slack ``` -### Setting up the bridge +### 4. Setting up the bridge Clone the bridge repo to a place of your choice, again for simplicity we will use the env directory. Make sure you clone your **fork** and not the upstream repo. @@ -101,9 +144,12 @@ The steps above should make sure you are running the latest development version and have built the typescript. To ensure that it's all working, you can run `npm test` which will run the unit and integration tests. -You can follow the instructions in the [README](../README.md) to generate and update the registration file, as well as creating a testing Slack app and workspace. You should enable RTM support in the config, as Slack will not be able to push events to your local bridge. +You can follow the instructions in the [README](../README.md), under the to generate and update the +registration file. You should follow the "Recommended - Events API" to setup a Slack app. This guide +strongly recommends using the RTM API for local development as it does not require a webserver. -Make sure that you copy the generated registration file to `~/slack-bridge-env/synapse` and add an entry for it in the `homeserver.yaml` before starting the synapse container. +Make sure that you copy the generated registration file to `~/slack-bridge-env/synapse` and +add an entry for it in the `homeserver.yaml` before starting the synapse container. ## Making changes @@ -112,10 +158,13 @@ Whenever you want to make changes to the codebase, you must: ```bash git fetch matrix-org git checkout matrix-org/develop -git checkout -b yourfeaturname +git checkout -b yourfeaturename npm i ``` -You should always work within the `src` directory. It is helpful to have a code editor setup with linting enabled so you can see mistakes as you work. Always remember to run `npm run build` before commiting or testing so that you know the latest changes work. +You should always work within the `src` directory. It is helpful to have a code editor setup +with linting enabled so you can see mistakes as you work. Always remember to run `npm run build` +before commiting or testing so that you know the latest changes work. -Before commiting your work, ensure that the tests pass locally with `npm test`. If you are making changes to packages, ensure that `package-lock.json` is included in the commit. +Before commiting your work, ensure that the tests pass locally with `npm test`. If you are making +changes to packages, ensure that `package-lock.json` is included in the commit. From e6e5ff7e0c0699e4c949ce8e66da9215a59e5924 Mon Sep 17 00:00:00 2001 From: Half-Shot Date: Thu, 12 Sep 2019 11:45:23 +0100 Subject: [PATCH 5/8] Make the instructions for setting up the bridge a little more verbose --- README.md | 55 ++++++++++++++++++++++++++++++++----------------------- 1 file changed, 32 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index a7341b841..5c683075c 100644 --- a/README.md +++ b/README.md @@ -101,23 +101,40 @@ and bot users. This allows you to link as many channels as you would like with o 3. Click on `bot users` and add a new bot user. We will use this account to bridge the the rooms. -4. Click on `Event Subscriptions` and enable them. At this point, the bridge needs to be - started as Slack will do some verification of the request rul. The request url should be - `https://$HOST:$SLACK_PORT"`. Then add the following events and save: +4. Click on `Event Subscriptions` and enable them. - Bot User Events: + 1. You will need to decide on using either the **RTM API** or the **Events API**. + The RTM API is recommended because it does not require you to setup scopes or open ports + to a webserver like the Events API. + Events API provisioning is forwards compatible with the RTM API and if enabled in the config, + RTM is used by default. + The RTM API uses websockets and will pull information from Slack, whereas the Events API will + push informaton to the bridge over HTTP. - - team_domain_change - - message.channels - - chat:write:bot - - message.groups (if you want to bridge private channels) - - users:read - - team.info + 2. If you want to use the RTM API, ensure that you have enabled RTM support in the + config file. And then put your feet up, you are done. -5. Skip this step if you do not want to bridge files. - Click on `OAuth & Permissions` and add the following scopes: + 3. If you want to use the Events API, follow these steps: - - files:write:user + The bridge needs to be started as Slack will do some verification of the request url. The request url should be `https://$HOST:$SLACK_PORT"`. Then add the following events to "Subscribe to Bot Events" and save: + + - team_domain_change + - message.channels + - reaction_added + - reaction_removed + +5. Click on `OAuth & Permissions` and add the following scopes: + + - files:read (So the bot may upload files to Matrix) + - files:write:user (Upload files from Matrix) + - users:read (Profile information on users) + - team:read (Get basic information about the workspace) + - chat:write:bot (Write messages as the bridge bot) + - reactions:write (Add/remove reactions on Slack) + + The following are needed if you plan to integrate the bridge with an Integration Manager: + + - channels:read (Used to fetch information about public channels on the workspace) Note: any media uploaded to matrix is currently accessible by anyone who knows the url. In order to make Slack files visible to matrix users, this bridge will make Slack files @@ -126,9 +143,8 @@ and bot users. This allows you to link as many channels as you would like with o posted in private channels. See [MSC701](https://github.com/matrix-org/matrix-doc/issues/701) for details. -6. Click on `Install App` and `Install App to Workspace`. Note the access tokens show. - You will need the `Bot User OAuth Access Token` and if you want to bridge files, the - `OAuth Access Token` whenever you link a room. +6. Click on `Install App` and `Install App to Workspace`. Note the access token shown. + You will need the `OAuth Access Token` whenever you link a room. 7. For each channel you would like to bridge, perform the following steps: @@ -147,16 +163,9 @@ and bot users. This allows you to link as many channels as you would like with o 3. Issue a ``link`` command in the administration control room with these collected values as arguments: - with file bridging: - ``` link --channel_id CHANNELID --room !the-matrix:room.id --slack_bot_token xoxb-xxxxxxxxxx-xxxxxxxxxxxxxxxxxxxx --slack_user_token xoxp-xxxxxxxx-xxxxxxxxx-xxxxxxxx-xxxxxxxx ``` - without file bridging: - - ``` - link --channel_id CHANNELID --room !the-matrix:room.id --slack_bot_token xoxb-xxxxxxxxxx-xxxxxxxxxxxxxxxxxxxx - ``` These arguments can be shortened to single-letter forms: From 73ac8079d229f9bbefe698499691942b375b47dc Mon Sep 17 00:00:00 2001 From: Half-Shot Date: Thu, 12 Sep 2019 11:52:54 +0100 Subject: [PATCH 6/8] Add development section to readme --- README.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/README.md b/README.md index 5c683075c..a5cef48c4 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,14 @@ A Matrix <--> Slack bridge This bridge allows you to connect Slack channels to Matrix rooms. +Development +----------- + +[![#slack:half-shot.uk](https://img.shields.io/matrix/slack:half-shot.uk.svg?server_fqdn=matrix.half-shot.uk&label=%23slack:half-shot.uk&logo=matrix)](https://matrix.to/#/#slack:half-shot.uk) + +If you want to help out, please give our helpful [Development Guide](./docs/development_guide.md) +a read. It covers all you need to know to get hacking on the bridge. + Installation ------------ From 3b67d04930c71d92136b6ab0ae48bca043db5b6a Mon Sep 17 00:00:00 2001 From: Half-Shot Date: Mon, 16 Sep 2019 17:08:07 +0100 Subject: [PATCH 7/8] Fix missing port --- docs/development_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development_guide.md b/docs/development_guide.md index a34149c42..497aa307c 100644 --- a/docs/development_guide.md +++ b/docs/development_guide.md @@ -117,7 +117,7 @@ docker run -d --name slackpg -p 59999:5432 -e POSTGRES_PASSWORD=pass postgres You should also create a new database in preparation for using the bridge ```bash -psql -h localhost -U postgres -W postgres +psql -h localhost -p 59999 -U postgres -W postgres ``` And then on the postgres shell: From 86e16a28ec4c6c784d8c54568d751366ce5c1e73 Mon Sep 17 00:00:00 2001 From: Half-Shot Date: Mon, 16 Sep 2019 17:08:34 +0100 Subject: [PATCH 8/8] Fix riot docker command --- docs/development_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development_guide.md b/docs/development_guide.md index 497aa307c..fedd2c7c1 100644 --- a/docs/development_guide.md +++ b/docs/development_guide.md @@ -102,7 +102,7 @@ As an example: } ``` -Finally, run `docker run -v /home/will/git/scalar-env/riot-config.json:/app/config.json -p 8080:80 vectorim/riot-web` +Finally, run `docker run -d --name riot -v ~/slack-bridge-env/riot-config.json:/app/config.json -p 8080:80 vectorim/riot-web` to start your Riot instance. You should be able to register a new user on your local synapse instance through Riot.