Skip to content

Commit 90ad5eb

Browse files
vchrombievchrombie
authored
Merge pull request #208 from vchrombie/dev-setup-troubleshooting-cleanup
Update content and fix formatting issues
2 parents dd1607c + 1fb7854 commit 90ad5eb

2 files changed

Lines changed: 421 additions & 216 deletions

File tree

‎docs/getting-started/dev-setup.md‎

Lines changed: 132 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -4,38 +4,59 @@ title: Developer Installation
44
nav_order: 2
55
parent: Getting Started
66
---
7+
78
# Developer Installation
89

9-
SirMordred is the tool used to coordinate the execution of the GrimoireLab platform, via two main configuration files, the [setup.cfg](./setup-cfg.md) and [projects.json](./projects-json.md), which are summarized in their corresponding sections.
10+
SirMordred is the tool used to coordinate the execution of the GrimoireLab
11+
platform, via two main configuration files, the [setup.cfg](./setup-cfg.md) and
12+
[projects.json](./projects-json.md), which are summarized in their corresponding
13+
sections.
1014

11-
SirModred relies on ElasticSearch, Kibiter and MySQL/MariaDB. The current versions used are:
15+
SirModred relies on Elasticsearch, Kibiter and MySQL/MariaDB. The current
16+
versions used are:
1217

13-
- ElasticSearch 6.8.6
18+
- Elasticsearch 6.8.6
1419
- Kibiter 6.8.6
1520
- MySQL/MariaDB (5.7.24/10.0)
1621

1722
There are mainly 2 options to get started with SirMordred:
18-
- [Source code and docker](#source-code-and-docker-):
19-
In this method, the applications (ElasticSearch, Kibiter and MariaDB) are installed using docker and the GrimoireLab Components are installed using the source code.
20-
- [Only docker](#only-docker-):
21-
In this method, the applications (ElasticSearch, Kibiter and MariaDB) and the GrimoireLab Components are installed using docker.
23+
- [Source code and docker](#source-code-and-docker): In this method, the
24+
applications (Elasticsearch, Kibiter and MariaDB) are installed using docker and
25+
the GrimoireLab Components are installed using the source code.
26+
- [Only docker](#only-docker): In this method, the applications (Elasticsearch,
27+
Kibiter and MariaDB) and the GrimoireLab Components are installed using docker.
2228

2329
## Source code and docker
2430

2531
### Getting the containers
2632

27-
You will have to install ElasticSearch (6.8.6), Kibiter (6.8.6) and a MySQL/MariaDB database (5.7.24/10.0). You can use the following docker-compose to have them running.
33+
You will have to install Elasticsearch (6.8.6), Kibiter (6.8.6) and a
34+
MySQL/MariaDB database (5.7.24/10.0). You can use the following docker-compose
35+
to have them running.
36+
37+
> Help: You need to install docker and docker-compose for this. Please refer the
38+
> documentation.
39+
- [https://docs.docker.com/install/linux/docker-ce/ubuntu/](https://docs.docker.com/install/linux/docker-ce/ubuntu/)
40+
- [https://docs.docker.com/compose/install/](https://docs.docker.com/compose/install/)
2841

29-
> Help: You need to install docker and docker-compose for this. Please refer the documentation.
30-
> - https://docs.docker.com/install/linux/docker-ce/ubuntu/
31-
> - https://docs.docker.com/compose/install/
3242
> Note:
33-
> 1. You can omit (comment/remove) the `mariadb` section in case you have MariaDB or MySQL already installed in your system.
34-
> 2. It is not mandatory to use docker to install ElasticSearch, Kibiter and MySQL/MariaDB database. They can be installed by other means too (source code). We are not much concerned about the method they are installed. Docker is the easiest way as it mostly avoids the errors caused by them.
43+
> 1. You can omit (comment/remove) the `mariadb` section in case you have
44+
> MariaDB or MySQL already installed in your system.
45+
> 2. It is not mandatory to use docker to install Elasticsearch, Kibiter and
46+
> MySQL/MariaDB database. They can be installed by other means too (source
47+
> code). We are not much concerned about the method they are installed.
48+
> Docker is the easiest way as it mostly avoids the errors caused by them.
49+
50+
There are two versions of Elasticsearch and Kibiter which can be used, with
51+
SearchGuard (secured one) and the without SearchGuard (not secured). You can
52+
choose either of the versions.
3553

3654
**docker-compose (with SearchGuard)**
3755

38-
> **Note**: For accessing Kibiter and/or creating indexes login is required, the `username:password` is `admin:admin` in [`setup.cfg`](https://github.com/chaoss/grimoirelab-sirmordred/blob/master/sirmordred/utils/setup.cfg) file.
56+
For accessing Elasticsearch/Kibiter, login credentials (`admin:admin`) is
57+
required. You are also expected to update the configurations in the `setup.cfg`
58+
file. See [#defining-the-es-configurations](#defining-the-es-configurations).
59+
3960

4061
```
4162
elasticsearch:
@@ -45,6 +66,7 @@ elasticsearch:
4566
- 9200:9200
4667
environment:
4768
- ES_JAVA_OPTS=-Xms2g -Xmx2g
69+
4870
kibiter:
4971
restart: on-failure:5
5072
image: bitergia/kibiter:secured-v6.8.6-3
@@ -59,14 +81,24 @@ kibiter:
5981
- elasticsearch
6082
ports:
6183
- 5601:5601
84+
85+
mariadb:
86+
image: mariadb:10.0
87+
expose:
88+
- "3306"
89+
environment:
90+
- MYSQL_ROOT_PASSWORD=
91+
- MYSQL_ALLOW_EMPTY_PASSWORD=yes
6292
```
6393

64-
**docker-compose (without SearchGuard)**
94+
**docker-compose (without SearchGuard)**
6595

66-
> **Note**: Here, access to kibiter and elasticsearch don't need credentials.
96+
For accessing Elasticsearch/Kibiter, no login credentials is required. The
97+
configurations in the `setup.cfg` file need to be updated. See
98+
[#defining-the-es-configurations](#defining-the-es-configurations).
6799

68-
```
69100

101+
```
70102
elasticsearch:
71103
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.6
72104
command: elasticsearch -Enetwork.bind_host=0.0.0.0 -Ehttp.max_content_length=2000mb
@@ -75,6 +107,7 @@ elasticsearch:
75107
environment:
76108
- ES_JAVA_OPTS=-Xms2g -Xmx2g
77109
- ANONYMOUS_USER=true
110+
78111
kibiter:
79112
restart: on-failure:5
80113
image: bitergia/kibiter:community-v6.8.6-3
@@ -86,17 +119,26 @@ kibiter:
86119
- elasticsearch
87120
ports:
88121
- 5601:5601
122+
123+
mariadb:
124+
image: mariadb:10.0
125+
expose:
126+
- "3306"
127+
environment:
128+
- MYSQL_ROOT_PASSWORD=
129+
- MYSQL_ALLOW_EMPTY_PASSWORD=yes
89130
```
90131

91132
Save the above into a docker-compose.yml file and run
92133
```
93134
$ docker-compose up -d
94135
```
95-
to get ElasticSearch, Kibiter and MariaDB running on your system.
136+
to get Elasticsearch, Kibiter and MariaDB running on your system.
96137

97138
### Cloning the repositories
98139

99-
In the next step, you will need to fork all the GitHub repos below and clone them to a target local folder (e.g., `sources`).
140+
In the next step, you will need to fork all the GitHub repos below and clone
141+
them to a target local folder (e.g., `sources`).
100142

101143
- [SirModred](https://github.com/chaoss/grimoirelab-sirmordred)
102144
- [ELK](https://github.com/chaoss/grimoirelab-elk)
@@ -113,7 +155,8 @@ In the next step, you will need to fork all the GitHub repos below and clone the
113155
- [Cereslib](https://github.com/chaoss/grimoirelab-cereslib)
114156
- [Manuscripts](https://github.com/chaoss/grimoirelab-manuscripts)
115157

116-
Each local repo should have two `remotes`: `origin` points to the forked repo, while `upstream` points to the original CHAOSS repo.
158+
Each local repo should have two `remotes`: `origin` points to the forked repo,
159+
while `upstream` points to the original CHAOSS repo.
117160

118161
An example is provided below.
119162
```
@@ -131,43 +174,94 @@ $ git remote add upstream https://github.com/chaoss/grimoirelab-perceval
131174

132175
#### ProTip
133176

134-
You can use this use this [script](https://gist.github.com/vchrombie/4403193198cd79e7ee0079259311f6e8) to automate this whole process.
177+
You can use this use this
178+
[script](https://gist.github.com/vchrombie/4403193198cd79e7ee0079259311f6e8) to
179+
automate this whole process.
135180
```
136181
$ python3 glab-dev-env-setup.py --create --token xxxx --source sources
137182
```
138183

139184
### Setting up PyCharm
140185

141-
> Help:
142-
> You need to install PyCharm (**Community Edition**) for this. Please refer the documentation.
143-
> - https://www.jetbrains.com/help/pycharm/installation-guide.html
144-
>
145-
> You can follow this [tutorial](https://www.jetbrains.com/help/pycharm/quick-start-guide.html) to get familiar with PyCharm.
146-
Once PyCharm is installed create a project in the grimoirelab-sirmordred directory.
147-
PyCharm will automatically create a virtual env, where you should install the dependencies listed in each
148-
requirements.txt, **excluding** the ones concerning the grimoirelab components.
186+
> Help: You need to install PyCharm (**Community Edition**) for this. Please
187+
> refer the [JetBrain's PyCharm
188+
> documentation](https://www.jetbrains.com/help/pycharm/installation-guide.html).
189+
> You can follow this [PyCharm
190+
> Quickstart](https://www.jetbrains.com/help/pycharm/quick-start-guide.html) to
191+
> get familiar with PyCharm.
192+
193+
Once PyCharm is installed create a project in the grimoirelab-sirmordred
194+
directory.
195+
196+
PyCharm will automatically create a virtual env, where you should install the
197+
dependencies listed in each requirements.txt, **excluding** the ones concerning
198+
the grimoirelab components.
199+
```
200+
PyJWT>=1.7.1 (install)
201+
cryptography>=3.3.1 (install)
202+
-e git+https://github.com/chaoss/grimoirelab-toolkit/#egg=grimoirelab-toolkit (don't install)
203+
```
149204

150-
To install the dependencies, you can click on `File` -> `Settings` -> `Project` -> `Project Interpreter`, and then the `+` located on the top right corner (see figure below).
205+
To install the dependencies, you can click on `File` -> `Settings` -> `Project`
206+
-> `Project Interpreter`, and then the `+` located on the top right corner (see
207+
figure below).
151208

152209
![project-interpreter-configuration](https://user-images.githubusercontent.com/25265451/78168870-3e612580-746e-11ea-9df1-7ba94b84d07b.gif)
153210

154-
Later, you can add the dependencies to the grimoirelab components via `File` -> `Settings` -> `Project` -> `Project Structure`.
155-
The final results should be something similar to the image below.
211+
Later, you can add the dependencies to the grimoirelab components via `File` ->
212+
`Settings` -> `Project` -> `Project Structure`. The final results should be
213+
something similar to the image below.
156214

157215
![project-structure-configuration](https://user-images.githubusercontent.com/25265451/78168879-41f4ac80-746e-11ea-9e40-dbdb1b5d32f2.gif)
158216

159217
### Execution
160218

161-
Now that you have the ElasticSearch, Kibiter and MariaDB running on your system and the project configured in the PyCharm, we can execute micro-mordred/sirmordred.
219+
Now that you have the Elasticsearch, Kibiter and MariaDB running on your system
220+
and the project configured in the PyCharm, we can execute
221+
micro-mordred/sirmordred.
222+
223+
To execute micro-mordred, define a
224+
[setup.cfg](https://github.com/chaoss/grimoirelab-sirmordred/blob/master/sirmordred/utils/setup.cfg)
225+
and
226+
[projects.json](https://github.com/chaoss/grimoirelab-sirmordred/blob/master/sirmordred/utils/projects.json).
227+
228+
#### Defining the ES configurations
229+
230+
- For accessing Elasticsearch/Kibiter, login credentials (`admin:admin`) is
231+
required. You are also expected to update the configurations in the
232+
`setup.cfg` file.
233+
234+
```
235+
[es_collection]
236+
url = https://admin:admin@localhost:9200
237+
238+
[es_enrichment]
239+
url = https://admin:admin@localhost:9200
240+
```
241+
242+
- For accessing Elasticsearch/Kibiter, no login credentials is required. The
243+
configurations in the `setup.cfg` file can be updated as below.
244+
245+
```
246+
[es_collection]
247+
url = http://localhost:9200
162248
163-
To execute micro-mordred, define a [setup.cfg](https://github.com/chaoss/grimoirelab-sirmordred/blob/master/sirmordred/utils/setup.cfg) and [projects.json](https://github.com/chaoss/grimoirelab-sirmordred/blob/master/sirmordred/utils/projects.json), and
164-
run the following commands, which will collect and enrich the data coming from the git sections and upload the corresponding panels to Kibiter:
249+
[es_enrichment]
250+
url = http://localhost:9200
165251
```
252+
253+
### Micro Mordred
254+
255+
Run the following commands, which will collect and enrich the data coming from
256+
the git sections and upload the corresponding panels to Kibiter:
257+
258+
```bash
166259
micro.py --raw --enrich --cfg ./setup.cfg --backends git cocom
167260
micro.py --panels --cfg ./setup.cfg
168261
```
169262

170-
Optionally, you can create a configuration in PyCharm to speed up the executions (`Run` -> `Edit configuration` -> `+`).
263+
Optionally, you can create a configuration in PyCharm to speed up the executions
264+
(`Run` -> `Edit configuration` -> `+`).
171265

172266
![add-micro-configuration](https://user-images.githubusercontent.com/25265451/78168875-402ae900-746e-11ea-8bd8-4b3e68992bdf.gif)
173267

@@ -177,6 +271,5 @@ The final results should be something similar to the image below.
177271

178272
## Only docker
179273

180-
Follow the instruction in the GrimoireLab tutorial to have [SirMordred in a container](../../sirmordred/container.md)
181-
182-
---
274+
Follow the instruction in the GrimoireLab tutorial to have [SirMordred in a
275+
container](../../sirmordred/container.md)

0 commit comments

Comments
 (0)