You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
SirMordred is the tool used to coordinate the execution of the GrimoireLab platform, via two main configuration files, the [setup.cfg](./setup-cfg.md) and [projects.json](./projects-json.md), which are summarized in their corresponding sections.
10
+
11
+
SirModred relies on ElasticSearch, Kibiter and MySQL/MariaDB. The current versions used are:
12
+
13
+
- ElasticSearch 6.8.6
14
+
- Kibiter 6.8.6
15
+
- MySQL/MariaDB (5.7.24/10.0)
16
+
17
+
There are mainly 2 options to get started with SirMordred:
18
+
-[Source code and docker](#source-code-and-docker-):
19
+
In this method, the applications (ElasticSearch, Kibiter and MariaDB) are installed using docker and the GrimoireLab Components are installed using the source code.
20
+
-[Only docker](#only-docker-):
21
+
In this method, the applications (ElasticSearch, Kibiter and MariaDB) and the GrimoireLab Components are installed using docker.
22
+
23
+
## Source code and docker
24
+
25
+
### Getting the containers
26
+
27
+
You will have to install ElasticSearch (6.8.6), Kibiter (6.8.6) and a MySQL/MariaDB database (5.7.24/10.0). You can use the following docker-compose to have them running.
28
+
29
+
> Help: You need to install docker and docker-compose for this. Please refer the documentation.
> 1. You can omit (comment/remove) the `mariadb` section in case you have MariaDB or MySQL already installed in your system.
34
+
> 2. It is not mandatory to use docker to install ElasticSearch, Kibiter and MySQL/MariaDB database. They can be installed by other means too (source code). We are not much concerned about the method they are installed. Docker is the easiest way as it mostly avoids the errors caused by them.
35
+
36
+
**docker-compose (with SearchGuard)**
37
+
38
+
> **Note**: For accessing Kibiter and/or creating indexes login is required, the `username:password` is `admin:admin` in [`setup.cfg`](https://github.com/chaoss/grimoirelab-sirmordred/blob/master/sirmordred/utils/setup.cfg) file.
> You can follow this [tutorial](https://www.jetbrains.com/help/pycharm/quick-start-guide.html) to get familiar with PyCharm.
146
+
Once PyCharm is installed create a project in the grimoirelab-sirmordred directory.
147
+
PyCharm will automatically create a virtual env, where you should install the dependencies listed in each
148
+
requirements.txt, **excluding** the ones concerning the grimoirelab components.
149
+
150
+
To install the dependencies, you can click on `File` -> `Settings` -> `Project` -> `Project Interpreter`, and then the `+` located on the top right corner (see figure below).
Now that you have the ElasticSearch, Kibiter and MariaDB running on your system and the project configured in the PyCharm, we can execute micro-mordred/sirmordred.
162
+
163
+
To execute micro-mordred, define a [setup.cfg](https://github.com/chaoss/grimoirelab-sirmordred/blob/master/sirmordred/utils/setup.cfg) and [projects.json](https://github.com/chaoss/grimoirelab-sirmordred/blob/master/sirmordred/utils/projects.json), and
164
+
run the following commands, which will collect and enrich the data coming from the git sections and upload the corresponding panels to Kibiter:
Copy file name to clipboardExpand all lines: docs/getting-started/troubleshooting.md
+190-1Lines changed: 190 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -25,12 +25,14 @@ grimoirelab/docker-compose$ docker-compose up
25
25
26
26
---
27
27
28
-
##Table of contents
28
+
### Following is a list of common problems encountered while setting up GrimoireLab
29
29
{: .no_toc .text-delta }
30
30
31
31
1. TOC
32
32
{:toc}
33
33
34
+
> **NOTE**: In order to see the logs, run ```docker-compose up``` without the ```-d``` or ```--detach``` option while starting/(re)creating/building/attaching containers for a service.
35
+
>
34
36
---
35
37
36
38
### Port already in use
@@ -65,3 +67,190 @@ by looking at the filter bar as shown in the following screenshot or the <span
65
67
style="color: #2f4bff">time window</span>.
66
68
67
69
70
+
71
+
### Low Virtual Memory
72
+
73
+
* Indications:
74
+
Cannot open ```https://localhost:9200/``` in browser. Shows ```Secure connection Failed```,
75
+
```PR_END_OF_FILE_ERROR```, ```SSL_ERROR_SYSCALL in connection to localhost:9200``` messages.
76
+
* Diagnosis:
77
+
Check for the following log in the output of ```docker-compose up```
elasticsearch_1 | [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
81
+
````
82
+
* Solution:
83
+
Increase the kernel ```max_map_count``` parameter of vm. Execute the following command
84
+
```sudo sysctl -w vm.max_map_count=262144```
85
+
Now stop the container services and re-run ```docker-compose up```.
86
+
Note that this is valid only for current session. To set this value permanently, update the ```vm.max_map_count``` setting
87
+
in /etc/sysctl.conf. To verify after rebooting, run sysctl vm.max_map_count.
88
+
89
+
### Processes have conflicts with SearchGuard
90
+
91
+
* Indications:
92
+
- Cannot open ```localhost:9200``` in browser, shows ```Secure connection Failed```
93
+
- ```curl -XGET localhost:9200 -k``` gives
94
+
```curl: (52) Empty reply from server```
95
+
* Diagnosis:
96
+
Check for the following log in the output of ```docker-compose up```
97
+
```
98
+
elasticsearch_1 | [2020-03-12T13:05:34,959][WARN ][c.f.s.h.SearchGuardHttpServerTransport] [Xrb6LcS] Someone (/172.18.0.1:59838) speaks http plaintext instead of ssl, will close the channel
99
+
```
100
+
Check for conflicting processes by running ```sudo lsof -i:58888``` (e.g. 58888 is the port number)
101
+
* Solution:
102
+
1. Try to close the conflicting processes:
103
+
You can do this easily with fuser (```sudo apt-get install fuser```),
104
+
run ```fuser -k 58888/tcp``` (e.g. 58888 is the port number).
105
+
Re-run ```docker-compose up``` and check if ```localhost:9200``` shows up.
106
+
2. Use a [docker-compose without SearchGuard](#docker-compose-without-searchguard-):
107
+
Use the docker-compose above, this doesn't include SearchGuard.
108
+
Note: With this docker-compose, access to the Kibiter and ElasticSearch don't require credentials.
109
+
Re-run ```docker-compose up``` and check if ```localhost:9200``` shows up.
110
+
111
+
### Permission Denied
112
+
113
+
* Indications:
114
+
Can't create indices in Kibana. Nothing happens after clicking create index.
115
+
* Diagnosis:
116
+
Check for the following log in the output of ```docker-compose up```
117
+
```
118
+
elasticsearch_1 |[INFO ][c.f.s.c.PrivilegesEvaluator] No index-level perm match for User [name=readall, roles=[readall], requestedTenant=null][IndexType [index=.kibana, type=doc]][Action [[indices:data/write/index]]][RolesChecked [sg_own_index, sg_readall]]
119
+
elasticsearch_1 | [c.f.s.c.PrivilegesEvaluator] No permissions for {sg_own_index=[IndexType [index=.kibana, type=doc]], sg_readall=[IndexType [index=.kibana, type=doc]]}
This message generally appears when you try to create an index pattern but you are not logged in Kibana.
126
+
127
+
128
+
129
+
130
+
Try logging in to Kibana (the login button is on the bottom left corner).
131
+
The credentials used for login should be username: `admin` and password: `admin`.
132
+
133
+
### Empty Index
134
+
135
+
* Indications and Diagnosis:
136
+
Check for the following error after executing [Micro Mordred](https://github.com/chaoss/grimoirelab-sirmordred/tree/master/sirmordred/utils/micro.py)
137
+
using ```micro.py --raw --enrich --panels --cfg ./setup.cfg --backends git```(Here, using git as backend)
138
+
```
139
+
[git] Problem executing study enrich_areas_of_code:git, RequestError(400, 'search_phase_execution_exception', 'No mapping
140
+
found for [metadata__timestamp] in order to sort on')
141
+
```
142
+
* Solution:
143
+
This error appears when the index is empty (here, ```git-aoc_chaoss_enriched``` index is empty). An index can be empty when
144
+
the local clone of the repository being analyzed is in sync with the upstream repo, so there will be no new commits to
145
+
ingest to grimoirelab.
146
+
147
+
There are 2 methods to solve this problem:
148
+
149
+
Method 1: Disable the param [latest-items](https://github.com/chaoss/grimoirelab-sirmordred/blob/master/sirmordred/utils/setup.cfg#L78) by setting it to false.
150
+
151
+
Method 2: Delete the local clone of the repo (which is stored in ```~/.perceval/repositories```).
152
+
153
+
Some extra details to better understand this behavior:
154
+
155
+
The Git backend of perceval creates a clone of the repository (which is stored in ```~/.perceval/repositories```) and keeps the local
156
+
copy in sync with the upstream one. This clone is then used to ingest the commits data to grimoirelab.
157
+
Grimoirelab periodically collects data from different data sources (in this specific case, a git repository) in an incremental way.
158
+
A typical execution of grimoirelab for a git repository consists of ingesting only the new commits to the platform. These
159
+
commits are obtained by comparing the local copy with the upstream one, thus if the two repos are synchronized, then no
160
+
commits are returned and hence Index will be empty. In the case where all commits need to be extracted even if there is already a
161
+
local clone, latest-items param should be disabled. Another option is to delete the local clone (which is stored at ```~/.perceval/repositories```),
162
+
and by doing so the platform will clone the repo again and extract all commits.
163
+
164
+
### Low File Descriptors
165
+
166
+
* Indications:
167
+
- Cannot open ```localhost:9200``` in browser, shows ```Secure connection Failed```
168
+
- ```curl -XGET localhost:9200 -k``` gives
169
+
```curl: (7) Failed to connect to localhost port 9200: Connection refused```
170
+
171
+
* Diagnosis:
172
+
Check for the following log in the output of ```docker-compose up```
Retrying (Retry(total=10,connected=21,read=0,redirect=5,status=None)) after connection broken by
226
+
'SSLError(SSLError{1,'[SSL: WRONG_VERSION_NUMBER] wrong version number {_ssl.c:852}'},)': /
227
+
```
228
+
* Solution: Change 'https' to 'http' in the setup.cfg file
229
+
```
230
+
[es_collection]
231
+
# arthur = true
232
+
# arthur_url = http://127.0.0.1:8080
233
+
# redis_url = redis://localhost/8
234
+
url = http://localhost:9200
235
+
[es_enrichment]
236
+
url = http://localhost:9200
237
+
```
238
+
239
+
### Cloc installation
240
+
241
+
* Diagnosis:
242
+
```
243
+
: [Errno 2]No such file or directory : 'cloc': 'cloc'
244
+
```
245
+
* Solution:
246
+
Execute the following command to install `cloc` (more details are available in the [Graal](https://github.com/chaoss/grimoirelab-graal#how-to-installcreate-the-executables) repo)
247
+
```
248
+
sudo apt-get install cloc
249
+
```
250
+
### Incomplete data
251
+
252
+
* Indication: Not all the data is being retrieved when rebuilding an index - only from a point in time forward.
253
+
* Diagnosis: After a rebuild of git-based indices you are not receiving a full dataset as expected, but only from the date of the re-index forward. That data is complete, but anything prior to that is missing.
254
+
* Solution: The `setup.cfg` file has an option under the Git configuration section: `latest-items = true` - set this to `latest-items = false` to pull in all data from the beginnning. Once this has been processed, remember to set it back to `latest-items = true`!
0 commit comments