Skip to content

Commit e11a8c8

Browse files
filipecosta90filipecosta90
authored
[add] Updated docs to include how-to add a new benchmark platform (#51)
1 parent 5334ed5 commit e11a8c8

File tree

5 files changed

+355
-136
lines changed

5 files changed

+355
-136
lines changed

‎Readme.md

Lines changed: 176 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,18 @@
22
[![codecov](https://codecov.io/gh/filipecosta90/redis-benchmarks-specification/branch/main/graph/badge.svg?token=GS64MV1H4W)](https://codecov.io/gh/filipecosta90/redis-benchmarks-specification)
33
[![CI tests](https://github.com/filipecosta90/redis-benchmarks-specification/actions/workflows/tox.yml/badge.svg)](https://github.com/filipecosta90/redis-benchmarks-specification/actions/workflows/tox.yml)
44
[![PyPI version](https://badge.fury.io/py/redis-benchmarks-specification.svg)](https://badge.fury.io/py/redis-benchmarks-specification)
5+
6+
## Installation
7+
8+
To have access to the latest SPEC and Tooling impletamtion you only need to install one python package.
9+
10+
Installation is done using pip, the package installer for Python, in the following manner:
11+
12+
```bash
13+
python3 -m pip install redis-benchmarks-specification --ignore-installed PyYAML
14+
```
15+
16+
517
## Benchmark specifications goal
618

719
The Redis benchmarks specification describes the cross-language/tools requirements and expectations to foster performance and observability standards around redis related technologies.
@@ -109,7 +121,8 @@ relevant git information.
109121

110122
The HTTP request is then converted into an event ( tracked within redis ) that will trigger multiple build variants requests based upon the distinct platforms described in [`platforms`](redis_benchmarks_specification/setups/platforms/).
111123

112-
As soon as a new build variant request is received, the build agent ([`redis-benchmarks-spec-builder`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/__builder__/)) prepares the artifact(s) and proceeds into adding an artifact benchmark event so that the benchmark coordinator ([`redis-benchmarks-spec-sc-coordinator`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/__self_contained_coordinator__/)) can deploy/manage the required infrastructure and DB topologies, run the benchmark, and export the performance results.
124+
As soon as a new build variant request is received, the build agent ([`redis-benchmarks-spec-builder`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/__builder__/))
125+
prepares the artifact(s) and proceeds into adding an artifact benchmark event so that the benchmark coordinator ([`redis-benchmarks-spec-sc-coordinator`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/__self_contained_coordinator__/)) can deploy/manage the required infrastructure and DB topologies, run the benchmark, and export the performance results.
113126
## Directory layout
114127

115128
### Specifications
@@ -144,9 +157,169 @@ TBD
144157

145158
TBD
146159

147-
### Adding new test platforms
160+
### Joining the performance initiative and adding a continuous benchmark platform
161+
162+
#### Joining the performance initiative
163+
In order to join the performance initiative the only requirement is that you provide a steady-stable infrastructure
164+
platform to run the benchmark suites, and you reach out to one of the Redis Performance Initiative member via
165+
`performance <at> redis <dot> com` so that we can provide you with the required secrets to actively listen for benchmark events.
166+
167+
If you check the above "Architecture diagram", this means you only need to run the last moving part of the arch, meaning you will have
168+
one or more benchmark coordinator machines actively running benchmarks and pushing the results back to our datasink.
169+
170+
#### Adding a continuous benchmark platform
171+
172+
In order to be able to run the benchmarks on the platform you need pip installer for python3, and docker.
173+
Apart from it, we recommend you manage the `redis-benchmarks-spec-sc-coordinator` process(es) state via a process monitoring tool like
174+
supervisorctl, lauchd, daemon tools, or other.
175+
176+
For this example we relly uppon `supervisorctl` for process managing.
177+
178+
##### Installing package requirements
179+
180+
```bash
181+
# install pip installer for python3
182+
sudo apt install python3-pip -y
183+
sudo pip3 install --upgrade pip
184+
185+
# install docker
186+
sudo apt install docker.io -y
187+
188+
# install supervisord
189+
sudo apt install supervisor -y
190+
191+
# install benchmark specs
192+
python3 -m pip install redis-benchmarks-specification --ignore-installed PyYAML
193+
```
194+
195+
196+
##### Testing out redis-benchmarks-spec-sc-coordinator
197+
198+
You should now be able to print the following installed benchmark runner help:
199+
200+
```bash
201+
$ redis-benchmarks-spec-sc-coordinator -h
202+
usage: redis-benchmarks-spec-sc-coordinator [-h] --event_stream_host
203+
EVENT_STREAM_HOST
204+
--event_stream_port
205+
EVENT_STREAM_PORT
206+
--event_stream_pass
207+
EVENT_STREAM_PASS
208+
--event_stream_user
209+
EVENT_STREAM_USER
210+
[--cpu-count CPU_COUNT]
211+
[--platform-name PLATFORM_NAME]
212+
[--logname LOGNAME]
213+
[--consumer-start-id CONSUMER_START_ID]
214+
[--setups-folder SETUPS_FOLDER]
215+
[--test-suites-folder TEST_SUITES_FOLDER]
216+
[--datasink_redistimeseries_host DATASINK_REDISTIMESERIES_HOST]
217+
[--datasink_redistimeseries_port DATASINK_REDISTIMESERIES_PORT]
218+
[--datasink_redistimeseries_pass DATASINK_REDISTIMESERIES_PASS]
219+
[--datasink_redistimeseries_user DATASINK_REDISTIMESERIES_USER]
220+
[--datasink_push_results_redistimeseries]
221+
222+
redis-benchmarks-spec runner(self-contained) 0.1.13
223+
224+
optional arguments:
225+
-h, --help show this help message and exit
226+
--event_stream_host EVENT_STREAM_HOST
227+
--event_stream_port EVENT_STREAM_PORT
228+
--event_stream_pass EVENT_STREAM_PASS
229+
--event_stream_user EVENT_STREAM_USER
230+
--cpu-count CPU_COUNT
231+
Specify how much of the available CPU resources the
232+
coordinator can use. (default: 8)
233+
--platform-name PLATFORM_NAME
234+
Specify the running platform name. By default it will
235+
use the machine name. (default: fco-ThinkPad-T490)
236+
--logname LOGNAME logname to write the logs to (default: None)
237+
--consumer-start-id CONSUMER_START_ID
238+
--setups-folder SETUPS_FOLDER
239+
Setups folder, containing the build environment
240+
variations sub-folder that we use to trigger different
241+
build artifacts (default: /home/fco/redislabs/redis-
242+
benchmarks-
243+
specification/redis_benchmarks_specification/setups)
244+
--test-suites-folder TEST_SUITES_FOLDER
245+
Test suites folder, containing the different test
246+
variations (default: /home/fco/redislabs/redis-
247+
benchmarks-
248+
specification/redis_benchmarks_specification/test-
249+
suites)
250+
--datasink_redistimeseries_host DATASINK_REDISTIMESERIES_HOST
251+
--datasink_redistimeseries_port DATASINK_REDISTIMESERIES_PORT
252+
--datasink_redistimeseries_pass DATASINK_REDISTIMESERIES_PASS
253+
--datasink_redistimeseries_user DATASINK_REDISTIMESERIES_USER
254+
--datasink_push_results_redistimeseries
255+
uploads the results to RedisTimeSeries. Proper
256+
credentials are required (default: False)
257+
```
258+
259+
Note that the minimum arguments to run the benchmark coordinator are: `--event_stream_host`, `--event_stream_port`, `--event_stream_pass`, `--event_stream_user`
260+
You should use the provided credentials to be able to access the event streams.
261+
Apart from it, you will need to discuss with the Performance Group the unique platform name that will be used to showcase results, coordinate work, among other thigs.
262+
263+
If all runs accordingly you should see the following sample log when you run the tool with the credentials:
264+
265+
```bash
266+
$ poetry run redis-benchmarks-spec-sc-coordinator --platform-name example-platform \
267+
--event_stream_host <...> \
268+
--event_stream_port <...> \
269+
--event_stream_pass <...> \
270+
--event_stream_user <...>
271+
2021-09-22 10:47:12 INFO redis-benchmarks-spec runner(self-contained) 0.1.13
272+
2021-09-22 10:47:12 INFO Using topologies folder dir /home/fco/redislabs/redis-benchmarks-specification/redis_benchmarks_specification/setups/topologies
273+
2021-09-22 10:47:12 INFO Reading topologies specifications from: /home/fco/redislabs/redis-benchmarks-specification/redis_benchmarks_specification/setups/topologies/topologies.yml
274+
2021-09-22 10:47:12 INFO Using test-suites folder dir /home/fco/redislabs/redis-benchmarks-specification/redis_benchmarks_specification/test-suites
275+
2021-09-22 10:47:12 INFO Running all specified benchmarks: /home/fco/redislabs/redis-benchmarks-specification/redis_benchmarks_specification/test-suites/redis-benchmark-full-suite-1Mkeys-100B.yml
276+
2021-09-22 10:47:12 INFO There are a total of 1 test-suites in folder /home/fco/redislabs/redis-benchmarks-specification/redis_benchmarks_specification/test-suites
277+
2021-09-22 10:47:12 INFO Reading event streams from: <...>:<...> with user <...>
278+
2021-09-22 10:47:12 INFO checking build spec requirements
279+
2021-09-22 10:47:12 INFO Will use consumer group named runners-cg:redis/redis/commits-example-platform.
280+
2021-09-22 10:47:12 INFO Created consumer group named runners-cg:redis/redis/commits-example-platform to distribute work.
281+
2021-09-22 10:47:12 INFO Entering blocking read waiting for work.
282+
```
283+
284+
You're now actively listening for benchmarks requests to Redis!
285+
286+
##### Adding redis-benchmarks-spec-sc-coordinator to supervisord
287+
288+
Let's add a supervisord entry as follow
289+
```
290+
vi /etc/supervisor/conf.d/redis-benchmarks-spec-sc-coordinator-1.conf
291+
```
292+
You can use the following template and update according to your credentials:
293+
294+
```bash
295+
[supervisord]
296+
loglevel = debug
297+
298+
[program:redis-benchmarks-spec-sc-coordinator]
299+
command = redis-benchmarks-spec-sc-coordinator --platform-name bicx02 \
300+
--event_stream_host <...> \
301+
--event_stream_port <...> \
302+
--event_stream_pass <...> \
303+
--event_stream_user <...> \
304+
--datasink_push_results_redistimeseries \
305+
--datasink_redistimeseries_host <...> \
306+
--datasink_redistimeseries_port <...> \
307+
--datasink_redistimeseries_pass <...> \
308+
--logname /var/opt/redis-benchmarks-spec-sc-coordinator-1.log
309+
startsecs = 0
310+
autorestart = true
311+
startretries = 1
312+
```
313+
314+
After editing the conf, you just need to reload and confirm that the benchmark runner is active:
315+
316+
```bash
317+
:~# supervisorctl reload
318+
Restarted supervisord
319+
:~# supervisorctl status
320+
redis-benchmarks-spec-sc-coordinator RUNNING pid 27842, uptime 0:00:00
321+
```
148322

149-
TBD
150323

151324
## License
152325

0 commit comments

Comments
 (0)