You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The definitive all-in-one [SnapRAID](https://github.com/amadvance/snapraid) script.
2
+
The definitive all-in-one [SnapRAID](https://github.com/amadvance/snapraid) script. I hope you'll agree.
3
3
4
-
There are many SnapRAID scripts out there, but none could fit my needs. So I took the best of them to start a new one.
4
+
There are many SnapRAID scripts out there, but none has the features I want. So I made my own, inspired by existing solutions.
5
5
6
-
It is meant to be run periodically (e.g. daily) and do the heavy lifting, then send an email you will actually read.
6
+
It is meant to be run periodically (daily), do the heavy lifting and send an email you will actually read.
7
7
8
-
Supports single and dual parity configurations.
8
+
Supports single and dual parity configurations. It is highly customizable and has been tested with Debian 10 and [OpenMediaVault 5](https://github.com/openmediavault/openmediavault).
9
9
10
-
It is customizable and has been tested with Debian 10 and [OpenMediaVault 5](https://github.com/openmediavault/openmediavault).
11
-
12
-
Contributions are welcome: there's always room for improvement!
13
-
14
-
_This readme has some rough edges which will be smoothened over time._
10
+
Contributions are welcome!
15
11
16
12
# Highlights
17
13
@@ -21,84 +17,100 @@ _This readme has some rough edges which will be smoothened over time._
21
17
- If parity info is out of sync **and** the number of deleted or changed files exceed the threshold you have configured it **stops**. You may want to take a look to the output log.
22
18
- If parity info is out of sync **and** the number of deleted or changed files exceed the threshold, you can still **force a sync** after a number of warnings. It's useful If you often get a false alarm but you're confident enough. This is called "Sync with threshold warnings"
23
19
- If parity info is out of sync **but** the number of deleted or changed files did not exceed the threshold, it **executes a sync** to update the parity info.
24
-
- When the parity info is in sync, either because nothing has changed or after a successfully sync, it runs the `snapraid scrub` command to validate the integrity of the data, both the files and the parity info. If sync was cancelled or other issues were found, scrub will not be run. _Note that each run of the scrub command will validate only a configurable portion of parity info to avoid having a long running job and affecting the performance of the server._
25
-
- Extra information is be added, like SnapRAID's disk health report.
20
+
- When the parity info is in sync, either because nothing has changed or after a successfully sync, it runs the `snapraid scrub` command to validate the integrity of the data, both the files and the parity info. If sync was cancelled or other issues were found, scrub will not be run. _Note that each run of the scrub command will validate only a configurable portion of parity info to avoid having a long running job and affecting the performance of the server._ Scrub frequency can also be customized in case you don't want to do it every time the script runs. It is still recommended to run scrub frequently.
21
+
- Extra information can be added, like SnapRAID's disk health report o SnapRAID array status.
26
22
- When the script is done sends an email with the results, both in case of error or success.
27
23
24
+
### Additional Information
25
+
- Docker container management, if enabled, will manage containers before SnapRAID activity and restore them when finished. It avoids nasty errors aboud data being written during SnapRAID sync.
26
+
- You can either choose to pause or stop your containers and manage a remote docker host.
27
+
- Healthchecks.io can be used to track script execution time and to promptly alert about errors.
28
+
- Important messages are sent to the system log.
29
+
28
30
## Customization
29
-
Many options can be changed to your taste, their behavior is documented in the script config file.
31
+
Many options can be changed to your taste, their behavior is documented in the config file.
30
32
If you don't know what to do, I recommend using the default values and see how it performs.
31
33
32
34
### Customizable features
33
35
- Sync options
34
-
- Sync always (forced sync)
35
-
- Sync after a number of breached threshold warnings
36
-
- Sync only if thresholds warnings are not breached (enabled by default)
37
-
- Thresholds for deleted and updated files
36
+
- Sync always (forced sync).
37
+
- Sync after a number of breached threshold warnings.
38
+
- Sync only if thresholds warnings are not breached (enabled by default).
39
+
- User definable thresholds for deleted and updated files.
38
40
- Scrub options
39
-
- Enable or disable scrub
40
-
- Data to be scrubbed - by default 5% older than 10 days
41
-
- Pre-hashing - enabled by default to avoid silent read errors. It mitigates the lack of ECC memory.
42
-
- SMART Log - enabled by default, a SnapRAID report for disks health status
43
-
- Verbosity - disabled by default, does not include the TOUCH and DIFF output to have a better email
44
-
- Spindown - to spindown drives after the script, disabled because is currently not working
45
-
- Snapraid Status - show the status of the array, disabled because the report output is not rendered correctly
41
+
- Enable or disable scrub job.
42
+
- Delayed option, disabled by default. Run scrub only after a number of script executions, e.g. every 7 times. If you don't want to scrub your array every time, this one is for you.
43
+
- Data to be scrubbed - by default 5% older than 10 days.
44
+
- Pre-hashing - enabled by default. Mitigate the lack of ECC memory, reading data twice to avoid silent read errors.
45
+
- SMART Log - enabled by default. A SnapRAID report for disks health status.
46
+
- Healthchecks.io integration
47
+
- Script execution result can be reported to Healthchecks.io.
48
+
- If the script ends with a **_WARNING_** message, it will be reported **_DOWN_** to Healthchecks.io, instead if the message is **_COMPLETED_** it will be **_UP_**. If you don't read your emails every day, this is a great one for you, since the service can report you in other ways.
49
+
- The reason of the failure, which is the email subject, is included as well.
50
+
- Container management - disabled by default.
51
+
- A list of containers you want to be interrupted before running actions and restored when completed.
52
+
- Docker mode - choose to pause/unpause or to stop/restart your containers
53
+
- Docker remote - if docker is running on a remote machine
54
+
- Verbosity option - disabled by default. When enabled, includes the TOUCH and DIFF commands output. Please note email will be huge and mostly unreadable.
55
+
- Spindown - spindown drives after the script, disabled because is currently not working.
56
+
- Snapraid Status - shows the status of the array, disabled by default.
46
57
47
-
48
58
You can also change more advanced options such as mail binary (by default uses `mailx`), SnapRAID binary location, log file location.
49
59
50
60
## A nice email report
51
61
This report produces emails that don't contain a list of changed files to improve clarity.
52
62
53
-
You can re-enable full output in the email by switching the option `VERBOSITY` but the full report will always be available in `/tmp/snapRAID.out` but will be replaced after each run, or deleted when the system is shut down. You can change the location of the file, if needed.
63
+
You can re-enable full output in the email by switching the option `VERBOSITY` but the full report will always be available in `/tmp/snapRAID.out` but will be replaced after each run, or deleted when the system is shut down. You can change the location of the file if you need to keep it.
64
+
65
+
Here's a sneak peek of the email report.
54
66
55
-
Here's a sneak peek of the email report.
56
67
57
68
```markdown
58
69
## [COMPLETED] DIFF + SYNC + SCRUB Jobs (SnapRAID on omv-test.local)
59
-
60
-
SnapRAID Script Job started [Sat Jan 9 02:07:46 CET 2021]
61
-
Running SnapRAID version 11.5
62
-
SnapRAID Script version 2.7.0
70
+
SnapRAID Script Job started [Tue 20 Apr 11:43:37 CEST 2021]
71
+
Running SnapRAID version 11.5
72
+
SnapRAID AIO Script version X.YZ
63
73
64
74
----------
65
75
66
76
## Preprocessing
67
-
68
-
Configuration file found! Proceeding.
69
-
Testing that all parity files are present.
70
-
All parity files found. Continuing...
77
+
Healthchecks.io integration is enabled.
78
+
Configuration file found.
79
+
Checking if all parity and content files are present.
80
+
All parity files found.
81
+
All content files found.
82
+
Docker containers management is enabled.
83
+
84
+
### Stopping Containers [Tue 20 Apr 11:43:37 CEST 2021]
85
+
Stopping Container - Code-server
86
+
code-server
87
+
Stopping Container - Portainer
88
+
portainer
71
89
72
90
----------
73
91
74
92
## Processing
75
-
76
-
### SnapRAID TOUCH [Sat Jan 9 02:07:46 CET 2021]
77
-
78
-
Checking for zero sub-second files.
79
-
No zero sub-second timestamp files found.
80
-
TOUCH finished [Sat Jan 9 02:07:46 CET 2021]
81
-
82
-
### SnapRAID DIFF [Sat Jan 9 02:07:46 CET 2021]
83
-
84
-
DIFF finished [Sat Jan 9 02:07:46 CET 2021]
85
-
86
-
**SUMMARY of changes - Added [2] - Deleted [0] - Moved [0] - Copied [0] - Updated [0]**
87
-
93
+
### SnapRAID TOUCH [Tue 20 Apr 11:43:37 CEST 2021]
94
+
Checking for zero sub-second files.
95
+
No zero sub-second timestamp files found.
96
+
TOUCH finished [Tue 20 Apr 11:43:38 CEST 2021]
97
+
98
+
### SnapRAID DIFF [Tue 20 Apr 11:43:38 CEST 2021]
99
+
DIFF finished [Tue 20 Apr 11:43:38 CEST 2021]
100
+
**SUMMARY of changes - Added [0] - Deleted [0] - Moved [0] - Copied [0] - Updated [1]**
88
101
There are no deleted files, that's fine.
89
-
There are no updated files, that's fine.
90
-
SYNC is authorized.
91
-
92
-
### SnapRAID SYNC [Sat Jan 9 02:07:46 CET 2021]
102
+
There are updated files. The number of updated files (1) is below the threshold of (500).
103
+
SYNC is authorized. [Tue 20 Apr 11:43:38 CEST 2021]
93
104
105
+
### SnapRAID SYNC [Tue 20 Apr 11:43:38 CEST 2021]
94
106
Self test...
95
107
Loading state from /srv/dev-disk-by-label-DISK1/snapraid.content...
96
108
Scanning disk DATA1...
97
109
Scanning disk DATA2...
98
110
Using 0 MiB of memory for the file-system.
99
111
Initializing...
100
112
Hashing...
101
-
SYNC_JOB--Everything OK
113
+
SYNC - Everything OK
102
114
Resizing...
103
115
Saving state to /srv/dev-disk-by-label-DISK1/snapraid.content...
104
116
Saving state to /srv/dev-disk-by-label-DISK2/snapraid.content...
@@ -113,20 +125,18 @@ Verified /srv/dev-disk-by-label-DISK3/snapraid.content in 0 seconds
113
125
Verified /srv/dev-disk-by-label-DISK2/snapraid.content in 0 seconds
114
126
Verified /srv/dev-disk-by-label-DISK1/snapraid.content in 0 seconds
The FP column is the estimated probability (in percentage) that the disk
173
-
is going to fail in the next year.
202
+
is going to fail in the next year.
203
+
Probability that at least one disk is going to fail in the next year is 0%.
204
+
205
+
## Restarting Containers [Tue 20 Apr 11:43:41 CEST 2021]
174
206
175
-
Probability that at least one disk is going to fail in the next year is 0%.
176
-
All jobs ended. [Sat Jan 9 02:07:49 CET 2021]
177
-
Email address is set. Sending email report to example@example.com [Sat Jan 9 02:07:49 CET 2021]
207
+
Restarting Container - Code-server
208
+
code-server
209
+
Restarting Container - Portainer
210
+
portainer
211
+
All jobs ended. [Tue 20 Apr 11:43:41 CEST 2021]
212
+
Email address is set. Sending email report to yourmail@example.com [Tue 20 Apr 11:43:41 CEST 2021]
178
213
```
179
214
180
215
# Requirements
181
216
- Markdown to have nice emails - will be installed if not found
182
-
-~~Hd-idle to spin down disks - [Link TBD] - currently not required since spin down does not work properly.~~
217
+
- Hd-idle to spin down disks - [Link TBD] - currently not required since spin down does not work properly.
183
218
184
219
# Installation
185
-
If you want to use this script on OMV, don't worry about the section _Diff Script Settings_ in the main page of the SnapRAID plugin. These settings only apply to the plugin built-in script. Also don't forget to remove from scheduling the built-in script.
186
220
187
221
1. Install markdown `apt install python-markdown`. You can skip this step since the script will check and install it for you.
188
-
2. Download config file and script, then place wherever you prefer e.g. `/usr/sbin/snapraid`
222
+
2. Download the zip and extract wherever you prefer e.g. `/usr/sbin/snapraid`
189
223
3. Give executable rights to the main script - `chmod +x snapraid-aio-script.sh`
190
224
4. Edit the config file and add your email address at line 9
191
-
5.Tweak the config file if needed
192
-
6. Schedule the script execution time
225
+
5.Make other changes to the config file as required
226
+
6. Schedule the script execution time
193
227
194
228
It is tested on OMV5, but will work on other distros. In such case you may have to change the mail binary or SnapRAID location.
195
229
196
-
If you want to use this script on OMV, don't worry about the section _Diff Script Settings_ in the main page of the SnapRAID plugin, since it only applies to the built-in plugin script. Also don't forget to remove from scheduling the built-in script.
230
+
**OMV5 and SnapRAID plugin**
231
+
Ignore the page at OMV GUI > Services > SnapRAID > Diff Script Settings, since it only applies to the plugin's built-in script. Also don't forget to remove from scheduling such built-in script.
197
232
198
233
# Known Issues
199
234
- Hard disk spin down does not work: they are immediately woken up. The script probably does not handle this correctly while running.
200
-
- The report is not perfect, we can't be solve this because SnapRAID does not natively support Markdown.
201
235
202
236
# Credits
203
237
All rights belong to the respective creators.
204
-
Thanks to:
238
+
This script would not exist without:
205
239
-[Zack Reed](https://zackreed.me/snapraid-split-parity-sync-script/) for most of the original script
206
240
-[mtompkins](https://gist.github.com/mtompkins/91cf0b8be36064c237da3f39ff5cc49d) for most of the original script
207
241
-[sburke](https://zackreed.me/snapraid-split-parity-sync-script/#comment-300) for the Debian 10 fix
208
-
- metagliatore (a friend, not on Github) for removing the DIFF output from the email
242
+
- metagliatore (a friend not on Github) for helping out on several BASH issues
0 commit comments