v1.3: reset_times, logfile size limit

Author: davidplotzki

CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## 1.3 (2025-07-13)
++ New config option for logbook columns: `reset_times` to reset data at specific times of day.
++ New config option: `logfile_max_lines` to limit logfile size (defaults to 10000 lines).
++ Bugfix: crash when reading big integers from JSON structures.
+
 ## 1.2 (2022-10-03)
 + Added sensor retry time (`retry_time` and `default_retry_time`): time for another attempt after a failed sensor reading.
 + New config options for `general`: `http_timeout`, `default_rest_period`, `default_retry_time`, `loglevel`.

README.md

@@ -19,7 +19,7 @@ In the following listing you see a very simple example for a logbook file. It ke
 	2021-01-31 18:00:00    -0.926       89.06
 	2021-01-31 18:30:00    -1.14067     89.1933
 
-## Data accumu­lation and statisti­cal summary
+## Data accumulation and statistical summary
 
 Sensorlogger is able to read values from Tinkerforge sensors and JSON structures in user-defined **time intervals.** After reading, these values can be forwarded immediately to MQTT brokers or to a HomeMatic CCU (the XML-API is used to set system variables using their ISE-ID). The values are also stored internally until they become irrelevant for any further statistical analysis.
 
@@ -68,7 +68,7 @@ The virtual package `libcurl-dev` offers different implementations. All of them
 
 ### MQTT Support
 
-If you need support for MQTT, the Paho MQTT libraries for C and C++ must be installed. You can get packages for Arch-based distributions for the AUR: [paho-mqtt-c-git](https://aur.archlinux.org/packages/paho-mqtt-c-git/) and [paho-mqtt-cpp-git](https://aur.archlinux.org/packages/paho-mqtt-cpp-git/). For Debian-based systems (e.g. Raspberry Pi, Tinkerforge Red Brick) we have to compile and install the libraries on our own. To do this, we need to make sure that we already have all the required packages installed:
+If you need support for MQTT, the Paho MQTT libraries for C and C++ must be installed. You can get packages for Arch-based distributions from the AUR: [paho-mqtt-c](https://aur.archlinux.org/packages/paho-mqtt-c/) and [paho-mqtt-cpp](https://aur.archlinux.org/packages/paho-mqtt-cpp/). There also exist [Debian packages](https://packages.debian.org/search?suite=stable&section=all&arch=any&searchon=names&keywords=paho-mqtt) for newer distributions. If not, we have to compile and install the libraries on our own. To do this, we need to make sure that we already have all the required packages installed:
 
     apt-get install libssl-dev build-essential gcc make
     apt-get install cmake cmake-gui cmake-curses-gui
@@ -168,6 +168,7 @@ The `general` section of the configuration file may contain the following settin
 ```
 "general": {
     "logfile": "/home/username/sensorlogger.log",
+    "logfile_max_lines": 10000,
     "loglevel": "info",
     "http_timeout": 20,
     "default_rest_period": {"value": 60, "unit": "s"},
@@ -179,6 +180,12 @@ The `general` section of the configuration file may contain the following settin
 
     Standard value: `null`
 
++ `"logfile_max_lines":` Maximum number of lines in the log file. If the log file grows bigger than this limit, it will be trimmed by removing the oldest lines from the top. Note that for performance reasons, the log file is only trimmed when Sensorlogger starts and then every time after 100 lines have been written.
+
+    Set this value to `0` to deactivate trimming of the log file.
+
+    Standard value: `10000`
+
 + `"loglevel":` Type of events that are logged. Can be any of the following:
     - `"error"`: Only log error messages.
     - `"warning"`: Log errors and warnings.
@@ -746,12 +753,14 @@ The `"columns"` key (as a child node of a logbooks object) always hosts a JSON a
         "confidence_absolute": 10.0,
     },
     {
-        "title": "Humidity",
+        "title": "Maximum humidity today",
         "unit":  "%rel",
         "sensor_id": "Weather/Humidity",
-        "mqtt_publish": "House/Weather/Humidity/median",
+        "mqtt_publish": "House/Weather/Humidity/max24h",
         "homematic_publish": "37856",
-        "operation": "median",
+        "operation": "max",
+        "evaluation_period": {"value": 24, "unit": "h"},
+        "reset_times": ["0:00"],
         "confidence_sigma": 3.0
     },
     {
@@ -803,6 +812,10 @@ The `"columns"` key (as a child node of a logbooks object) always hosts a JSON a
 
     Standard value: `null`
 
++ `"reset_times":` Specific points in time when the column values are reset, i.e., the value cache for this column is cleared. For example, when an evaluation period is defined to calculate a moving average, the accumulated data can be cleared at midnight such that the previous day is not part of the moving average. Likewise, a daily maximum for a value can be computed.
+
+    The reset times are listed in an array. An arbitrary number of reset times can be defined. Each reset time is a string that contains hour and minute in 24-hour format, such as `"14:30"`. To repeat a reset at a certain minute every hour, the star notation can be used: `"*:00"` does a reset every full hour.
+
 + `"count_factor":` This factor can be used to weight the number of events of a pulse counter. It only affects the counter operations `"count"`, `"freq"`, `"freq_min"` and `"freq_max"`. In the example above, the wind sensor triggers two pulses per rotation, so we scale the pulse frequency by a factor of 0.5 to get the rotation frequency.
 
     Standard value: `1`
@@ -832,7 +845,7 @@ The `"columns"` key (as a child node of a logbooks object) always hosts a JSON a
 
 ### Tinkerforge: Supported Bricklets
 
-Currently, version 2.1.33 (2022-05-11) of the [Tinkerforge](https://www.tinkerforge.com/) C/C++ bindings are used. The following Tinkerforge Bricklets are supported. The table also lists the channel IDs if a Bricklet supports different types of measurements.
+Currently, version 2.1.34 (2024-02-27) of the [Tinkerforge](https://www.tinkerforge.com/) C/C++ bindings are used. The following Tinkerforge Bricklets are supported. The table also lists the channel IDs if a Bricklet supports different types of measurements.
 
 | Bricklet (and versions)             | Channels                                          |
 | :---------------------------------- | :------------------------------------------------ |

include/column.h

@@ -2,13 +2,15 @@
 #define _COLUMN_H
 
 #define E_NO_VALUES_FOR_COLUMN  7002
+#define E_INVALID_RESET_TIME    7003
 
+#include <cstdint>
 #include <string>
-#include <sstream>
 #include <vector>
 
 enum operation {mean, median, max, min, sum, count, freq, freq_min, freq_max, stdDevMean, stdDevMedian};
 
+class logger;
 class logbook;
 class sensor;
 class counter;
@@ -20,26 +22,30 @@ class column
 	counter*      _pulseCounter;
 	std::string   _title;
 	std::string   _unit;
-	uint64_t      _evaluationPeriod;
+	uint64_t      _evaluationPeriod;  // in ms
+	std::vector<uint64_t> _evaluationPeriod_resetHours;
+	std::vector<uint64_t> _evaluationPeriod_resetMinutes;
 	operation     _op;
 	double        _confidenceSigma;
 	double        _confidenceAbsolute;
 	std::string   _mqttPublishTopic;
 	std::string   _homematicPublishISE;
 	double        _countFactor;
 
+	logger*       _root;
 	logbook*      _rootLogbook;
 
 	// Calculated from evaluation period / logbook cycle time:
 	size_t        _nCycles;  // How many logbook cycles does the evaluation period represent?
 
 public:
-	column(sensor* s, logbook* lb, const std::string &title, const std::string &unit, uint64_t evaluationPeriod, operation op, double confidenceAbsolute, double confidenceSigma, const std::string &mqttPublishTopic, const std::string &homematicPublishISE, double countFactor);
+	column(logger* root, sensor* s, logbook* lb, const std::string &title, const std::string &unit, uint64_t evaluationPeriod, operation op, double confidenceAbsolute, double confidenceSigma, const std::string &mqttPublishTopic, const std::string &homematicPublishISE, double countFactor);
 
 	std::string getTitle() const;
 	std::string getUnit() const;
 	operation getOperation() const;
 	uint64_t getEvaluationPeriod() const;
+	uint64_t getEvaluationPeriod_consideringResetTimes(int current_hour, int current_minute, int current_second) const;
 	double getConfidenceAbsolute() const;
 	double getConfidenceSigma() const;
 	std::string getMQTTPublishTopic() const;
@@ -53,6 +59,8 @@ class column
 	void setUnit(const std::string &unit);
 	void setOperation(operation op);
 	void setEvaluationPeriod(uint64_t evaluationPeriod);
+	void addEvaluationPeriodResetTime(std::string resetTime);
+	void addEvaluationPeriodResetTime(uint64_t hour, uint64_t minute);
 	void calculateColumnCycles();
 	void setConfidenceAbsolute(double confidenceAbsolute);
 	void setConfidenceSigma(double confidenceSigma);

include/homematic.h

@@ -12,7 +12,7 @@ class homematic
 private:
 	std::string _xmlAPI_URL;  // Full Homematic XML API URL, including http(s)
 
-	logger*     _root;
+	logger* _root;
 
 public:
 	homematic(logger* root);

include/json.h

@@ -1,13 +1,10 @@
 #ifndef _JSON_H
 #define _JSON_H
 
+#include <cstdint>
 #include <string>
 #include <cstring>
-#include <sstream>
-#include <fstream>
 #include <vector>
-#include <iostream>
-#include <cmath>
 
 #define E_VALUE_IS_NOT_BOOL               2001
 #define E_VALUE_IS_NOT_INT                2002
@@ -31,15 +28,15 @@
 enum jsonType {jsonNull, jsonBool, jsonInt, jsonDouble, jsonString};
 
 bool isIn(const char& c, const char* chain);
-bool isIntInString(const std::string &s);
-bool isFloatInString(const std::string &s);
+bool isIntInString(const std::string *s);
+bool isFloatInString(const std::string *s);
 
 class jsonValue
 {
 private:
 	jsonType    _type;
 	bool        _vBool;
-	long        _vInt;
+	long long   _vInt;
 	double      _vDouble;
 	std::string _vString;
 
@@ -49,13 +46,13 @@ class jsonValue
 
 	void setNull();
 	void setBool(bool v);
-	void setInt(long v);
+	void setInt(long long v);
 	void setDouble(double v);
 	void setString(const std::string &v);
 
 	bool        isNull() const;
 	bool        getBool() const;
-	long        getInt() const;
+	long long   getInt() const;
 	double      getDouble() const;
 	std::string getString() const;
 

include/logbook.h

@@ -1,23 +1,19 @@
 #ifndef _LOG_H
 #define _LOG_H
 
+#include <cstdint>
 #include <cstdio>
 #include <vector>
 #include <string>
-#include <iostream>
 #include <ctime>
-#include <chrono>
-#include <iomanip>
-#include <sstream>
-#include <fstream>
 
 class sensor;
 class counter;
 class mqttManager;
 class homematic;
 class logbook;
-class logger;
 class column;
+class logger;
 
 #define E_COLUMN_DOES_NOT_EXIST 5001
 

include/logger.h

@@ -6,16 +6,13 @@
 #define E_JSON_READ             6003
 #define E_HTTP_REQUEST_FAILED   6004
 
+#include <cstdint>
 #include <cstdio>
-#include <iostream>
-#include <iomanip>
-#include <sstream>
 #include <string>
 #include <vector>
-#include <thread>
 
 #ifdef OPTION_CURL
-	#include <curl/curl.h>
+#include <curl/curl.h>
 #endif
 
 enum loglevel {
@@ -40,6 +37,10 @@ class logger
 	std::string _lastLogfileMessage;
 	int _logLevel;
 
+	// Trim logfile every
+	size_t _logfile_max_lines;
+	size_t _logfile_trim_counter;
+
 	uint64_t _default_rest_period;
 	uint64_t _default_retry_time;
 
@@ -80,6 +81,10 @@ class logger
 	std::string logLevelString();
 	void welcomeMessage(const std::string &configJSON);
 
+	static void removeWhiteSpace(std::string *s);
+
+	void trim_logfile() const;
+
 	void loadConfig(const std::string &configJSON);
 
 	std::string getLogFilename() const;

include/measurements.h

@@ -5,8 +5,6 @@
 
 #include <cstdint>
 #include <vector>
-#include <cmath>
-#include <algorithm>
 
 uint64_t timeDiff(uint64_t timestamp1, uint64_t timestamp2);
 

include/mqttbroker.h

@@ -3,7 +3,6 @@
 
 #ifdef OPTION_MQTT
 
-#include <sstream>
 #include "mqtt/async_client.h"
 
 class logger;

include/readoutbuffer.h

@@ -6,12 +6,9 @@
 #define BUFFERTIME 800  // milliseconds
 #define N_TRIALS 2
 
+#include <cstdint>
 #include <string>
-#include <sstream>
-#include <iostream>
-#include <fstream>
 #include <vector>
-#include <thread>
 
 class logger;