High-performance AWS Greengrass v2 component for industrial data acquisition using National Instruments DAQ hardware with real-time streaming capabilities.
- {____} AWS Greengrass NIDAQ Component
The {____} NIDAQ component is a high-performance AWS Greengrass v2 component designed for industrial data acquisition using National Instruments (NI) DAQ hardware. It provides real-time data streaming, advanced signal processing, and seamless integration with AWS IoT Device Shadow for remote configuration management.
- High-Speed Data Acquisition: Real-time sampling up to 2 MS/s with minimal latency
- Multi-Channel Support: Simultaneous acquisition from multiple analog/digital channels
- AWS Integration: Native AWS IoT Device Shadow integration for remote configuration
- Signal Processing: Built-in filtering, scaling, and calibration capabilities
- Robust Architecture: Event-driven design with comprehensive error handling
- Industrial Grade: Designed for 24/7 operation in harsh industrial environments
- High-Speed Sampling: Support for sampling rates up to 2 MS/s per channel
- Multi-Channel: Simultaneous acquisition from up to 32 analog input channels
- Flexible Triggering: Software, hardware, and external triggering options
- Data Buffering: Circular buffering with configurable buffer sizes
- Real-Time Processing: Low-latency data processing and streaming
- NI DAQ Devices: Support for all NI-DAQmx compatible devices
- Input Types: Voltage, current, temperature, strain gauge, accelerometer
- Signal Conditioning: Built-in scaling, filtering, and calibration
- Isolation: Galvanic isolation support for industrial environments
- Connector Types: Support for various connector configurations
- Device Shadow: Remote configuration via AWS IoT Device Shadow
- IPC Communication: Native Greengrass IPC for data streaming
- Status Reporting: Real-time status and health monitoring
- Configuration Management: Dynamic configuration updates without restart
- Digital Filtering: Low-pass, high-pass, band-pass, and notch filters
- Calibration: Multi-point calibration with polynomial fitting
- Unit Conversion: Automatic conversion to engineering units
- Data Validation: Range checking and error detection
┌─────────────────┐ ┌───────────────────┐ ┌─────────────────┐
│ NI DAQ │───▶│ NIDAQ Component │───▶│ Stream Filter │
│ Hardware │ │ (Acquisition) │ │ (Output) │
└─────────────────┘ └───────────────────┘ └─────────────────┘
│
┌─────────────────┐
│ Device Shadow │
│ (Configuration)│
└─────────────────┘
-
NIDAQ Module (
src/nidaq_module.py)- NI-DAQmx driver interface and hardware abstraction
- High-speed data acquisition and buffering
- Signal conditioning and processing
- Error handling and recovery mechanisms
-
Shadow Manager (
src/shadow_manager.py)- AWS IoT Device Shadow integration for configuration management
- Real-time configuration updates and synchronization
- Status reporting and health monitoring
- Secure configuration validation
-
Main Service (
src/main.py)- Component lifecycle management and orchestration
- IPC communication setup and data streaming
- Signal handling and graceful shutdown
- Performance monitoring and logging
- Hardware Configuration: Configure DAQ hardware based on shadow settings
- Data Acquisition: Continuous data sampling from configured channels
- Signal Processing: Apply filters, scaling, and calibration
- Data Streaming: Publish processed data via Greengrass IPC
- Status Monitoring: Report acquisition status to device shadow
- NI DAQ Device: Any NI-DAQmx compatible data acquisition device
- Processor: x86_64 architecture (Intel/AMD)
- Memory: Minimum 2GB RAM (4GB+ recommended for high-speed acquisition)
- Storage: Sufficient space for logs and temporary buffers
- USB/PCI: Available connection for DAQ device
- Operating System: Linux amd64 (tested on Ubuntu 24.04.3 LTS)
- Python: 3.12 or higher
- NI-DAQmx: Version 2024Q1 or higher (tested on 2025Q2, 2025Q3)
- AWS Greengrass Core: Nucleus Classic v2.15.0 or higher
- CompactDAQ: cDAQ-9171, cDAQ-9172, cDAQ-9174, cDAQ-9178
https://www.ni.com/zh-tw/support/downloads/drivers/download.ni-daq-mx.html
uname -a-
Open the Command Prompt and run the following commands to apply the latest system updates to avoid installation issues.
sudo apt update sudo apt dist-upgrade
-
Restart your system.
-
Open the Command Prompt and run the following command to install the repository registration package.
sudo apt install ./{filename}.debWhere filename.deb is the repository registration package you chose in Installing NI Drivers and Software on Linux Desktop.
-
Run the following command to refresh the package list.
sudo apt update
-
Install the following basic Ubuntu packages on the target system.
- dkms
- expat
- libopenal1
sudo apt install dkms expat libopenal1
-
Run the following command for each driver or application you want to install.
# sudo apt install {package-name}, e.g. ni-daqmx in this project. sudo apt install ni-daqmx -
Run the following command to build NI kernel drivers.
sudo dkms autoinstall
-
Restart your system.
# Clone the repository
git clone <repository-url>
cd <repository-name>
# Setup development environment
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python3 -m pip install -U git+https://github.com/aws-greengrass/aws-greengrass-gdk-cli.git@v1.6.2# Deploy locally (for dev & testing)
./local_deploy.sh
# Remove local deployment
./remove_local_deploy.sh# Build component package
gdk component buildgdk component publishsudo /greengrass/v2/bin/greengrass-cli pubsub sub -t /TOPIC_NAME/Where
TOPIC_NAMEis the name of the topic you want to subscribe to.
Branch model (left to right = promotion path): feature branch → dev → staging → main
Environments and version patterns:
- feature branches: isolated feature / fix work (no version tagging)
- dev: integration / active development (pre-release versions vX.Y.Z-alpha.N)
- staging: pre-production verification (pre-release versions vX.Y.Z-beta.N)
- main: production (stable releases vX.Y.Z)
Rules:
- Only merge into dev via PR from feature branches
- staging only receives squash merges from dev
- main only receives fast-forward or squash from staging (no direct commits)
- Hotfix (critical) may branch from main, then merge back to staging and dev
semantic-release (single source of truth) runs on push to dev, staging, main:
- Analyze commits (Conventional Commits) to determine version bump:
- BREAKING CHANGE: / feat! → major
- feat: → minor
- fix / perf / refactor: → patch
- Other types (docs, chore, ci, style, test) → no version bump
- Prepare phase writes the computed version to:
- gdk-config.json (component version)
- recipe.yaml (ComponentVersion)
- Generate changelog + Git tag:
- dev → vX.Y.Z-alpha.N
- staging → vX.Y.Z-beta.N
- main → vX.Y.Z
- Create GitHub Release / Pre-release with artifacts
- Build & publish to AWS Greengrass ONLY for staging and main
- No separate version-bump workflow; remove any legacy bumps
ci.yml (runs on every PR and push):
- Formatting / style: black, isort, flake8
- (Soft) typing: mypy (non-blocking initially)
- JSON / YAML schema & syntax validation
- Greengrass component structure & recipe consistency checks
- (Soft) security & secret scanning (upgradeable to blocking later)
- Fails fast on critical structural or syntax errors
Format: [optional scope]:
[optional body]
[optional footer(s)]
Common types:
- feat: new feature (triggers minor)
- fix: bug fix (patch)
- perf: performance improvement (patch)
- refactor: code change w/o behavior change (patch)
- docs, style, test, chore, ci: no version bump by default
Breaking changes:
- Use feat! / refactor! suffix OR
- Add footer: BREAKING CHANGE: explanation
Recommended workflow:
- Use squash merge
- Ensure PR title is a valid Conventional Commit (final squashed commit = PR title)
Examples: feat: add new sensor data filtering algorithm fix: resolve memory leak in data processing loop feat!: change API response format feat: change API response format BREAKING CHANGE: response changed from array to object
Automated (preferred):
- Develop on feature branch → open PR to dev
- Ensure commits / PR title comply with Conventional Commits
- Merge → semantic-release assigns pre-release (alpha) on dev
- Promote dev → staging → beta release (build + publish to Greengrass)
- Promote staging → main → stable release (build + publish)
Manual trigger (if needed):
- Go to Actions: "Release (semantic-release → build → publish)"
- Select target branch (dev / staging / main)
- Workflow auto-detects:
- dev: pre-release alpha (no publish to AWS)
- staging: pre-release beta (publish)
- main: stable (publish)
Safeguards:
- Never edit versions manually
- All version sources are derived; commit modified version files produced by workflow only
- Avoid tagging manually to prevent semantic-release drift
Operational Notes:
- alpha churn is expected; do not pin production to alpha/beta
- Consumers should pin only stable main releases unless testing
- Monitor release notes for BREAKING CHANGE sections
Copyright (c) 2025 imCloud Co., Ltd. All rights reserved.
This software is proprietary and confidential. Unauthorized copying, modification, distribution, or use of this software is strictly prohibited.