-
Notifications
You must be signed in to change notification settings - Fork 0
Development Workflows Debugging and Troubleshooting
Alex J Lennon edited this page Oct 10, 2025
·
1 revision
⏱️ Time Required: 5-30 minutes per issue
Prerequisites: Basic familiarity with Linux command line and board hardware
# 1. Check power supply
# Requirement: 5V 2A minimum, stable power
# Test: Try different USB-C cable and power adapter
# 2. Check serial console connection
ls /dev/ttyUSB* # Should show ttyUSB0 or ttyUSB1
screen /dev/ttyUSB0 115200 # Connect to console
# Expected: U-Boot messages, then Linux boot log
# 3. Check for boot loop
# Symptoms: Repeating U-Boot messages, never reaches Linux
# Fix: Reflash bootloader and image
./scripts/fio-program-board.sh --machine [your-machine] --program# 1. Check WiFi interface
ip link show wlan0 # Should show UP state
# If DOWN: ip link set wlan0 up
# 2. Scan for networks
nmcli dev wifi list # Should show available networks
# If empty: Check antenna connections, reboot board
# 3. Connect to network
nmcli dev wifi connect "NetworkName" password "password"
ping -c 3 8.8.8.8 # Test internet connectivity# 1. Check SSH service
systemctl status sshd # Should show active (running)
# If not: systemctl start sshd
# 2. Check firewall
iptables -L # Look for blocking rules
# If blocked: iptables -F (temporary fix)
# 3. Check network configuration
ip addr show # Get current IP address
# Connect via serial console if SSH fails# Serial console shows U-Boot but stops there
# Common causes:
# - Missing or corrupted kernel/device tree
# - Wrong bootargs
# - Storage device not detected
# Debug commands in U-Boot:
# printenv # Show environment variables
# mmc list # Show detected storage
# fatls mmc 0:1 # List boot partition files
# boot # Manually trigger boot# Kernel starts but panics or stops
# Check dmesg output via serial console
# Common patterns:
# "Kernel panic - not syncing: VFS: Unable to mount root fs"
# Fix: Check rootfs partition, reflash image
# "Unable to handle kernel paging request"
# Fix: Hardware issue or wrong device tree
# "Failed to execute /sbin/init"
# Fix: Corrupted rootfs, reflash image# System boots but services fail
systemctl --failed # Show failed services
journalctl -xb # Show boot log with details
# Common service failures:
systemctl status NetworkManager # Network issues
systemctl status sshd # SSH access issues
systemctl status bluetooth # Bluetooth problems# Scan I2C buses for devices
i2cdetect -y 1 # Scan bus 1
i2cdetect -y 2 # Scan bus 2
# Expected devices (board-specific):
# 0x44 - SHT40 temperature/humidity sensor
# 0x4C - TAS2563 audio codec (Edge AI)
# 0x28 - LP5024 LED driver (Edge AI)
# If devices missing:
# 1. Check power supply stability
# 2. Check device tree I2C configuration
# 3. Check physical connections/soldering# Check GPIO configuration
cat /sys/kernel/debug/gpio # Show all GPIO states
# Test GPIO manually
echo 555 > /sys/class/gpio/export # Export GPIO 555
echo out > /sys/class/gpio/gpio555/direction
echo 1 > /sys/class/gpio/gpio555/value # Set high
echo 0 > /sys/class/gpio/gpio555/value # Set low
# Check GPIO interrupts
cat /proc/interrupts | grep gpio# Check audio devices
aplay -l # List playback devices
arecord -l # List capture devices
# Test audio playback
speaker-test -t sine -f 1000 -c 1 # Generate test tone
# Check ALSA mixer
amixer scontrols # Show available controls
amixer sget Master # Get master volume
# Audio codec specific (TAS2563)
i2cget -y 1 0x4C 0x00 # Read codec register# WiFi debugging
dmesg | grep -i wifi # Check WiFi driver messages
lsmod | grep wifi # Check loaded WiFi modules
rfkill list # Check RF kill switches
# Bluetooth debugging
dmesg | grep -i bluetooth # Check BT driver messages
hciconfig -a # Show BT interface status
bluetoothctl # Interactive BT control
# Power management check (i.MX93)
cat /sys/kernel/debug/gpio | grep -E "(632|633|634)" # BT/WiFi power GPIOs# Analyze boot performance
systemd-analyze # Overall boot time
systemd-analyze blame # Services by time taken
systemd-analyze critical-chain # Critical path analysis
# Target boot times:
# i.MX8MM: < 20 seconds
# i.MX93: < 15 seconds (optimized)# Check CPU frequency scaling
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
# Monitor CPU usage
htop # Interactive process monitor
iostat 1 # I/O and CPU statistics
# Check thermal throttling
cat /sys/class/thermal/thermal_zone*/temp
# Temperatures > 85°C may cause throttling# Check memory usage
free -h # Memory summary
cat /proc/meminfo # Detailed memory info
# Check for memory leaks
ps aux --sort=-%mem | head -10 # Top memory consumers
cat /proc/slabinfo # Kernel memory usage
# Out of memory events
dmesg | grep -i "killed process" # OOM killer messages# Enable kernel debugging (add to kernel command line)
# debug loglevel=8 dyndbg="file drivers/net/wireless/nxp/mxm_wifiex/* +p"
# Check kernel logs
dmesg -w # Watch kernel messages live
journalctl -k -f # Follow kernel journal
# Kernel crash debugging
cat /proc/sys/kernel/panic # Panic behavior
echo 1 > /proc/sys/kernel/sysrq # Enable SysRq keys# Packet capture
tcpdump -i wlan0 -w capture.pcap # Capture WiFi traffic
wireshark capture.pcap # Analyze on development PC
# Network interface debugging
ethtool wlan0 # Interface statistics (if supported)
iwconfig wlan0 # Wireless configuration
iw dev wlan0 link # Connection status# Check storage health
smartctl -a /dev/mmcblk0 # eMMC health status
badblocks -v /dev/mmcblk0 # Check for bad blocks
# Filesystem debugging
fsck.ext4 -v /dev/mmcblk0p2 # Check rootfs (unmounted)
tune2fs -l /dev/mmcblk0p2 # Filesystem info| Error Pattern | Cause | Solution |
|---|---|---|
Kernel panic - not syncing: VFS: Unable to mount root |
Wrong root partition | Check U-Boot bootargs, reflash image |
Failed to execute /sbin/init |
Corrupted rootfs | Reflash complete image |
Unable to handle kernel paging request |
Hardware/DT issue | Check device tree, test hardware |
Waiting for root device |
Storage not detected | Check eMMC/SD card, power supply |
| Symptom | Cause | Solution |
|---|---|---|
| No wlan0 interface | Driver not loaded | Check kernel modules, device tree |
| Can't scan networks | Antenna/power issue | Check antenna, GPIO power control |
| Connects but no internet | DNS/routing issue | Check /etc/resolv.conf, gateway |
| Frequent disconnections | Power management | Disable WiFi power save |
| Symptom | Cause | Solution |
|---|---|---|
| I2C devices missing | Power/clock/pinmux | Check device tree, power rails |
| GPIO not working | Wrong pin config | Check device tree GPIO definitions |
| Audio not working | Codec not initialized | Check I2C communication, ALSA config |
| High power consumption | Runaway process/HW | Check htop, disable unused peripherals |
- Board doesn't power on with known-good supply
- Persistent I2C device failures across multiple images
- Physical damage to connectors or components
- Thermal issues with proper cooling
- Kernel panics with latest stable image
- Persistent boot failures after reflashing
- Performance significantly below specifications
- Security vulnerabilities or concerns
# System information
uname -a > debug-info.txt
cat /etc/os-release >> debug-info.txt
dmesg > dmesg.log
journalctl -b > journal.log
# Hardware information
lsusb > hardware-info.txt
lspci >> hardware-info.txt
i2cdetect -y 1 >> hardware-info.txt
cat /proc/cpuinfo >> hardware-info.txt- GitHub Issues: Technical problems with reproducible steps
- Email Support: info@dynamicdevices.co.uk
- Documentation: Check Advanced-Topics-Cm33-Firmware-Guide for complex issues
💡 Pro Tip: Keep a "known-good" image backup for each board - it's the fastest way to determine if issues are hardware or software related!