Skip to content

Redis AI docker image redis-cli command crash exit while running info modules #906

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
xuning97 opened this issue Mar 17, 2022 · 2 comments
Open

Redis AI docker image redis-cli command crash exit while running info modules #906

xuning97 opened this issue Mar 17, 2022 · 2 comments
Assignees
@chayim

Comments

@xuning97
Copy link

xuning97 commented Mar 17, 2022 •
edited
Loading

Describe the bug

On MacOS with Intel Chip, following the steps of https://developer.redis.com/howtos/redisai/getting-started/
Enter docker container and run "127.0.0.1:6379> info modules", the cli crashed out directly

To Reproduce
Steps to reproduce the behavior:

  1. run the command to pull down the image
    docker run \ -p 6379:6379 \ redislabs/redismod \ --loadmodule /usr/lib/redis/modules/redisai.so \ ONNX redisai_onnxruntime/redisai_onnxruntime.so
  2. docker exec -it xxxxxx bash
  3. redis-cli
  4. info modules
  5. interactive command exit
  6. server side crash with the following output
1:M 17 Mar 2022 08:22:40.823 * Module 'ai' loaded from /usr/lib/redis/modules/redisai.so
1:M 17 Mar 2022 08:22:40.824 * Ready to accept connections



=== REDIS BUG REPORT START: Cut & paste starting from here ===
1:M 17 Mar 2022 11:13:49.050 # Redis 6.2.6 crashed by signal: 11, si_code: 1
1:M 17 Mar 2022 11:13:49.053 # Accessing address: 0x20
1:M 17 Mar 2022 11:13:49.053 # Crashed running the instruction at: 0x7f65d33bb810

------ STACK TRACE ------
EIP:
/usr/lib/redis/modules/backends/redisai_onnxruntime/redisai_onnxruntime.so(_ZNK15RAIOrtAllocator11MemoryInUseEv+0x0)[0x7f65d33bb810]

Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14140)[0x7f65d4312140]
/usr/lib/redis/modules/backends/redisai_onnxruntime/redisai_onnxruntime.so(_ZNK15RAIOrtAllocator11MemoryInUseEv+0x0)[0x7f65d33bb810]
/usr/lib/redis/modules/redisai.so(RAI_moduleInfoFunc+0x275)[0x7f65d35d50b5]
redis-server *:6379(modulesCollectInfo+0x6c)[0x555e66b5e93c]
redis-server *:6379(infoCommand+0x34)[0x555e66ad5a14]
redis-server *:6379(call+0xa1)[0x555e66ad11a1]
redis-server *:6379(processCommand+0x593)[0x555e66ad2d23]
redis-server *:6379(processInputBuffer+0xf8)[0x555e66ae5ec8]
redis-server *:6379(+0xfbbb8)[0x555e66b7dbb8]
redis-server *:6379(aeProcessEvents+0x292)[0x555e66ac9e32]
redis-server *:6379(aeMain+0x1d)[0x555e66aca09d]
redis-server *:6379(main+0x316)[0x555e66ac62b6]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f65d415fd0a]
redis-server *:6379(_start+0x2a)[0x555e66ac678a]

------ REGISTERS ------
1:M 17 Mar 2022 11:13:49.127 #
RAX:0000000000000000 RBX:00007f65d37fefa8
RCX:0000555e66c1d7bc RDX:0000000000000002
RDI:0000000000000000 RSI:000000002e392e31
RBP:00007f65d37ff1a0 RSP:00007fffa178e9a8
R8 :00007f65d2fa0180 R9 :00007f65d3e00900
R10:00007f65d4133760 R11:00007f65d2fa0180
R12:00007f65d37ff4f8 R13:0000555e66b57100
R14:00007f65d37ff580 R15:00007f65d37ff158
RIP:00007f65d33bb810 EFL:0000000000010202
CSGSFS:002b000000000033
1:M 17 Mar 2022 11:13:49.127 # (00007fffa178e9b7) -> 0000000000000006
1:M 17 Mar 2022 11:13:49.127 # (00007fffa178e9b6) -> 00007f65d1207118
1:M 17 Mar 2022 11:13:49.127 # (00007fffa178e9b5) -> 00007fffa178ea50
1:M 17 Mar 2022 11:13:49.127 # (00007fffa178e9b4) -> 00007f65d3e21910
1:M 17 Mar 2022 11:13:49.128 # (00007fffa178e9b3) -> 00007f65d4133730
1:M 17 Mar 2022 11:13:49.128 # (00007fffa178e9b2) -> 00007f65d12000c0
1:M 17 Mar 2022 11:13:49.128 # (00007fffa178e9b1) -> 0000000000000000
1:M 17 Mar 2022 11:13:49.128 # (00007fffa178e9b0) -> 00007f65d3e0a0c0
1:M 17 Mar 2022 11:13:49.128 # (00007fffa178e9af) -> 00007f65d4133730
1:M 17 Mar 2022 11:13:49.128 # (00007fffa178e9ae) -> 00007f65d3e02638
1:M 17 Mar 2022 11:13:49.128 # (00007fffa178e9ad) -> 00007f65d3e09fc0
1:M 17 Mar 2022 11:13:49.128 # (00007fffa178e9ac) -> 0000555e66be3748
1:M 17 Mar 2022 11:13:49.128 # (00007fffa178e9ab) -> 00007f65d3e0a040
1:M 17 Mar 2022 11:13:49.128 # (00007fffa178e9aa) -> 00007fffa178ec00
1:M 17 Mar 2022 11:13:49.128 # (00007fffa178e9a9) -> 0000555e66cbce78
1:M 17 Mar 2022 11:13:49.128 # (00007fffa178e9a8) -> 00007f65d35d50b5

------ INFO OUTPUT ------
# Server
redis_version:6.2.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:19160d57cbe39e4d
redis_mode:standalone
os:Linux 5.10.104-linuxkit x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:10.2.1
process_id:1
process_supervised:no
run_id:6399cae6798fccb5650e1188ecf6d023b8fbd8eb
tcp_port:6379
server_time_usec:1647515629024106
uptime_in_seconds:10269
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:3348460
executable:/data/redis-server
config_file:
io_threads_active:0

# Clients
connected_clients:1
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:16
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

# Memory
used_memory:906048
used_memory_human:884.81K
used_memory_rss:11907072
used_memory_rss_human:11.36M
used_memory_peak:967008
used_memory_peak_human:944.34K
used_memory_peak_perc:93.70%
used_memory_overhead:861808
used_memory_startup:841312
used_memory_dataset:44240
used_memory_dataset_perc:68.34%
allocator_allocated:1000384
allocator_active:1314816
allocator_resident:3768320
total_system_memory:4125892608
total_system_memory_human:3.84G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.31
allocator_frag_bytes:314432
allocator_rss_ratio:2.87
allocator_rss_bytes:2453504
rss_overhead_ratio:3.16
rss_overhead_bytes:8138752
mem_fragmentation_ratio:13.81
mem_fragmentation_bytes:11044648
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:20496
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1647505360
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:2
total_commands_processed:3
instantaneous_ops_per_sec:0
total_net_input_bytes:61
total_net_output_bytes:43604
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:7858
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:4
total_writes_processed:2
io_threaded_reads_processed:0
io_threaded_writes_processed:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:7987ac72be720c2a83099aca8966ce77b71b9aa2
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:804.720351
used_cpu_user:272.835871
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:126.189286
used_cpu_user_main_thread:34.482595

# Modules
module:name=ai,ver=10205,api=1,filters=0,usedby=[],using=[],options=[handle-io-errors]

# Commandstats
cmdstat_info:calls=1,usec=150,usec_per_call=150.00,rejected_calls=0,failed_calls=0
cmdstat_command:calls=2,usec=16169,usec_per_call=8084.50,rejected_calls=0,failed_calls=0

# Errorstats

# Cluster
cluster_enabled:0

# Keyspace

------ CLIENT LIST OUTPUT ------
id=6 addr=127.0.0.1:53362 laddr=127.0.0.1:6379 fd=8 name= age=19 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=27 qbuf-free=40927 argv-mem=11 obl=0 oll=0 omem=0 tot-mem=61467 events=r cmd=info user=default redir=-1

------ CURRENT CLIENT INFO ------
id=6 addr=127.0.0.1:53362 laddr=127.0.0.1:6379 fd=8 name= age=19 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=27 qbuf-free=40927 argv-mem=11 obl=0 oll=0 omem=0 tot-mem=61467 events=r cmd=info user=default redir=-1
argv[0]: 'info'
argv[1]: 'modules'

------ MODULES INFO OUTPUT ------`
**Expected behavior**
Out put like in the tutorial
 ```127.0.0.1:6379> info modules
 # Modules
 module:name=ai,ver=10003,api=1,filters=0,usedby=[],using=[],options=[]

 # ai_git
 ai_git_sha:7f808a934dff121e188cb76fdfcc3eb1f9ec7cbf

 # ai_load_time_configs
 ai_threads_per_queue:1
 ai_inter_op_parallelism:0
 ai_intra_op_parallelism:0

Screenshots

image

Environment (please complete the following information):

  • OS: [macOS]
  • Version [Monterey]
  • Platfrom [Intel Core i3]
  • Runtime [CPU]
@xuning97
Copy link
Author

It's start point to try RedisAI, it's a surprise to see such kind of failure.
Does it mean redisai module is still in alpha state?

@DvirDukhan
Copy link
Collaborator

@xuning97 thanks for reporting
we will check why redismod is acting like that
if redismod is not a must, please use redisai own dockers
https://oss.redis.com/redisai/quickstart/#on-a-cpu-only-machine

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chayim chayim

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@chayim @xuning97 @DvirDukhan