Oasis Node not working after upgrade to v2.7.0

[mayank-daga]

Hi i am getting following logs after upgrading to v2.7.0:

oasis-rosetta | ok: run: node: (pid 10) 0s
oasis-rosetta | {"caller":"node.go:451","level":"info","mode":"client","module":"oasis-node","msg":"Starting oasis-node","ts":"2024-07-29T13:14:23.665671707Z","version":"23.0.7-git510099930"}
oasis-rosetta | level=debug ts=2024-07-29T13:14:23.800124334Z caller=oasis.go:136 module=oasis msg="Establishing connection" grpc_addr=unix:/data/internal.sock
oasis-rosetta | level=error ts=2024-07-29T13:14:23.80118495Z caller=grpc.go:242 module=grpc/client msg="request failed" method=/oasis-core.NodeController/GetStatus req_seq=1 rsp="unsupported value type" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused""
oasis-rosetta | level=debug ts=2024-07-29T13:14:23.801251651Z caller=oasis.go:149 module=oasis msg="Failed to get status from node" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused""
oasis-rosetta | level=error ts=2024-07-29T13:14:23.801263652Z caller=main.go:210 module=oasis-rosetta-gateway msg="failed to obtain chain ID from Oasis node" err="failed to get status from node: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused""
oasis-rosetta | ok: run: node: (pid 10) 1s
oasis-rosetta | level=debug ts=2024-07-29T13:14:24.808221535Z caller=oasis.go:136 module=oasis msg="Establishing connection" grpc_addr=unix:/data/internal.sock
oasis-rosetta | level=error ts=2024-07-29T13:14:24.808822745Z caller=grpc.go:242 module=grpc/client msg="request failed" method=/oasis-core.NodeController/GetStatus req_seq=1 rsp="unsupported value type" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused""
oasis-rosetta | level=debug ts=2024-07-29T13:14:24.808864345Z caller=oasis.go:149 module=oasis msg="Failed to get status from node" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused""
oasis-rosetta | level=error ts=2024-07-29T13:14:24.808874145Z caller=main.go:210 module=oasis-rosetta-gateway msg="failed to obtain chain ID from Oasis node" err="failed to get status from node: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused""
oasis-rosetta | ok: run: node: (pid 10) 2s
oasis-rosetta | level=debug ts=2024-07-29T13:14:25.815408523Z caller=oasis.go:136 module=oasis msg="Establishing connection" grpc_addr=unix:/data/internal.sock
oasis-rosetta | level=error ts=2024-07-29T13:14:25.816005832Z caller=grpc.go:242 module=grpc/client msg="request failed" method=/oasis-core.NodeController/GetStatus req_seq=1 rsp="unsupported value type" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused""
oasis-rosetta | level=debug ts=2024-07-29T13:14:25.816046732Z caller=oasis.go:149 module=oasis msg="Failed to get status from node" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused""

[peternose]

Based on the error message, it seems that decoding the Status response from the GetStatus gRPC function failed because your node is running on version 23.0.7-git510099930.

Recently, we did a breaking upgrade to 24.0 and latter to 24.1. Did you upgrade your oasis node to the latest version 24.1?

[mayank-daga]

Hi @peternose

we are getting same logs with newest image as well:

ok: run: node: (pid 9) 0s,
level=debug ts=2024-08-01T09:24:29.323551788Z caller=oasis.go:136 module=oasis msg="Establishing connection" grpc_addr=unix:/data/internal.sock,
level=error ts=2024-08-01T09:24:29.324201196Z caller=grpc.go:242 module=grpc/client msg="request failed" method=/oasis-core.NodeController/GetStatus req_seq=1 rsp="unsupported value type" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=debug ts=2024-08-01T09:24:29.324231096Z caller=oasis.go:149 module=oasis msg="Failed to get status from node" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=error ts=2024-08-01T09:24:29.324239896Z caller=main.go:210 module=oasis-rosetta-gateway msg="failed to obtain chain ID from Oasis node" err="failed to get status from node: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
{"caller":"node.go:457","level":"info","mode":"client","module":"oasis-node","msg":"Starting oasis-node","ts":"2024-08-01T09:24:29.329123054Z","version":"24.0-gitfb49717f3"},
ok: run: node: (pid 9) 1s,
level=debug ts=2024-08-01T09:24:30.331955078Z caller=oasis.go:136 module=oasis msg="Establishing connection" grpc_addr=unix:/data/internal.sock,
level=error ts=2024-08-01T09:24:30.332491184Z caller=grpc.go:242 module=grpc/client msg="request failed" method=/oasis-core.NodeController/GetStatus req_seq=1 rsp="unsupported value type" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=debug ts=2024-08-01T09:24:30.332526285Z caller=oasis.go:149 module=oasis msg="Failed to get status from node" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=error ts=2024-08-01T09:24:30.332536185Z caller=main.go:210 module=oasis-rosetta-gateway msg="failed to obtain chain ID from Oasis node" err="failed to get status from node: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
ok: run: node: (pid 9) 2s,
level=debug ts=2024-08-01T09:24:31.340739473Z caller=oasis.go:136 module=oasis msg="Establishing connection" grpc_addr=unix:/data/internal.sock,
level=error ts=2024-08-01T09:24:31.34131288Z caller=grpc.go:242 module=grpc/client msg="request failed" method=/oasis-core.NodeController/GetStatus req_seq=1 rsp="unsupported value type" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=debug ts=2024-08-01T09:24:31.34134428Z caller=oasis.go:149 module=oasis msg="Failed to get status from node" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=error ts=2024-08-01T09:24:31.34136648Z caller=main.go:210 module=oasis-rosetta-gateway msg="failed to obtain chain ID from Oasis node" err="failed to get status from node: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
ok: run: node: (pid 9) 3s,
level=debug ts=2024-08-01T09:24:32.349094963Z caller=oasis.go:136 module=oasis msg="Establishing connection" grpc_addr=unix:/data/internal.sock,
level=error ts=2024-08-01T09:24:32.349858772Z caller=grpc.go:242 module=grpc/client msg="request failed" method=/oasis-core.NodeController/GetStatus req_seq=1 rsp="unsupported value type" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=debug ts=2024-08-01T09:24:32.349913173Z caller=oasis.go:149 module=oasis msg="Failed to get status from node" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=error ts=2024-08-01T09:24:32.349927673Z caller=main.go:210 module=oasis-rosetta-gateway msg="failed to obtain chain ID from Oasis node" err="failed to get status from node: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
{"caller":"init.go:91","err":"signature/signer/file: invalid PEM file permissions 700 on /data/identity.pem","level":"error","module":"oasis-node","msg":"failed to load/generate identity","ts":"2024-08-01T09:24:32.845111013Z"},
{"caller":"node.go:457","level":"info","mode":"client","module":"oasis-node","msg":"Starting oasis-node","ts":"2024-08-01T09:24:32.87375195Z","version":"24.0-gitfb49717f3"},
ok: run: node: (pid 53) 1s,
level=debug ts=2024-08-01T09:24:33.358322565Z caller=oasis.go:136 module=oasis msg="Establishing connection" grpc_addr=unix:/data/internal.sock,
level=error ts=2024-08-01T09:24:33.358836871Z caller=grpc.go:242 module=grpc/client msg="request failed" method=/oasis-core.NodeController/GetStatus req_seq=1 rsp="unsupported value type" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=debug ts=2024-08-01T09:24:33.358870971Z caller=oasis.go:149 module=oasis msg="Failed to get status from node" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=error ts=2024-08-01T09:24:33.358892071Z caller=main.go:210 module=oasis-rosetta-gateway msg="failed to obtain chain ID from Oasis node" err="failed to get status from node: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
ok: run: node: (pid 53) 2s,
level=debug ts=2024-08-01T09:24:34.366351252Z caller=oasis.go:136 module=oasis msg="Establishing connection" grpc_addr=unix:/data/internal.sock,
level=error ts=2024-08-01T09:24:34.366936959Z caller=grpc.go:242 module=grpc/client msg="request failed" method=/oasis-core.NodeController/GetStatus req_seq=1 rsp="unsupported value type" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=debug ts=2024-08-01T09:24:34.36697356Z caller=oasis.go:149 module=oasis msg="Failed to get status from node" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=error ts=2024-08-01T09:24:34.36698286Z caller=main.go:210 module=oasis-rosetta-gateway msg="failed to obtain chain ID from Oasis node" err="failed to get status from node: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
ok: run: node: (pid 53) 3s,
level=debug ts=2024-08-01T09:24:35.409274252Z caller=oasis.go:136 module=oasis msg="Establishing connection" grpc_addr=unix:/data/internal.sock,
level=error ts=2024-08-01T09:24:35.409847159Z caller=grpc.go:242 module=grpc/client msg="request failed" method=/oasis-core.NodeController/GetStatus req_seq=1 rsp="unsupported value type" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=debug ts=2024-08-01T09:24:35.410018861Z caller=oasis.go:149 module=oasis msg="Failed to get status from node" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=error ts=2024-08-01T09:24:35.410067761Z caller=main.go:210 module=oasis-rosetta-gateway msg="failed to obtain chain ID from Oasis node" err="failed to get status from node: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
ok: run: node: (pid 53) 4s,
level=debug ts=2024-08-01T09:24:36.46132456Z caller=oasis.go:136 module=oasis msg="Establishing connection" grpc_addr=unix:/data/internal.sock,
level=error ts=2024-08-01T09:24:36.461889466Z caller=grpc.go:242 module=grpc/client msg="request failed" method=/oasis-core.NodeController/GetStatus req_seq=1 rsp="unsupported value type" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=debug ts=2024-08-01T09:24:36.461924767Z caller=oasis.go:149 module=oasis msg="Failed to get status from node" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=error ts=2024-08-01T09:24:36.461935667Z caller=main.go:210 module=oasis-rosetta-gateway msg="failed to obtain chain ID from Oasis node" err="failed to get status from node: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
{"caller":"init.go:91","err":"signature/signer/file: invalid PEM file permissions 700 on /data/identity.pem","level":"error","module":"oasis-node","msg":"failed to load/generate identity","ts":"2024-08-01T09:24:37.163036136Z"},

[peternose]

This will probably not solve the problem, but you should change file permissions on /data/identity.pem from 700 to 600.

[mayank-daga]

ok @peternose please let us know if you have any solution

[matevz]

I don't think this is related to the upgraded rosetta gateway, but the oasis-node. Can you check/attach the oasis-node logs too? Make sure you are running the latest version (see here https://docs.oasis.io/node/mainnet/)?

[mayank-daga]

Hi @matevz the above logs are from node only and we running latest version only

[mayank-daga]

Hi @matevz just following up

[matevz]

Did you change the permission of the data folder?

This will probably not solve the problem, but you should change file permissions on /data/identity.pem from 700 to 600.

[mayank-daga]

Hi @matevz yes i have done that

[pro-wh]

does the oasis node crash?

[mayank-daga]

Hi @pro-wh node is up but since not syncing and curl queries not working also

[mayank-daga]

HI @pro-wh we are getting following type of logs :

level=error ts=2024-09-19T09:37:32.787336593Z caller=main.go:210 module=oasis-rosetta-gateway msg="failed to obtain chain ID from Oasis node" err="failed to get status from node: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused"",
level=error ts=2024-09-19T09:37:33.796742044Z caller=grpc.go:242 module=grpc/client msg="request failed" method=/oasis-core.NodeController/GetStatus req_seq=1 rsp="unsupported value type" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /data/internal.sock: connect: connection refused""

[pro-wh]

these are logs from the gateway, can you check on the oasis node?

https://github.com/oasisprotocol/oasis-rosetta-gateway?tab=readme-ov-file#running-the-gateway

[mayank-daga]

Hi @pro-wh we are not running oasis node / cli. We are simple running oasis rosetta gateway using docker.
These were steps we followed that were suggested by you only over the issue #494

Steps:

mkdir -p /tmp/rosetta-data/etc
chmod 700 /tmp/rosetta-data
cp config.yml /tmp/rosetta-data/etc
wget -O /tmp/rosetta-data/etc/genesis.json https://github.com/oasisprotocol/mainnet-artifacts/releases/download/2023-11-29/genesis.json
sudo chown -R www-data:www-data /tmp/rosetta-data
docker run -it -v /tmp/rosetta-data:/data oasis-rosetta-gateway:v2.7.0

then

sudo ls -al /tmp/rosetta-data
shows files

Also, the logs that i have shared are the only logs from docker we are getting

Can you help me with tcp and udp ports to be opened for firewall?

[pro-wh]

but the docker image does run oasis-node, and it should be logging things if it has any trouble

https://github.com/oasisprotocol/oasis-rosetta-gateway/blob/v2.7.0/docker/service/node/run

can you investigate why there's no output?

firewall

I think the oasis-node program needs to connect to peers over the publicly routable internet in general