Multi-agent environment with early agents termination

[ColdFrenzy]

Hi, I'm trying to work on a simple multi-agent navigation task where agents need to reach a goal position.
I'm giving a little bit of context:
Whenever an agent reaches the desired goal, its done flag becomes true, and whenever an action is passed, I just skip this particular agent and I return a special observation with all -1 and a 0 reward.
I've noticed that in the tensordict generated by a rollout, there are multiple keys {a1,a2,..., done, termination, truncation}.
Since I'm not setting these global done, termination, and truncation keys manually, I was wondering how they are set. In my case, I would like done to be true only when all the agents reach the goal, i.e., {a1: {done==True},a2: {done==True} ...}
Is this already the case?
If not, is it possible to change it? I would like the episode to terminate only when all the agents reach the goal or when the max_timestep is reached.
I'm asking this because I saw that agents are able to reach the goal individually (by looking at my metrics), but they were not even once able to reach the goal at the same time

[matteobettini]

Are you using pettingzoo? if so there is a kwarg which is called done_on_any which you want to set to false. https://docs.pytorch.org/rl/0.6/reference/generated/torchrl.envs.PettingZooWrapper.html

[ColdFrenzy]

Yes, I tried to use done_on_any, but I got some error, and then I noticed that in this issue it's stated that there is no support for dead agents.
At the moment, I'm using the concept of absorbing state and returning done only if all agents are done.
In the next days, I'm going to try again with done_on_any=True and use_mask=True.
Is there any variable that I should take into consideration other than the self.agents, to list the active agents?

[matteobettini]

So you should keep use_mask=False and done_on_any=False from what I read you want.

Agents can be done, but do not remove them from the task and just give them the 0 reward and the special obs.

Like this it should work.

self.agents should always list all agents (let's not make them die, benchmarl does not like that)