addressing cat empty tensor case.Fixes gpt2 data distributed example

[apbose]

Fixes #3865

[narendasan]

Can you make this warning much more specific? Print information like the current node, if you can where in the graph it comes from etc. Because users will not understand what you mean by this. Also where is the replacing with zeros?

[narendasan]

Also if this is caught by the validator then should this be an error? Will conversion fail or can we just ignore it?

[apbose]

Thanks for pointing out the error. I was earlier replacing with zeros, but later changed to continue since replacing with zeros is not required. I will change the warning comment.

The difference between this and the validator is that, if the empty tensor is a torch.Tensor, we can handle it in the converter.

Whereas if the empty tensor comes as an ITensor input to the converter, TensorRT complains. (I was trying to implement it earlier via replacing it with zeros, but that still leads to the error [RemoveDeadLayers] Input Tensor y is unused or used only at compile-time, but is not being removed. To point the difference,

This will pass

def test_cat_with_empty_tensor(self, _, dim):
       # Handle empty tensor in concat
       class Cat(nn.Module):
           def forward(self, x):
               y = torch.empty(0, 2, 3, device="cuda")
               return torch.ops.aten.cat.default((x, y), dim)

       inputs = [
           torch.randn(1, 2, 3, device="cuda"),
       ]
       self.run_test(Cat(), inputs)

This will fail

 def test_cat_with_empty_tensor(self, _, dim):
        # Handle empty tensor in concat
        class Cat(nn.Module):
            def forward(self, x, y):
                return torch.ops.aten.cat.default((x, y), dim)

        inputs = [
            torch.randn(1, 2, 3, device="cuda"),
            y = torch.empty(0, 2, 3, device="cuda")
        ]
        self.run_test(Cat(), inputs)

[narendasan]

I dont really understand this condition. So if we have a TRT ITensor that has a 0 in any dimension then we should break the graph? I dont think at validation time any of these ITensors will be available. Since validation is run prior to paritioning

[narendasan]

Should we be checking for empty PyTorch tensors?

[apbose]

Yes ideally. The validation would be based on the ITensor shape. Yes should use the meta data

[apbose]

But then this won't distinguish between ITensor and torch Tensor case.