It can be that simple. It's just if it's too small, you might have done minutes or hours of computation before you see the error at the end. This function is designed for huge data that is fed in a chunk at a time.

The nice thing about the CrackleArray is that it can be cheap to change the dtype. It will incur (usually) a copy of the compressed binary, but that will ordinarily be hundreds or thousands of times smaller than the numpy representation. It could be even cheaper (in-place modification of one or two bits), but it would violate the immutability of a bytes object.

The reason for the out_dtype argument in connected_components is that if you allocate e.g. a np.uint64 from the jump, you won't have to do:

x:np.ndarray[np.uint32] = cc3d.connected_components(...)
y = x.astype(np.uint64) # memory usage = 1.5x (uint64 + uint32)
del x

Instead you just have:

# 1x memory usage
x:np.ndarray[np.uint64] = cc3d.connected_components(..., out_dtype=np.uint64)

So maybe you are right, perhaps the argument should just be removed. I think I initially put it in for compatibility with the function signature of connected_components, but it's not really needed.

Connected components stack out dtype argument #138

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions