Changing how we build dockers

[chayim]

Background Information: We build docker images in CI, on a specific need. We build these any time we run the platform build, but don't necessarily distribute them. Occasionally (master, version branches, release tags) we publish them to GitHub.

External Issue: We have an external requirement on making a buster build of redisai, for edge. RedisAI wants to support multiple published versions of dockers.

Goal: RedisAI wants to build and release docker images of our product across multiple operating systems. This covers a matrix of cpu and gpu, x64, and arm, as well as specific operating systems such as Ubuntu versions xenial and bionic.

Proposed Solution

1. Migrate redisai dockers to use the versioning scheme described in readies #22.
2. Build the various platforms in parallel if possible, if not serially (as today) using docker-compose, as begun in this branch.
3. Modify Makefiles in opt/build/docker to build (and optionally publish, as needed) the dockers via docker-compose where possible. Publishing might need to continue using docker push.
4. Modify CI to call the new make targets.

Impact

1. Pursuing the above solution means that all supported branches of RedisAI (i.e master, 1.2, etc) will release dockers for multiple versions to dockerhub.
2. All released versions will release to the redislabs/redisai hub.
3. Releases to dockerhub can be used in downstream projects (i.e clients)
4. Downstream projects will need to be notified about the new versions.

[chayim]

@K-Jo @gkorland @rafie @DvirDukhan @alonre24 do you have anything to add?

[DvirDukhan]

overall seems a good solution, with fewer scripts. Now we need to discuss RedisEdge and RedisMod as they are combining different modules. If we will not provide a buster build it might cause a problem. My suggestion is to avoid stacking dockers with different file system structures, meaning you will have to know where the binaries+deps are stored and download the binaries from s3 instead. This is "eat your own dog food", and you will have one docker image that is easier and lighter to build.

my take on naming:
redisai-{CPU_ARCH}-{acclerator}-{OS}:{version}

CPU_ARCH = [x86, x86_intel, armv8, armv9]. x86_intel should be provided with avx512 backends
accelerator = [cpu, CUDA] in the future after we will make an effort to have a single CUDA version per build, it should be [cpu, CUDA-{cuda version}, ROCm-{version}, other...]
armv{8/9}+cuda should support jetson/ new Nvidia servers
OS = {ubuntu, centos7, macOS (CPU variants only)}
version = [actual version, latest, edge]

I don't want to discuss the backend builds but for example redisai-x86_intel-CUDA-11.0-ubuntu-16.04:edge, the backend should be built with both CUDA and avx512. This is a different story and related to a different issue.

[chayim]

@DvirDukhan We could provide a buster image (I build one in my tests). Ultimately it's a platform that's between xenial and bionic, so it shouldn't have an extra support cost (yes, famous last words).

To make sure I understand your versioning, and how it differs from readies #22 can you write out a few, fully qualified names? I just want to make sure we're 100% on the same page. Versioning is the easiest part of this for me to change, but I want to do it, while I'm in the space.

For RedisEdge and RedisMod, we could simply pull dockers from several builds (rather than s3), mount them during the build, and copy out the file. This would rely on some sort of latest (ish) tag. S3 pulls are just as easy - and will definitely run faster, for the record. Assuming we literally upload the same binaries to S3 that we build in our dockers, I'm fine with both approaches. If they continue to be separate - then we should take from the docker.

[DvirDukhan]

redisai-x86-cpu-ubuntu_20.04:1.2.3
redisai-x86-cuda_11.0-ubuntu_20.04:edge
redisai-x86_intel-cuda_11.0-ubuntu_20.04:1latest
redisai_arm-cpu-ubuntu_20.04:1.2.3
redisai_arm-cuda-10.2-ubuntu_20.04:1.2.3

Regarding RedisEdge and RedisMod, I think it is preferable to avoid the "stacking dockers" approach. You will need prior knowledge on how the dockers are built, additional runtime libraries location, etc
I suggest having a single builder (like simple ubuntu, or cuda dev docker) setting up what is required for runtime and download the binaries. For me, it is a "foolproof" approach, and moreover, it mimics RS behavior.

[chayim]

Regarding RedisEdge + RedisMod, I agree with your approach. I prefer disting as a strategy in all cases.

Regarding the versioning:

1. Is there any specific reason it was redisai-x86 and then redisai_arm (notice the undescore).

This makes each of these look like individual products within the redislabs repo. My preference is for the detail to be similar to what [nvidia did in dockerhub]](https://hub.docker.com/r/nvidia/cuda/tags?page=1&ordering=last_updated). Theirs are $VERSION-$VARIANT-$OSNICK. I was proposing having $VARIANT (your x86_intenl-cuda, for example) at the end, as $VERSION-$OSNICK-$VARIANT.

I view these all as one product with multiple variants - hence them all being available within the same docker organization/product combination.

[chayim]

@DvirDukhan For x64 we're at the point of just tagging them. Instead of using the version numbers (i.e 20.04) for Ubuntu, we're instead relying on the current name (i.e focal).

[AlexMikhalev]

Bump.
while the discussion has buster, there is no docker hub manifest

FROM redislabs/redisai:edge-cpu-buster AS builder 
FROM redislabs/rgcluster:latest
RUN apt-get update && apt-get install -y build-essential libgomp1
ENV REDIS_MODULES /usr/lib/redis/modules
ENV LD_LIBRARY_PATH $REDIS_MODULES

RUN mkdir -p $REDIS_MODULES/

COPY --from=builder /usr/lib/redis/modules/ $REDIS_MODULES/

COPY --from=builder /var/opt/redislabs/artifacts/ /var/opt/redislabs/artifacts

WORKDIR /cluster

CMD []

which means it's no longer possible to build docker image for redisai and redisgears

[chayim]

@AlexMikhalev I'll see why the manifest isn't automatically generated on the docker side - but as the documents should say:

cd build/docker
make OSNICK=buster

[AlexMikhalev]

FROM redislabs/redisai:edge-cpu-bionic AS builder 
FROM redislabs/rgcluster:edge
RUN apt-get update && apt-get install -y build-essential libgomp1
ENV REDIS_MODULES /usr/lib/redis/modules
ENV LD_LIBRARY_PATH $REDIS_MODULES

RUN mkdir -p $REDIS_MODULES/

COPY --from=builder /usr/lib/redis/modules/ $REDIS_MODULES/

COPY --from=builder /var/opt/redislabs/artifacts/ /var/opt/redislabs/artifacts

WORKDIR /cluster

CMD []

works for now, but not sure it's by intent or luck. Both docker images build by the Redis team, I would think all will use the same Linux distribution - redismod also uses bionic for redis-ai and buster or bullseye for all other modules.