diff --git a/Assets/Buildah-logo.png b/Assets/Buildah-logo.png new file mode 100644 index 0000000..d0d4476 Binary files /dev/null and b/Assets/Buildah-logo.png differ diff --git a/Assets/Kubernetes-logo.png b/Assets/Kubernetes-logo.png new file mode 100644 index 0000000..5b39d56 Binary files /dev/null and b/Assets/Kubernetes-logo.png differ diff --git a/Assets/Podman-logo.jpg b/Assets/Podman-logo.jpg new file mode 100644 index 0000000..23c13f1 Binary files /dev/null and b/Assets/Podman-logo.jpg differ diff --git a/Assets/Skopeo-logo.png b/Assets/Skopeo-logo.png new file mode 100644 index 0000000..b4ea36b Binary files /dev/null and b/Assets/Skopeo-logo.png differ diff --git a/Assets/containerization.png b/Assets/containerization.png new file mode 100644 index 0000000..a825cb8 Binary files /dev/null and b/Assets/containerization.png differ diff --git a/Assets/docker-logo.svg b/Assets/docker-logo.svg new file mode 100644 index 0000000..28f3b3b --- /dev/null +++ b/Assets/docker-logo.svg @@ -0,0 +1,19 @@ + + + + + + + + + + + + + + + + + + + diff --git a/Concepts/Containerization/Container Lifecycle.md b/Concepts/Containerization/Container Lifecycle.md new file mode 100644 index 0000000..5616c08 --- /dev/null +++ b/Concepts/Containerization/Container Lifecycle.md @@ -0,0 +1,51 @@ +--- +aliases: + - Concepts/Container Lifecycle + - Container Lifecycle +tags: + - seedling +publish: false +--- + +The container lifecycle describes the stages a container goes throughβ€”from image creation, initialization, running, pausing, stopping, and ultimately removal. +*All example code below user Docker as the runtime environment* +## Creation + +`docker create --name my-container my-image:latest` + +The container has been defined from an image and assigned resources but is not yet running. Docker (or another runtime) initializes the container metadata, filesystem, and environment. + +## Running + +`docker start` or `docker run` (create and start at the same time) + +The container is executing the application or command. To monitor running containers: + +`docker ps -a` + +## Paused + +`docker pause my_container` + +The container process is temporarily suspended. To unpause a container: + +`docker unpause my_container` + +## Exit + +To manually stop/kill a container: `docker kill my_container` or `docker stop my_container` + +The container has completed execution or has been manually stopped. Alternatively, you can use: + +`docker rm -f my_container` to remove the container + +Documentation for these command can be read at [Dockerdocs - CLI commands](https://docs.docker.com/reference/cli/docker/) + +%% wiki footer: Please don't edit anything below this line %% + +## This note in GitHub + +[Edit In GitHub](https://github.dev/data-engineering-community/data-engineering-wiki/blob/main/Concepts/Container%20Lifecycle.md "git-hub-edit-note") | [Copy this note](https://raw.githubusercontent.com/data-engineering-community/data-engineering-wiki/main/Concepts/Container%20Lifecycle.md "git-hub-copy-note") + +Was this page helpful? +[πŸ‘](https://tally.so/r/mOaxjk?rating=Yes&url=https://dataengineering.wiki/Concepts/Container%20Lifecycle) or [πŸ‘Ž](https://tally.so/r/mOaxjk?rating=No&url=https://dataengineering.wiki/Concepts/Container%20Lifecycle) diff --git a/Concepts/Software Engineering/Containerization.md b/Concepts/Software Engineering/Containerization.md new file mode 100644 index 0000000..2dd0964 --- /dev/null +++ b/Concepts/Software Engineering/Containerization.md @@ -0,0 +1,76 @@ +--- +aliases: + - Concepts/Containerization + - Containerization +tags: + - incubating +publish: false +--- + +Containerization is a special form of **virtualization** that packages the software code with its operating system libraries and dependencies into a light-weight executable unit called a **Container**. +![[containerization.png| 550]] + +## Containerization Advantages + +- **Portability**: By packaging the code, its dependencies and OS, it solves the problem of "But it works on my machine", allowing the containerized software to run uniformly and consistently on various platforms. +- **Efficiency**: The software running in the containerized environment utilizes the host machine's OS kernel. Hence, it comes with a smaller size and requires less time to start up. +- **Faster deployment**: An application running from a container can be deployed easily and rapidly scaled due to their portability and efficiency. +- **Security**: The usage of containerization reduces the risk of the application being negatively impacted by security threats or exploits. +- **Microservices architecture**: Enables the development of modular, independently deployable services by using **Container** as deployment method. +- **Automation workflow**: Containerization can be integrated with **CI/CD** workflows, collaborating with various tools to create an automatic and consistent pipeline for building, testing, and deploying applications. + +## Containerization Disadvantages + +- **OS kernel**: Containerization uses the OS kernel to operate so if there are any vulnerabilities in the kernel it can potentially lead to errors on the application. +- **More security concerns**: Containerization require many components to function such as **Container**, **Container Image**, services,... These could be targets for exploits and attacks. +- **Increase Complexity**: The process of creating and managing **Container** is a thorny task that require deep and wide system knowledge. +- **Compatibility**: Containerized applications may face compatibility issues when interacting with legacy systems. +- **Require efforts**: Adopting Containerization requires significant amount of time and effort to master and apply into real-world system. + +## Container + +A container is a lightweight, portable unit that packages an application along with its dependencies and runs it in isolation using the host system's OS kernel. Containers ensure consistency across environments, making them ideal for scalable and reproducible deployments. + +## Container Image + +A container image is a read-only template used to create containers, containing all the necessary code, libraries, configurations, and dependencies. It serves as the blueprint for running containers and can be stored, shared, and versioned through container registries. + +## Container Orchestration + +Container orchestration refers to the automated management of containerized applications, including deployment, scaling, networking, and lifecycle management. Tools like Kubernetes help coordinate these containers across clusters, ensuring reliability and high availability in production environments. + +## When to use Containerization + +### Do's + +- **Working with a Microservices architecture**: If you're working with microservices, containerization is a great fit. Containers allow you to deploy, scale, and manage each microservice independently making your application more scalable and resilient. +- **DevOps workflow**: This combined with containerization delivers consistency and speed to the process of CI/CD, enables efficient development, testing, deployment pipelines. +- **Complex dependencies**: By encapsulating dependencies within a **Container**, this allow applications to run consistently regardless of infrastructure. + +### Don'ts + +- **Simple application**: Using containerization for simple apps is overkill, straight deployment might be better regarding management complexity and speed. +- **Legacy system adoption**: "If it works, leave it be", refactor a working legacy system to newer technology might contain potential risks and tradeoffs that need consideration. +- **Fear of missing out**: "Everyone is using containerization so we need to use it too", this a false understanding of containerization usage. It is true that it provides many benefits to the development and deployment process, but when deciding to use a technology, you need to consider many aspects of a bigger picture. Everything has a tradeoff and containerization is not doing everything by itself, it needs to cooperate with its surroundings. + +## Containerization Tools + +- [[Docker]]: A widely used platform for building, running, and managing containerized applications with a robust ecosystem and CLI support. +- [[Podman]]: A daemonless (serviceless) container engine that offers Docker-compatible commands while supporting rootless containers for enhanced security. +- [[Kubernetes]]: An open-source orchestration system that automates the deployment, scaling, and management of containerized applications across clusters. +- [[Skopeo]]: A command-line tool for managing container images, allowing you to inspect, copy, and sign images without needing to pull them locally. +- [[Buildah]]: A tool for building Open Container Initiative (OCI) and Docker images from scratch or using **Dockerfiles**, often integrated with Podman for complete container workflows. + +## Best Practices + +- [Containerization best practices - Simform](https://www.simform.com/blog/containerization-best-practices) +- [Containerization best practices - Dev Communities](https://dev.to/aws-builders/the-art-of-creating-container-images-and-best-practices-3p9d) + +%% wiki footer: Please don't edit anything below this line %% + +## This note in GitHub + +[Edit In GitHub](https://github.dev/data-engineering-community/data-engineering-wiki/blob/main/Concepts/Software%20Engineering/Containerization.md "git-hub-edit-note") | [Copy this note](https://raw.githubusercontent.com/data-engineering-community/data-engineering-wiki/main/Concepts/Software%20Engineering/Containerization.md "git-hub-copy-note") + +Was this page helpful? +[πŸ‘](https://tally.so/r/mOaxjk?rating=Yes&url=https://dataengineering.wiki/Concepts/Software%20Engineering/Containerization) or [πŸ‘Ž](https://tally.so/r/mOaxjk?rating=No&url=https://dataengineering.wiki/Concepts/Software%20Engineering/Containerization) diff --git a/Templates/Tool Template - Containerization.md b/Templates/Tool Template - Containerization.md new file mode 100644 index 0000000..12b0879 --- /dev/null +++ b/Templates/Tool Template - Containerization.md @@ -0,0 +1,28 @@ +--- +Aliases: [] +Tags: [seedling] +publish: false +--- + +(optional) Logo + +Brief description of the tool. + +## {{title}} Official Documentation + +## {{title}} Advantages + +## {{title}} Disadvantages + +## {{title}} Learning Resources + +## {{title}} Recent Posts + +%% wiki footer: Please don't edit anything below this line %% + +## This note in GitHub + +[Edit In GitHub](https://github.dev/data-engineering-community/data-engineering-wiki/blob/main/Tools/Containerization/{{title}}.md "git-hub-edit-note") | [Copy this note](https://raw.githubusercontent.com/data-engineering-community/data-engineering-wiki/main/Tools/Containerization/{{title}}.md "git-hub-copy-note") + +Was this page helpful? +[πŸ‘](https://tally.so/r/mOaxjk?rating=Yes&url=https://dataengineering.wiki/Tools/Conatinerization/{{title}}) or [πŸ‘Ž](https://tally.so/r/mOaxjk?rating=No&url=https://dataengineering.wiki/Tools/Conatinerization/{{title}}) diff --git a/Tools/Containerization/Buildah.md b/Tools/Containerization/Buildah.md new file mode 100644 index 0000000..5dd590e --- /dev/null +++ b/Tools/Containerization/Buildah.md @@ -0,0 +1,36 @@ +--- +aliases: + - Tools/Buildah +tags: + - seedling +publish: false +--- + +![[Buildah-logo.png]] + +A tool for building Open Container Initiative (OCI) and Docker images from scratch or using **Dockerfiles**, often integrated with [[Podman]] for complete container workflows. + +## Buildah Official Documentation + +[Buildah documentation on Github](https://github.com/containers/buildah/tree/main/docs) + +## Buildah Advantages + +## Buildah Disadvantages + +## Buildah Learning Resources + +[Buildah tutorial on Github](https://github.com/containers/buildah/tree/main/docs/tutorials) + +## Buildah Recent Posts + +[Buidah recent blogs](https://buildah.io/blogs/) + +%% wiki footer: Please don't edit anything below this line %% + +## This note in GitHub + +[Edit In GitHub](https://github.dev/data-engineering-community/data-engineering-wiki/blob/main/Tools/Containerization/Buildah.md "git-hub-edit-note") | [Copy this note](https://raw.githubusercontent.com/data-engineering-community/data-engineering-wiki/main/Tools/Containerization/Buildah.md "git-hub-copy-note") + +Was this page helpful? +[πŸ‘](https://tally.so/r/mOaxjk?rating=Yes&url=https://dataengineering.wiki/Tools/Conatinerization/Buildah) or [πŸ‘Ž](https://tally.so/r/mOaxjk?rating=No&url=https://dataengineering.wiki/Tools/Conatinerization/Buildah) diff --git a/Tools/Containerization/Docker.md b/Tools/Containerization/Docker.md new file mode 100644 index 0000000..80aaba6 --- /dev/null +++ b/Tools/Containerization/Docker.md @@ -0,0 +1,43 @@ +--- +aliases: + - Tools/Docker +tags: + - seedling +publish: false +--- + +![[docker-logo.svg | 300]] + +A widely used platform for building, running, and managing containerized applications with a robust ecosystem and CLI support. + +## Docker Official Documentation + +[Official documentation of Docker](https://docs.docker.com/) + +## Docker Advantages + +- **Portability**: It can work on many different environments from local machine to cloud-based environment. This is crucial for cross-platform deployment. +- **Resource utilization**: **Docker containers** share the host OS kernel, making them more lightweight and efficient, leading to faster startup times, reduced storage usage, and lower operational costs. +- **Isolation**: Each Docker container is an independent unit, providing a level of security by separating itself from other containers. Hence, it reduces the risk of unintended interference or access. +- **Version and dependency control**: By the use of **Dockerfile** which is easy to modify to manage **Docker image** environment, the process of version control and maintaining reproducible environments is become easier for developers. +## Docker Disadvantages + +- **Security**: **Docker containers** share the host OS, meaning vulnerabilities at the OS level could affect multiple containers. +- **Management complexity**: **Docker containers** deployment requires knowledge and effort to manage and proper measure for scaling up. +- **Complex workflow**: Setting up Docker to run might be easy but making it work cooperatively with other components in the workflow is a hard challenge, +## Docker Learning Resources + +[Docker learning resources - Dockerdocs](https://docs.docker.com/get-started/resources/) + +## Docker Recent Posts + +[Recent blogs on Docker](https://www.docker.com/blog/) + +%% wiki footer: Please don't edit anything below this line %% + +## This note in GitHub + +[Edit In GitHub](https://github.dev/data-engineering-community/data-engineering-wiki/blob/main/Tools/Containerization/Docker.md "git-hub-edit-note") | [Copy this note](https://raw.githubusercontent.com/data-engineering-community/data-engineering-wiki/main/Tools/Containerization/Docker.md "git-hub-copy-note") + +Was this page helpful? +[πŸ‘](https://tally.so/r/mOaxjk?rating=Yes&url=https://dataengineering.wiki/Tools/Conatinerization/Docker) or [πŸ‘Ž](https://tally.so/r/mOaxjk?rating=No&url=https://dataengineering.wiki/Tools/Conatinerization/Docker) diff --git a/Tools/Containerization/Kubernetes.md b/Tools/Containerization/Kubernetes.md new file mode 100644 index 0000000..721574e --- /dev/null +++ b/Tools/Containerization/Kubernetes.md @@ -0,0 +1,36 @@ +--- +aliases: + - Tools/Kubernetes +tags: + - seedling +publish: false +--- + +![[Kubernetes-logo.png]] + +An open-source orchestration system that automates the deployment, scaling, and management of containerized applications across clusters. + +## Kubernetes Official Documentation + +[Official Kubernetes Documentation](https://kubernetes.io/docs/home/) + +## Kubernetes Advantages + +## Kubernetes Disadvantages + +## Kubernetes Learning Resources + +[Kubernetes courses on Linux Foundation](https://training.linuxfoundation.org/full-catalog/?_sft_product_type=training&_sft_technology=kubernetes) + +## Kubernetes Recent Posts + +[Kubernetes recent blogs](kubernetes.io/blog/) + +%% wiki footer: Please don't edit anything below this line %% + +## This note in GitHub + +[Edit In GitHub](https://github.dev/data-engineering-community/data-engineering-wiki/blob/main/Tools/Containerization/Kubernetes.md "git-hub-edit-note") | [Copy this note](https://raw.githubusercontent.com/data-engineering-community/data-engineering-wiki/main/Tools/Containerization/Kubernetes.md "git-hub-copy-note") + +Was this page helpful? +[πŸ‘](https://tally.so/r/mOaxjk?rating=Yes&url=https://dataengineering.wiki/Tools/Conatinerization/Kubernetes) or [πŸ‘Ž](https://tally.so/r/mOaxjk?rating=No&url=https://dataengineering.wiki/Tools/Conatinerization/Kubernetes) diff --git a/Tools/Containerization/Podman.md b/Tools/Containerization/Podman.md new file mode 100644 index 0000000..18fc693 --- /dev/null +++ b/Tools/Containerization/Podman.md @@ -0,0 +1,38 @@ +--- +aliases: + - Tools/Podman +tags: + - seedling +publish: false +--- + +![[Podman-logo.jpg]] + +A daemonless (serviceless) container engine that offers Docker-compatible commands while supporting rootless containers for enhanced security. + +## Podman Official Documentation + +[Official Podman Documentation](https://podman.io/docs) + +## Podman Advantages + +## Podman Disadvantages + +## Podman Learning Resources + +[Podman tutorials on GitHub](https://github.com/containers/podman/tree/main/docs/tutorials) + +## Podman Recent Posts + +[Podman on GitHub](https://github.com/containers/podman) + +[Podman recent blogs](https://podman.io/blogs) + +%% wiki footer: Please don't edit anything below this line %% + +## This note in GitHub + +[Edit In GitHub](https://github.dev/data-engineering-community/data-engineering-wiki/blob/main/Tools/Containerization/Podman.md "git-hub-edit-note") | [Copy this note](https://raw.githubusercontent.com/data-engineering-community/data-engineering-wiki/main/Tools/Containerization/Podman.md "git-hub-copy-note") + +Was this page helpful? +[πŸ‘](https://tally.so/r/mOaxjk?rating=Yes&url=https://dataengineering.wiki/Tools/Conatinerization/Podman) or [πŸ‘Ž](https://tally.so/r/mOaxjk?rating=No&url=https://dataengineering.wiki/Tools/Conatinerization/Podman) diff --git a/Tools/Containerization/Skopeo.md b/Tools/Containerization/Skopeo.md new file mode 100644 index 0000000..cc77fed --- /dev/null +++ b/Tools/Containerization/Skopeo.md @@ -0,0 +1,32 @@ +--- +aliases: + - Tools/Skopeo +tags: + - seedling +publish: false +--- + +![[Skopeo-logo.png]] + +A command-line tool for managing container images, allowing you to inspect, copy, and sign images without needing to pull them locally. + +## Skopeo Official Documentation + +[Skopeo documentation on Github](https://github.com/containers/skopeo/tree/main/docs) + +## Skopeo Advantages + +## Skopeo Disadvantages + +## Skopeo Learning Resources + +[Skopeo on Github](https://github.com/containers/skopeo) + +%% wiki footer: Please don't edit anything below this line %% + +## This note in GitHub + +[Edit In GitHub](https://github.dev/data-engineering-community/data-engineering-wiki/blob/main/Tools/Containerization/Skopeo.md "git-hub-edit-note") | [Copy this note](https://raw.githubusercontent.com/data-engineering-community/data-engineering-wiki/main/Tools/Containerization/Skopeo.md "git-hub-copy-note") + +Was this page helpful? +[πŸ‘](https://tally.so/r/mOaxjk?rating=Yes&url=https://dataengineering.wiki/Tools/Conatinerization/Skopeo) or [πŸ‘Ž](https://tally.so/r/mOaxjk?rating=No&url=https://dataengineering.wiki/Tools/Conatinerization/Skopeo)