diff --git a/src/components/sidebar.tsx b/src/components/sidebar.tsx index c09fd3d07a..d72ff5a0c5 100644 --- a/src/components/sidebar.tsx +++ b/src/components/sidebar.tsx @@ -113,6 +113,7 @@ export default () => { Geolocation Single Sign-On (SSO) Content Security Policy (CSP) + Reverse Proxy Troubleshooting Support diff --git a/src/docs/self-hosted/index.mdx b/src/docs/self-hosted/index.mdx index 14843febf7..3309d142dd 100644 --- a/src/docs/self-hosted/index.mdx +++ b/src/docs/self-hosted/index.mdx @@ -8,6 +8,33 @@ In addition to making its source code available publicly, Sentry offers and main Our recommendation is to download the [latest release of the self-hosted repository](https://github.com/getsentry/self-hosted/releases/latest), and then run `./install.sh` inside this directory. This script will take care of all the things you need to get started, including a base-line configuration, and then will tell you to run `docker compose up -d` to start Sentry. Sentry binds to port `9000` by default. You should be able to reach the login page at [http://127.0.0.1:9000](http://127.0.0.1:9000/). +To have easy maintainability for future upgrades, it is recommended to use Git workflow by cloning the self-hosted repository and check out to a specific CalVer tag. More about this on [Releases & Upgrading](/self-hosted/releases/). + +```bash +VERSION="24.1.0" +git clone https://github.com/getsentry/self-hosted.git +cd self-hosted +git checkout ${VERSION} +sudo ./install.sh +``` + +## Recommended system resource + +Depending on your usage, required system resource to run Sentry may varies. Minimum system specification for running Sentry looks like this: + +- 2 CPU cores +- 4 GB RAM + +Having a machine that have lower system specification than that will fail the installation script. + +Current recommended system specification for running Sentry is: + +- 4 CPU cores +- 16 GB RAM +- 20 GB free disk storage space + +Depending on your load, you might want to increase your system specification to handle higher traffic load better. If increasing the disk storage space isn't possible, you can migrate your storage to use external storage such as AWS S3 or Google Cloud Storage (GCS). Decreasing your `SENTRY_RETENTION_DAYS` environment variable to lower numbers will save some storage space from being full, at the cost of having shorter data retention period. + ### Self-Hosted Monitoring When you run `./install.sh`, you have a choice to opt in or out of our monitoring. This monitoring is used for development and debugging purposes so that we're on top of issues you're facing, allowing us to provide a more seamless installation process. For more details please see the [section in the self-hosted README](https://github.com/getsentry/self-hosted#self-hosted-monitoring). @@ -32,7 +59,7 @@ The following information is reported: Note: The contact email is utilized for security announcements, and will never be used outside of such. You can change your opt in/out settings for sending contact info at any time in the settings of the admin panel. The data reported is minimal and it greatly helps the development team behind Sentry. With that said, you can disable the beacon with the following setting: -``` +```python SENTRY_BEACON = False ``` @@ -48,7 +75,7 @@ Here are the steps to follow: 1. Set `http_proxy`, `https_proxy` and `no_proxy` variables in the `/etc/environment` file. 2. To make the `docker pull` command respect your proxy settings, create a `/etc/systemd/system/docker.service.d/http-proxy.conf` file with these contents: -``` +```systemd [Service] Environment="HTTP_PROXY=http://proxy:3128" Environment="HTTPS_PROXY=http://proxy:3128" @@ -119,3 +146,9 @@ We strongly recommend using a dedicated load balancer in front of your Sentry se Once you have setup a load balancer or reverse proxy to your Sentry instance, you should modify the `system.url-prefix` in the `config.yml` file to match your new URL and protocol. You should also update the SSL/TLS section in the `sentry/sentry.conf.py` script, otherwise you may get CSRF-related errors when performing certain actions such as configuring integrations. Keep in mind that all this setup uses single-nodes for all services, including Kafka. For larger loads, you'd need a beefy machine with lots of RAM and disk storage. To scale up even further, you are very likely to use clusters with a more complex tool, such as Kubernetes. Due to self-hosted installations' very custom nature, we do not offer any recommendations or guidance around scaling up. We do what works for us for our thousands of customers over at [sentry.io](https://sentry.io/) and would love to have you over when you feel your local install's maintenance becomes a burden instead of a joy. + +## Licensing + +Sentry is using [FSL (Functional Source License)](https://fsl.software/). Although it's not an OSI-approved license, the source code of Sentry and its' surrounding service & libraries for self-hosted deployment are open and [available on GitHub](https://github.com/orgs/getsentry/repositories?q=has%3Atag-production). For self-hosted, this license means that users can use Sentry and deploy it anywhere (even inside an enterprise ecosystem), but users are prohibited to sell deployed self-hosted Sentry as any kind of offering (SaaS or any kind of business model) and users are prohibited to be a direct competitor of Sentry by using Sentry's code that is FSL-licensed. After two years, the license converts to Apache 2.0 or MIT. + +For further information on the FSL License, you can [visit the fsl.software website](https://fsl.software/) or [read the announcement blog post](https://blog.sentry.io/introducing-the-functional-source-license-freedom-without-free-riding/). \ No newline at end of file diff --git a/src/docs/self-hosted/releases.mdx b/src/docs/self-hosted/releases.mdx index de32742943..397b9b7137 100644 --- a/src/docs/self-hosted/releases.mdx +++ b/src/docs/self-hosted/releases.mdx @@ -32,6 +32,15 @@ Before starting the upgrade, we shut down all the services and then run some dat There are certain hard-stops you need to go through when upgrading from earlier versions. Please read the hard-stops section below for a list. +If you downloaded self-hosted repository using Git clone, the upgrade commands should look like this: + +```bash +VERSION="24.3.0" +git fetch +git checkout ${VERSION} +sudo ./install.sh +``` + ### Hard Stops We have three hards stops that you need to go through in order to pick up significant database changes: @@ -62,6 +71,7 @@ We provide nightly builds from the [master branch of the self-hosted repository - [Snuba](https://github.com/getsentry/snuba) - [Relay](https://github.com/getsentry/relay) - [Symbolicator](https://github.com/getsentry/symbolicator) +- [Vroom](https://github.com/getsentry/vroom) These builds are usually stable, but you may occasionally hit a broken version as these versions are not guaranteed to be deployed to [sentry.io](http://sentry.io/) first. There is also no guarantee that you will be able to do a clean upgrade to later versions without losing any data. **Use the nightly builds at your own risk.** diff --git a/src/docs/self-hosted/reverse-proxy.mdx b/src/docs/self-hosted/reverse-proxy.mdx new file mode 100644 index 0000000000..b48551f2bc --- /dev/null +++ b/src/docs/self-hosted/reverse-proxy.mdx @@ -0,0 +1,215 @@ +--- +title: 'Self-Hosted Reverse Proxy' +--- + +Adding a reverse proxy in front of your Sentry deployment is strongly recommended for one big reason: you can fine tune every configuration to fit your current setup. A dedicated reverse proxy that does SSL/TLS termination that also forwards the client IP address as Docker Compose internal network (as this is [close to impossible to get otherwise)](https://github.com/getsentry/self-hosted/issues/554) would give you the best Sentry experience. + +Once you have setup a reverse proxy to your Sentry instance, you should modify the `system.url-prefix` in the `config.yml` file to match your new URL and protocol. You should also update the SSL/TLS section in the `sentry/sentry.conf.py` script, otherwise you may get CSRF-related errors when performing certain actions such as configuring integrations. + +Keep in mind that all this setup uses single-nodes for all services, including Kafka. For larger loads, you'd need a beefy machine with lots of RAM and disk storage. To scale up even further, you are very likely to use clusters with a more complex tool, such as Kubernetes. Due to self-hosted installations' very custom nature, we do not offer any recommendations or guidance around scaling up. We do what works for us for our thousands of customers over at [sentry.io](https://sentry.io/) and would love to have you over when you feel your local install's maintenance becomes a burden instead of a joy. + +## Enabling HTTPS + +We recommend TLS termination to be done on your own dedicated load balancer or proxy. Although you can set it on the `nginx.conf` file, it is not recommended as newer self-hosted releases might alter some configurations on the file. Some examples are available on [Reverse Proxy Examples](#reverse-proxy-examples) section. + +## Expose Only Ingest Endpoint Publicly + +Certain self-hosted deployments requires the dashboard to be accessed only via internal network. But, they also need to provide public Sentry ingestion endpoint for client devices such as mobile and desktop apps. You can expose some of these endpoints publicly: + +- `/api/[1-9]\d*/envelope/` - Main endpoint for submitting event from SDK +- `/api/[1-9]\d*/minidump/` - Endpoint for submitting minidump from native SDKs +- `/api/[1-9]\d*/security/` - Endpoint for submitting security-related such as CSP errors +- `/api/[1-9]\d*/store/` - Old endpoint for submitting event from SDK, it is deprecated. +- `/api/[1-9]\d*/unreal/` - Endpoint for submitting crash report from Unreal Engine SDK + +The `[1-9]\d+` is a regular expression string that is acquired from the project DSN. + +## Rate Limiting + +By default, Sentry does not handle rate limiting for any incoming request. On hosted version of Sentry (SaaS), it has a feature called [spike protection](https://docs.sentry.io/product/accounts/quotas/spike-protection/) which can protect you from event flood. The code for that module is not available on the public [sentry](https://github.com/getsentry/sentry) repository, it lives on the private getsentry repository instead. + +For self-hosted deployment, it is recommended to have a rate limiter on your dedicated load balancer to prevent such things to happen. It is highly recommended than ever if you expose your Sentry instance publicly to the internet. + +## Health Checks + +Endpoint for health checks is available on `/_health/` endpoint using HTTP protocol. This will return a 200 if Sentry is up or a 500 with the list of problems. + +## Reverse Proxy Examples + + + +### Caddy + +[Caddy](https://caddyserver.com/) is one alternative similar to NGINX that automatically handles TLS certificate management via ACME. After you [install Caddy](https://caddyserver.com/docs/install), modify your Caddy configuration file that reside on `/etc/caddy/Caddyfile`. + +```caddyfile +sentry.yourcompany.com { + reverse_proxy your-sentry-ip:9000 { + health_uri /_health/ + health_status 2xx + header_up Host {upstream_hostport} + } + + # By default, the TLS is acquired from Let's Encrypt + tls name@yourcompany.com + + # If you have self-signed certificate + # tls /path/to/server-certificate.crt /path/to/server-certificate.key + + header { + # Delete "Server" header + -Server + } + + # To enable rate limiter, install additional module from + # https://github.com/mholt/caddy-ratelimit + # rate_limit { + # zone sentry { + # key {remote_host} + # window 1s + # events 100 + # } + # } + + # To expose only ingest endpoint publicly, add the named matcher below before `reverse_proxy` directive + # @ingest_endpoint { + # path_regexp /api/[1-9]\d+(envelope|minidump|security|store|unreal)/ + # } +} +``` + +For detailed documentation on Caddyfile configuration, see [Caddy documentation](https://caddyserver.com/docs/caddyfile). + +### Traefik + +[Traefik](https://doc.traefik.io/traefik/) is another reverse proxy that provides a lot of plugin and integrations out of the box. It automatically handles TLS certificate management via ACME, too. After you [install Traefik](https://doc.traefik.io/traefik/getting-started/install-traefik/), add a configuration to Traefik as follows (this example is using the YAML file provider, convert to your prefered configuration provider as needed). + +```yaml +http: + routers: + sentry: + entryPoints: + - web # Assuming this your HTTP entrypoint + - websecure # Assuming this is your HTTPS entrypoint + service: sentry@file + rule: "Host(`sentry.yourcompany.com`)" + # If you want to expose only ingest endpoint publicly + # rule: "Host(`sentry.yourcompany.com`) && PathPrefix(`/api/{id:[1-9]\d*}/envelope`, `/api/{id:[1-9]\d*}/minidump`, `/api/{id:[1-9]\d*}/security`, `/api/{id:[1-9]\d*}/store`, `/api/{id:[1-9]\d*}/unreal`)" + tls: + certResolver: letsencrypt # Assuming you have a TLS certificate resolver named "letsencrypt" + # Enable middleware as needed + middlewares: + - https_redirect@file + - cors_headers@file # For handling browser clients + - rate_limiter@file + + services: + sentry: + loadBalancer: + servers: + - url: "http://your-sentry-ip:9000" + healthCheck: + scheme: http + path: /_health/ + interval: 30s + timeout: 10s + passHostHeader: true + + middlewares: + https_redirect: + redirectScheme: + scheme: "https" + port: "443" + permanent: true + cors_headers: + headers: + customResponseHeaders: + # We can't remove header on Traefik, but we can put it to some other values + server: "Your Company Name" + addVaryHeader: true + # If you want to set this to true, adjust "accessControlAllowOriginList" to a valid domain and remove the asterisk wildcard + accessControlAllowCredentials: false + accessControlAllowOriginList: + - "*" + accessControlAllowHeaders: + - "sentry-trace" + - "baggage" + accessControlAllowMethods: + - GET + - POST + - PUT + - PATCH + - DELETE + accessControlExposeHeaders: + - "sentry-trace" + - "baggage" + sslRedirect: true + rate_limiter: + rateLimit: + average: 100 + period: 1s + burst: 150 +``` + +### HAProxy + +[HAProxy](https://www.haproxy.org/) is a high performance reverse proxy. This is the recommended reverse proxy to go when you have encountered network hiccups by using other reverse proxies due to its' performance. HAProxy requires external module to handles automatic TLS certificate management. + +To install HAProxy, it is recommended to acquire it from your distribution package manager (apt or yum). See [their official distribution repositories](https://github.com/haproxy/wiki/wiki/Packages#official-distribution-repositories). Then, you should be able to configure your HAProxy configuration file that should be on `/etc/haproxy/haproxy.cfg`. + +```haproxy +global + # Your global configuration (may varies between version and Linux distributions) + +defaults + mode http + log global + option httplog + option dontlognull + option forwardfor + option http-server-close + option http-keep-alive + timeout connect 10s # Connect timeout in 10s + timeout client 30s # Client timeout in 30s + timeout server 30s # Server timeout in 30s + timeout http-keep-alive 2m # HTTP keep alive in 2 minutes + # Your remaining defaults configuration + +frontend http_bind + bind *:80 name http_port + mode http + + acl sentry_domain hdr(host) -i sentry.yourcompany.com + + # HTTPS redirection + http-request redirect scheme https code 301 if sentry_domain !{ ssl_fc } + + use_backend sentry + + +frontend https_bind + bind *:443 ssl crt /etc/haproxy/certs/ name https_port + mode http + + acl sentry_domain hdr(host) -i sentry.yourcompany.com + use_backend sentry if sentry_domain + + # To expose only ingest endpoints publicly, add the acl below on `use_backend` directive + # acl ingest_endpoint path_reg -i /api/[1-9]\d+(envelope|minidump|security|store|unreal)/ + # use_backend sentry if sentry_domain ingest_endpoint + +backend sentry + mode http + option httpchk + server server1 your-sentry-ip:9000 check +``` + +To use HAProxy with ACME server such as Let's Encrypt, refer to this [blog post by HAProxy](https://www.haproxy.com/blog/haproxy-and-let-s-encrypt). diff --git a/src/docs/self-hosted/support.mdx b/src/docs/self-hosted/support.mdx index 5f9cb2fdd1..6af9a87b3a 100644 --- a/src/docs/self-hosted/support.mdx +++ b/src/docs/self-hosted/support.mdx @@ -2,7 +2,7 @@ title: 'Self-Hosted Support' --- -We strive to provide a great self-hosted experience for folks willing to try Sentry that way. The [self-hosted repository](https://github.com/getsentry/self-hosted) is geared towards low to medium loads with simplicity in mind. We believe keeping this setup simple is important to show +We strive to provide a great self-hosted experience for folks willing to try Sentry that way. The [self-hosted repository](https://github.com/getsentry/self-hosted) is geared towards low traffic loads (less than ~1 million submitted Sentry events per month). We believe keeping this setup simple is important to show how various components fit together. Folks needing larger setups can expand from here based on their specific needs and environments. diff --git a/src/docs/self-hosted/troubleshooting.mdx b/src/docs/self-hosted/troubleshooting.mdx index a4637d4dc7..ad393c0197 100644 --- a/src/docs/self-hosted/troubleshooting.mdx +++ b/src/docs/self-hosted/troubleshooting.mdx @@ -2,7 +2,7 @@ title: "Self-Hosted Troubleshooting" --- -Please keep in mind that the [self-hosted repository](https://github.com/getsentry/self-hosted) is geared towards low to medium loads with simplicity in mind. Folks needing larger setups or having event spikes can expand from here based on their specific needs and environments. If this is not your cup of tea, you are always welcome to [try out hosted Sentry](https://sentry.io/signup/). +Please keep in mind that the [self-hosted repository](https://github.com/getsentry/self-hosted) is geared towards low traffic loads (less than ~1 million submitted Sentry events per month). Folks needing larger setups or having event spikes can expand from here based on their specific needs and environments. If this is not your cup of tea, you are always welcome to [try out hosted Sentry](https://sentry.io/signup/). ## General @@ -146,7 +146,74 @@ container for service "${servicename}" is unhealthy This can usually be resolved by running `docker compose down` and `docker compose up -d` or rerunning the install script. +## Docker Network Conflicting IP Address + +Self-hosted Sentry is using Docker's bridge networking, in which use a specific private IP range. By default, Docker uses `172.17.0.0/16` range (`172.17.0.0`-`172.17.255.255`). This may cause conflict with your private network. You can change Docker's default IP range by configuring the `/etc/docker/daemon.json` file. If the file does not exists, you can create it yourself. + +Assuming your safe IP range is `10.147.0.0/16` and `10.146.0.0/16`, your configuration would be: + +```json +{ + "default-address-pools": [ + { + "base": "10.147.0.0/16", + "size": 24 + }, + { + "base": "10.146.0.0/16", + "size": 24 + } + ] +} +``` + +To apply new Docker daemon configuration, restart your Docker service with `systemctl restart docker`. + +Make sure you are using [valid private IP ranges](https://en.wikipedia.org/wiki/Reserved_IP_addresses), that is between these ranges: +- `10.0.0.0/8` (address range of `10.0.0.0`–`10.255.255.255`) +- `100.64.0.0/10` (address range of `100.64.0.0`–`100.127.255.255`) +- `172.16.0.0/12` (address range of `172.16.0.0`–`172.31.255.255`) +- `192.0.0.0/24` (address range of `192.0.0.0`–`192.0.0.255`) +- `192.168.0.0/16` (address range of `192.168.0.0`–`192.168.255.255`) +- `198.18.0.0/15` (address range of `198.18.0.0`–`198.19.255.255`) + +For further reading, you can see Matthew Stratiotto's article on [The definitive guide to docker's default-address-pools option](https://straz.to/2021-09-08-docker-address-pools/). + +## Docker Logs Disk Usage + +If you are suspecting persisted logs from Docker container logs consumes a lot of your disk space, you can configure the amount of persisted logs on Docker by configuring the `/etc/docker/daemon.json` file. If the file does not exists, you can create it yourself. + +```json +{ + "log-driver": "local", + "log-opts": {"max-size": "10m", "max-file": "3"} +} +``` + +To apply new Docker daemon configuration, restart your Docker service with `systemctl restart docker`. + +If you want to delete immediate Docker logs, you can execute this as `root` user: + +```shell +truncate -s 0 /var/lib/docker/containers/**/*-json.log +``` + +## Docker Image and Builder Cleanup + +Executing `./install.sh` will build a new Sentry Docker container, executing it often might cause Docker to consume your disk space. You can safely prune old or unneeded Docker containers, image, or builder to re-acquire used disk space. Executing these will not affect current running containers and volumes. + +```shell +docker container prune +docker builder prune +docker image prune --all +# WARNING: Executing "volume prune" might delete `sentry-vroom` volume as it's not an external volume. +docker volume prune +docker network prune +``` + +Append `-f` flag for no confirmation on deletions. + ## Other -If you are still stuck, you can always visit our [GitHub issues](https://github.com/getsentry/self-hosted/issues) to search for existing issues or create a new issue and ask for help. Please keep in mind that we expect the community to help itself, but Sentry employees also try to monitor and answer questions when they have time. +If you are still stuck, you can always visit our [GitHub issues](https://github.com/getsentry/self-hosted/issues) to search for existing issues or create a new issue and ask for help. You can also visit [Sentry Community Discord server](https://discord.gg/sentry) on #self-hosted channel. Please keep in mind that we expect the community to help itself, but Sentry employees also try to monitor and answer questions when they have time.