docs(self-hosted): revamp self hosted related documentation

[aldy505]

This is a work in progress, but I'll appreciate any reviews coming in. I need those anyways.

For context, on self-hosted we have this issue (getsentry/self-hosted#2267) that's going nowhere. My goal with this PR Is to add more documentation for resolving common self-hosted related issues.

Here are the things I'll write (or add) about on this PR (you can discuss it on the Discord thread):

• Specific upgrade commands (using Git workflow), because "download or check out version of self-hosted repository" is kinda ambiguous for new people self hosting sentry. (one Github issue here)
• Setting up load balancer (mainly HAProxy, Caddy, or Nginx) for production use
  • NGINX (@hubertdeng123 said he'll do this on the next PR)
  • HAProxy
  • Caddy
  • Traefik
• Some Docker specific configuration to lessen logs (see this Discord link) and to make it not break with people's private IP range (see this Github link).
• What are the things people can do to enable high traffic self-hosted issue other than migrating to SaaS (I know having users migrate to SaaS is a big win for Sentry, but some companies can't -- for me, it's due to government policy that anything related to customer's data must be kept on my company's own servers, having PII scrubing for this is not ideal, it's hard to debug stuff)
• How to set up filestore backend to use something other than filesystem such as 'gcs' or 's3' (from @hubertdeng123 -- to be honest I don't know how to setup storage backend other than filesystem) -- will be done later, we need to make vroom more configurable
• Enabling https for self hosted Sentry (from @hubertdeng123)
• Explain the licensing, because I've seen some people are afraid of the license. -- I'd need some review/feedbacks about these. It's legal stuff.
• Tell people to go to DIscord as alternative for GitHub issues

If anyone has any ideas or things they think is nice to add here, let me know.

[vercel]

@aldy505 is attempting to deploy a commit to the Sentry Team on Vercel.

A member of the Team first needs to authorize it.

[hubertdeng123]

Just curious, where did you get this number from?

[aldy505]

Observing my own instance as well as a few of my friends.

I found that if you need to go beyond 1 million events, you'd tweak retention settings or bump the machine resource.

It's very subjective, but I guess that's what the server can handle with the default machine resource.

[aldy505]

Hey @hubertdeng123 can you help with the NGINX configuration? I don't really use NGINX, so I don't know what's best for implementing reverse proxy + rate limit + specific endpoint handler + ACME TLS.

[aldy505]

I really wanted to add how they can actually use Kubernetes for scaling out, but that's not officially supported. Is it okay to put it here as "community stuff"?

A few stuff that I want to put here:

• https://github.com/sentry-kubernetes/charts
• https://github.com/kanadaj/sentry-operator

[hubertdeng123]

@chadwhitacre what is the company policy on doing this? I would refrain from putting specific links to repos that aren't getsentry repos if we can help it.

[chadwhitacre]

Yeah we need to draw the line somewhere and this seems like a good place to draw it @hubertdeng123.

[hubertdeng123]

Hey @hubertdeng123 can you help with the NGINX configuration? I don't really use NGINX, so I don't know what's best for implementing reverse proxy + rate limit + specific endpoint handler + ACME TLS.

Yep, I can go about adding this documentation for this.

[aldy505]

Hey @hubertdeng123 can you help with the NGINX configuration? I don't really use NGINX, so I don't know what's best for implementing reverse proxy + rate limit + specific endpoint handler + ACME TLS.

Yep, I can go about adding this documentation for this.

Do you want to do that on a separate PR? I'll finish up here if you do.

[hubertdeng123]

Do you want to do that on a separate PR? I'll finish up here if you do.

Yes, I'll do this in a follow up PR

[hubertdeng123]

This is a great start! Loving the details you're putting here.

[hubertdeng123]

@chadwhitacre what is the company policy on doing this? I would refrain from putting specific links to repos that aren't getsentry repos if we can help it.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
develop	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 24, 2024 10:58pm

[chadwhitacre]

Thanks for jumping in and doing this @aldy505! This makes me think we need to get clearer about the line between "small volume and proof of concept deploy" which we want to officially support and "high availability deploy" which we do not want to officially support. Docs for the former are great to have on develop.sentry.dev; docs for the latter belong elsewhere, and as @hubertdeng123 suggests, we would want to let people discover those resources on their own (we can be like "go google sentry kubernetes" but not link directly).

[chadwhitacre]

This makes me think we need to get clearer about the line between "small volume and proof of concept deploy"

My initial thoughts on this are that we support an architecture with one web/snuba container with data services external to that container. Scaling up web nodes would be out of scope. Does that seem like a good way to slice it?

[aldy505]

This makes me think we need to get clearer about the line between "small volume and proof of concept deploy"

My initial thoughts on this are that we support an architecture with one web/snuba container with data services external to that container. Scaling up web nodes would be out of scope. Does that seem like a good way to slice it?

I think the "data services external to that container" will cause confusion to some. For me, that'd mean configuring standalone Postgres using AWS RDS, standalone ClickHouse Cloud, and the like, alongside with the configuration of called services is supported officially. Although the fact that it is supported for this kind of deployment, it'd be hard to satisfy everyone's needs.

Maybe for simpler terms: We only support this kind of deployment model (docker compose stuff), for scaling up any of Sentry's containers, you'll need to figure it on your own, or find community implementation of it.

[aldy505]

Any chance to take a look at this before the end of the week? @hubertdeng123 @chadwhitacre @azaslavsky

[azaslavsky]

Hi @aldy505, sorry for the delayed response!

So here’s where we’re at: anything we add to the docs comes with some implicit level of support. We’re a small team, and are worried about increasing the scope of self-hosted to be much larger than it currently is, especially when it comes to products/services that we don’t have much experience with and that are not used internally at Sentry (ex: Traefik, Caddy).

The purpose of this repository is to be a small, self-contained instance of Sentry, suitable for deployment on a single server. Concerns regarding larger deployments are explicitly left out, since they are more related to generic DevOps and scaling work than to running Sentry as a product. For example: load balancing is a general concern related to any web service as it scales, not to Sentry in particular, and something that users comfortable with self-hosting should generally be able to set up on their own.

All that to say: we're talking internally about what our limits are, what we do/don’t want to support, and what kind of explicit commitments we want to make with that support. That is taking a bit of time on our end as we work out those boundaries.

We really appreciate you being patient with us, and for the incredible enthusiasm toward improving self-hosted. We’ll have something more concrete for you shortly.

[aldy505]

All that to say: we're talking internally about what our limits are, what we do/don’t want to support, and what kind of explicit commitments we want to make with that support. That is taking a bit of time on our end as we work out those boundaries.

Hi @azaslavsky, is there any news for this? Or do we need to setup a sync meet to make this faster, because docs for self-hosted is pretty outdated and there's so much repeating problem-debug-solution happening all around Discord and GitHub lately.

cc @hubertdeng123 @chadwhitacre

[chadwhitacre]

I finally took the time to read through these new docs. In general this looks pretty great, @aldy505, thank you! :) I see opportunities to clean up for style, but I don't think we need to block on style, we can clean that up later.

I do think we want some structural changes to the main new "Productionalizing" page:

1. Rename "Productionalizing" to "Proxying"
2. Search/replace "load balancer" with "proxy."
3. Move the "Recommended system resource" section from Proxying to Overview.
4. Move the "Installing Behind a Proxy" section from Overview to Proxying

Also, for future PRs, it will be easier for us to digest and accept smaller PRs rather than one so large as this. :-)

[aldy505]

1. Rename "Productionalizing" to "Proxying"
2. Search/replace "load balancer" with "proxy."
3. Move the "Recommended system resource" section from Proxying to Overview.
4. Move the "Installing Behind a Proxy" section from Overview to Proxying

@chadwhitacre I don't think it's a good idea to combine "network proxy" (number 4) with "reverse proxies" (number 1-3), this will be confusing for future readers. How about we keep the "Installing Behind a Proxy" on the Overview page, and rename "Productionalizing" to "Reverse Proxy"? I think most problems with users that want to self-hosted Sentry is not knowing what to settings put on their outermost reverse proxy/load balancer, like the event ingestion endpoints, what will happen if a rate limiter is placed, and so on forth. There's very minimal documentation for that.

[chadwhitacre]

How about we keep the "Installing Behind a Proxy" on the Overview page, and rename "Productionalizing" to "Reverse Proxy"?

That works for me. Thanks.

[chadwhitacre]

😭

[hubertdeng123]

#using-dedicated-load-balancer here is now removed, we'll need to update this to get the build passing

[chadwhitacre]

Thanks for seeing this through, @aldy505! 😁