Skip to content

Set charset utf8mb4 for all dbs #1236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: release
Choose a base branch
from
Open

Set charset utf8mb4 for all dbs #1236

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
494cf24
fix: set charset utf8mb4 for all databases
angonz May 20, 2025
e39c47c
docs: add changelog entry for setting charset
angonz May 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/20250520_174054_andres_set_charset_for_all_dbs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- [Improvement] Set charset utf8mb4 for all databases, not only ran by Tutor. (by @angonz)
2 changes: 0 additions & 2 deletions tutor/templates/apps/openedx/config/partials/auth.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,7 @@ DATABASES:
ATOMIC_REQUESTS: true
OPTIONS:
init_command: "SET sql_mode='STRICT_TRANS_TABLES'"
{%- if RUN_MYSQL %}
charset: "utf8mb4"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Danyal-Faheem do we even need this? Mysql 8.4 default charset is utf8mb4 (https://dev.mysql.com/doc/refman/8.4/en/charset.html).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure, but our database runs in a 8.0.mysql_aurora.3.05.2 and we had problems when this was not set, even though the charset and collation was properly set in all logical dbs and tables.

We noted this when migrating the forum db from MongoDB to MySQL. Many entries had emojis in the title and/or body, and the migration script raised a database error when the DB was open without this option. After adding this option, it worked perfectly well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with enforcing this for external databases is that we do not have control over which MySQL version is being used. This option is the default for Mysql v8+. If someone is running a prior version, that can cause issues.
Since openedx-auth patch is provided, that patch can be used to override these settings.

@Danyal-Faheem Please correct me if I misquoted anything.

DATABASES:
  default:
    ENGINE: "django.db.backends.mysql"
    HOST: "{{ MYSQL_HOST }}"
    PORT: {{ MYSQL_PORT }}
    NAME: "{{ OPENEDX_MYSQL_DATABASE }}"
    USER: "{{ OPENEDX_MYSQL_USERNAME }}"
    PASSWORD: "{{ OPENEDX_MYSQL_PASSWORD }}"
    ATOMIC_REQUESTS: true
    OPTIONS:
      init_command: "SET sql_mode='STRICT_TRANS_TABLES'"
      charset: "utf8mb4"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@angonz Hi, does this need further action? Thanks

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @DawoudSheraz, I want to support Andrés point a little bit. In one of our recent upgrades to sumac we faced the exact same issue.

I'm not quite sure why, but the MySQL client used by Django is defaulting to utf8mb3, so even though the tables in the database are using the correct collation and charset, the connection is wrong.

I would argue that these setting values shouldn't be about external or internal databases, but rather about the database version. If I'm using the same DB as tutor then all the defaults should work the same. If I'm not, then it should be my responsibility to adjust accordingly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Django defaulted the charset to utf8mb4 in Django v5.2 https://docs.djangoproject.com/en/5.2/releases/5.2/#database-backends.

Since this can be a potentially breaking change, this should be merged into main instead of release. Merging in the current release can impact any existing users who are using external DBs and are not using the tutor version.

{%- endif %}
EMAIL_HOST_USER: "{{ SMTP_USERNAME }}"
EMAIL_HOST_PASSWORD: "{{ SMTP_PASSWORD }}"
{{ patch("openedx-auth") }}