jupyterhub does not pass env variables

[jelmd]

Bug description

Even if c.Spawner.env_keep is set as expected, jupyterhub does not pass these variables to sudospawner and thus sudospawner fails with

...
[W 2021-09-06 05:20:18.191 JupyterHub auth:1079] Failed to open PAM session for foobar: [PAM Error 14] Cannot make/remove an entry for the specified session
[W 2021-09-06 05:20:18.192 JupyterHub auth:1080] Disabling PAM sessions from now on.
Traceback (most recent call last):
  File "/usr/local/jupyter/bin/sudospawner", line 8, in <module>
    from sudospawner import mediator
ModuleNotFoundError: No module named 'sudospawner'
[E 2021-09-06 05:20:18.275 JupyterHub spawner:95] Failed to get JSON result from mediator: ''
[E 2021-09-06 05:20:18.276 JupyterHub user:720] Unhandled error starting foobar's server: substring not found
[W 2021-09-06 05:20:18.327 JupyterHub web:1787] 500 GET /jupyter/n00-00/hub/spawn/foobar (::ffff:10.2.0.10): Error in Authenticator.pre_spawn_start: ValueError substring not found
...

Expected behaviour

sudospawner should run.

Actual behaviour

sudospawner invocation fails.

How to reproduce (Ubuntu 18.04 (bionic), minimal server)

Use any vanilla ubuntu 18.04 install and execute as unprivileged user (example uses user admin which belongs to group staff) - copy and paste should work:

bash

sudo apt-get install npm nodejs \
  python3-pip python3-setuptools python3-wheel python3-cffi libffi-dev \
  python3-dev python3-jinja2 python3-six python3-dbg python3-venv

cat >/var/tmp/jupyter.env<<EOF
export JUPYTER_PATH=/usr/local/jupyter
export PYTHONUSERBASE=\${JUPYTER_PATH}
PATH=\${JUPYTER_PATH}/bin:\${PATH}
export PYTHONPATH=\${JUPYTER_PATH}/lib/python3.6/site-packages
export NPM_CONFIG_GLOBALCONFIG=\${JUPYTER_PATH}/npmrc
EOF

source /var/tmp/jupyter.env
sudo mkdir -p ${JUPYTER_PATH}
sudo rm -rf ${JUPYTER_PATH}/*
sudo chown admin:staff ${JUPYTER_PATH}
printf "prefix=${JUPYTER_PATH}\n" >${JUPYTER_PATH}/npmrc
cp /var/tmp/jupyter.env  ${JUPYTER_PATH}/env

# just to make sure everything is clean
rm -rf ~/{.local,.cache/pip,node_modules}
python3 -m pip install --upgrade pip

# install jupyter
pip3 install jupyter
python3  -m ipykernel install --prefix=${JUPYTER_PATH}
sudo mkdir -p /usr/share/jupyter
sudo ln -sf ${JUPYTER_PATH}/share/jupyter/kernels /usr/share/jupyter/kernels

# install jupyterhub
pip3 install jupyterhub
pip3 install git+https://github.com/jupyter/sudospawner
cat >${JUPYTER_PATH}/jupyterhub_config.py<<EOF
c.JupyterHub.spawner_class = 'sudospawner.SudoSpawner'
c.SudoSpawner.sudospawner_path = '/usr/local/jupyter/bin/sudospawner'
c.Spawner.env_keep = [ 'PATH', 'PYTHONPATH', 'LD_LIBRARY_PATH', 'JUPITER_PATH' ]
import os
import sys
for var in os.environ:
    print (var, os.environ[var], file=sys.stderr)
    c.Spawner.env_keep.append(var)
print ('cfg DONE.', file=sys.stderr)
c.Spawner.notebook_dir = '~/'
c.Spawner.args = ['--NotebookApp.allow_origin=*']
c.NotebookApp.allow_origin = '*'
c.JupyterHub.base_url = '/jupyter/n00-00'
c.Authenticator.admin_users = set(['admin'])
EOF

# install the proxy
npm install --only=production --no-optional --global \
        configurable-http-proxy@4.2.2

# let user juppy run the service
sudo useradd -u 123 -g 4 -c 'JupiterHub Service' -m -r juppy
printf "juppy ALL=(%jupyter) NOPASSWD: ${JUPYTER_PATH}/bin/sudospawner\n" >/tmp/x
sudo mv /tmp/x /etc/sudoers.d/jupyterHub
# make sure /etc/nsswitch.conf contians a something like this:
# `sudoers: files`

cat >/tmp/x<<EOF
#!/bin/bash
cd ~ && pwd || exit 1
source ${JUPYTER_PATH}/env
${JUPYTER_PATH}/bin/jupyterhub -f ${JUPYTER_PATH}/jupyterhub_config.py >jupyterhub.log 2>&1 &
EOF
chmod 755 /tmp/x
sudo mv  /tmp/x /etc/init.d/jupyterhub
# that's it wrt. the setup

# optionally for debug one may overwrite sudospwaner, e.g.
cat >>${JUPYTER_PATH}/bin/sudospawner<<EOF
#!/usr/bin/python3

import os
import sys
for var in os.environ:
    print (var, os.environ[var],file=sys.stderr)

from sudospawner import mediator
mediator.main()
EOF

# to start the service manually, run e.g.:
sudo /bin/su -c /etc/init.d/jupyterhub juppy

# and go to the following URL and try to login
firefox http://${yourbox}:8000//jupyter/n00-00/

# and finally have a look at the log:
cat ~juppy/jupyterhub.log

Your personal set up

• OS: Ubuntu 18.04
• Version(s): jupiterhub = 1.4.2, python = 3.6

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[Frank-Steiner]

Having exactly the same problem. sudospawner fails to start as the neccessary PYTHONPATH variable get's lost despite c.Spawner.env_keep is listing it.

[Frank-Steiner]

Ok, here's how to solve it (should be documented somewhere in the sudospawner webpage I guess):

1. to make the neccessary variables be passed to sudospawner, use
   Defaults env_keep += "PYTHONPATH JUPYTER_PATH WHATEVER_ELSE"
   in the sudoers file.
2. Now sudospawner will start, but mediator removes the PYTHONPATH variable, so we run into the same problem just one step later

Feb 09 17:44:09 myhost start_jhub[8254]: [I 230209 17:44:09 mediator:89] Spawning /opt/jhub/bin/jupyterhub-singleuser
Feb 09 17:44:09 myhost start_jhub[8254]: [W 230209 17:44:09 mediator:91] PYTHONPATH env not allowed for security reasons
Feb 09 17:44:09 myhost start_jhub[8254]: Traceback (most recent call last):
Feb 09 17:44:09 myhost start_jhub[8254]:   File "/opt/jhub/bin/jupyterhub-singleuser", line 5, in <module>
Feb 09 17:44:09 myhost start_jhub[8254]:     from jupyterhub.singleuser import main
Feb 09 17:44:09 myhost start_jhub[8254]: ModuleNotFoundError: No module named 'jupyterhub'

According to https://github.com/jupyterhub/sudospawner we can solve this by creating a file sudospawner-singleuser. This will be called from sudospawner instead of jupyterhub-singleuser, so we can set PYTHONPATH again and then call jupyterhub-singleuser ourself:

#!/bin/bash 

export PYTHONPATH=<set your path>
exec "$(dirname "$0")/jupyterhub-singleuser" $@

And now it works.

[jelmd]

The problem is, that you do not pass PYTHONPATH but set it to a new, possibly unexpected value ...

Therefore I use the following workaround in our cluster:

export JUPYTER_PATH=/usr/local/jupyter
#...
    sed -i.orig -e "/self.sudo_args/ a\        if 'PYTHONPATH' in os.environ:\n            cmd.append('='.join(['PYTHONPATH', os.environ['PYTHONPATH']]))" \
        ${JUPYTER_PATH}/lib/python${PY_VERS}/site-packages/sudospawner/spawner.py
    sed -i.orig -e "/if'PYTHONPATH' in env:/ s/:/ and 'NO_PYTHONPATH' in env:/"\
        ${JUPYTER_PATH}/lib/python${PY_VERS}/site-packages/sudospawner/mediator.py
#...
    if [[ ! -e /usr/local/jupyter/bin/jupyterhub-singleuser ]]; then
        print '#!/bin/bash -l

unset HTTP_PROXY HTTPS_PROXY
if [[ -z ${PY_VERSION} ]]; then
    case "$( lsb_release -rs )" in
        20.04) export PY_VERSION=3.8 ;;
        18.04) export PY_VERSION=3.6 ;;
    esac
fi

PYTHONPATH="${PYTHONPATH}:/usr/local/jupyter/lib/python${PY_VERSION}/site-packages"
exec /usr/local/jupyter/bin/jupyterhub-singleuser "$@"
'           >/usr/local/jupyter/bin/jupyterhub-singleuser
    fi

Have fun ;-)