"Unable to get lock on the directory" as several git processes continue running

[chrisguettler]

Tried fixing this for quite some time, but couldn't find a solution by searching. Maybe this is known already, but couldn't find an issue, except #873

Description

After starting jupyterlab and entering any repository the git sidebar shows as expected. As soon as I run any git action (refresh, pull, push, checkouts) or the extension does the refresh by itself the pull / push buttons get greyed out and I get message "Unable to get lock on the directory", when I try to pull/push. No more git actions possible.

I checked if .git/index.lock got stuck, but it doesn't exist.
I checked processes. No git processes before git processes, any action in interface spins off several git processes, which continue to run, unless killed manually:

 PID TTY          TIME CMD
 23689 pts/8    00:00:58 screen
 48703 pts/13   00:00:37 screen
 94134 pts/1    00:00:00 ps
102221 pts/18   14:53:05 jupyter-noteboo
124261 pts/0    00:00:12 jupyter-lab
127234 pts/0    00:00:00 git
127246 pts/0    00:00:00 git
127253 pts/0    00:00:00 git
127255 pts/0    00:00:00 git-remote-http

When I run killall git, the pull/push buttons are black again, but only until next refresh or other action.
Further I checked permission rights for ./.git directory, which look fine:
drwxrwxr-x 8 christian christian 4096 Jul 12 14:40 .git
It looks like too many git processes are launched and block each other. In previous version's normal behaviour only one process runs and stops after finishing.

• git is working in terminal (tested separately, but not parallel), credentials stored with helper.

Reproduce

1. Go to jupyterlab and enter repository in file browser
2. Click on git extension sidebar
3. Either wait for automatic refresh or do any git action (pull/push/checkout branch)
4. Receive error message "Unable to get lock on the directory"
   I reproduced it with other linux machine as well.

Expected behavior

• git processes started by jupyterlab-git should finish after completion.
• Also tried running the extention in version 0.22.1 in jupyterlab 2.2.8 and it works just fine (of course, this was tested not in parallel). In this setup the git processes quit after finishing.

Command Line Output Error (jupyter lab --debug)

Context

• Python package version:

Name: jupyterlab-git
Version: 0.30.1

• Extension version:

JupyterLab v3.0.16
/home/christian/anaconda3/envs/jl3/share/jupyter/labextensions
        jupyterlab-execute-time v2.0.4 enabled OK (python, jupyterlab_execute_time)
        nbdime-jupyterlab v2.1.0 enabled OK
        @jupyter-widgets/jupyterlab-manager v3.0.0 enabled OK (python, jupyterlab_widgets)
        @ryantam626/jupyterlab_code_formatter v1.4.10 enabled OK (python, jupyterlab-code-formatter)
        @jupyterlab/git v0.30.1 enabled OK (python, jupyterlab-git)

• Git version:
  git version 2.30.0
• Operating System and its version:
  Ubuntu 18.04.5 LTS

Environment info

## Complete Environment :
Please see environment.yml file as txt attached (I counldn't upload yml).
[environment_jl3test.yml.txt](https://github.com/jupyterlab/jupyterlab-git/files/6802070/environment_jl3test.yml.txt)
```
$ conda list
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
anyio                     3.2.1            py39hf3d152e_0    conda-forge
appdirs                   1.4.4                    pypi_0    pypi
argon2-cffi               20.1.0           py39h3811e60_2    conda-forge
async_generator           1.10                       py_0    conda-forge
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
autopep8                  1.5.7                    pypi_0    pypi
babel                     2.9.1              pyh44b312d_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
black                     21.6b0                   pypi_0    pypi
bleach                    3.3.0              pyh44b312d_0    conda-forge
brotlipy                  0.7.0           py39h3811e60_1001    conda-forge
ca-certificates           2021.7.5             h06a4308_1
certifi                   2021.5.30        py39h06a4308_0
cffi                      1.14.5           py39he32792d_0    conda-forge
chardet                   4.0.0            py39hf3d152e_1    conda-forge
click                     8.0.1                    pypi_0    pypi
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
cryptography              3.4.7            py39hbca0aa6_0    conda-forge
decorator                 5.0.9              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
entrypoints               0.3             pyhd8ed1ab_1003    conda-forge
gitdb                     4.0.7              pyhd8ed1ab_0    conda-forge
gitpython                 3.1.18             pyhd8ed1ab_0    conda-forge
icu                       68.1                 h2531618_0
idna                      2.10               pyh9f0ad1d_0    conda-forge
importlib-metadata        4.6.0            py39hf3d152e_0    conda-forge
ipykernel                 5.5.5            py39hef51801_0    conda-forge
ipython                   7.25.0           py39hef51801_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jedi                      0.18.0           py39hf3d152e_2    conda-forge
jinja2                    3.0.1              pyhd8ed1ab_0    conda-forge
json5                     0.9.5              pyh9f0ad1d_0    conda-forge
jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge
jupyter-server-mathjax    0.2.3              pyhd8ed1ab_0    conda-forge
jupyter_client            6.1.12             pyhd8ed1ab_0    conda-forge
jupyter_core              4.7.1            py39hf3d152e_0    conda-forge
jupyter_server            1.9.0              pyhd8ed1ab_0    conda-forge
jupyterlab                3.0.16             pyhd8ed1ab_0    conda-forge
jupyterlab-git            0.30.1             pyhd8ed1ab_0    conda-forge
jupyterlab_code_formatter 1.4.10             pyhd8ed1ab_1    conda-forge
jupyterlab_execute_time   2.0.4              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
jupyterlab_server         2.6.0              pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        1.0.0              pyhd8ed1ab_1    conda-forge
ld_impl_linux-64          2.35.1               hea4e1c9_2    conda-forge
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-ng                 9.3.0               h2828fa1_19    conda-forge
libgomp                   9.3.0               h2828fa1_19    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libstdcxx-ng              9.3.0               h6de172a_19    conda-forge
libuv                     1.40.0               h7b6447c_0
markupsafe                2.0.1            py39h3811e60_0    conda-forge
matplotlib-inline         0.1.2              pyhd8ed1ab_2    conda-forge
mistune                   0.8.4           py39h3811e60_1003    conda-forge
mypy-extensions           0.4.3                    pypi_0    pypi
nbclassic                 0.3.1              pyhd8ed1ab_1    conda-forge
nbclient                  0.5.3              pyhd8ed1ab_0    conda-forge
nbconvert                 6.1.0            py39hf3d152e_0    conda-forge
nbdime                    3.1.0              pyhd8ed1ab_0    conda-forge
nbformat                  5.1.3              pyhd8ed1ab_0    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
nest-asyncio              1.5.1              pyhd8ed1ab_0    conda-forge
nodejs                    14.8.0               hda19d22_0
notebook                  6.4.0              pyha770c72_0    conda-forge
openssl                   1.1.1k               h27cfd23_0
packaging                 20.9               pyh44b312d_0    conda-forge
pandoc                    2.14.0.3             h7f98852_0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parso                     0.8.2              pyhd8ed1ab_0    conda-forge
pathspec                  0.8.1                    pypi_0    pypi
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pip                       21.1.3             pyhd8ed1ab_0    conda-forge
prometheus_client         0.11.0             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.19             pyha770c72_0    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pycodestyle               2.7.0                    pypi_0    pypi
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pygments                  2.9.0              pyhd8ed1ab_0    conda-forge
pyopenssl                 20.0.1             pyhd8ed1ab_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyrsistent                0.17.3           py39h3811e60_2    conda-forge
pysocks                   1.7.1            py39hf3d152e_3    conda-forge
python                    3.9.5           h49503c6_0_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python_abi                3.9                      2_cp39    conda-forge
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
pyzmq                     22.1.0           py39h37b5a0c_0    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
regex                     2021.4.4                 pypi_0    pypi
requests                  2.25.1             pyhd3deb0d_0    conda-forge
requests-unixsocket       0.2.0                      py_0    conda-forge
send2trash                1.7.1              pyhd8ed1ab_0    conda-forge
setuptools                49.6.0           py39hf3d152e_3    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     3.0.5              pyh44b312d_0    conda-forge
sniffio                   1.2.0            py39hf3d152e_1    conda-forge
sqlite                    3.36.0               h9cd32fc_0    conda-forge
terminado                 0.10.1           py39hf3d152e_0    conda-forge
testpath                  0.5.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.10               h21135ba_1    conda-forge
toml                      0.10.2                   pypi_0    pypi
tornado                   6.1              py39h3811e60_1    conda-forge
traitlets                 5.0.5                      py_0    conda-forge
typing_extensions         3.10.0.0           pyha770c72_0    conda-forge
tzdata                    2021a                he74cb21_0    conda-forge
urllib3                   1.26.6             pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          0.57.0           py39hf3d152e_4    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zeromq                    4.3.4                h9c3ff4c_0    conda-forge
zipp                      3.4.1              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge
```

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[fcollonval]

Thanks @chrisguettler for the detailed report. We definitely need to figure out what is going out as you seem to hit a lock situation.

Some questions:

• when listing the process could you run ps aux? so we have the full executed command, not only the application name.
• does your system as complex remote access like entreprise proxy?
• in the extension settings, you can increase the delay between two refresh status call. Could you try to increase it by a huge amount to see if this is the command that triggers the lock?
• is the credentials helper asking for some validation or are the credentials stored once and for all - so that you don't need to set them when starting JupyterLab?

[chrisguettler]

Hi @fcollonval,
sorry for not having answered earlier (business & vacation).

ps aux shows:

christi+  72915  0.0  0.0  18932  3964 pts/8    S+   07:03   0:00 git fetch --all --prune
christi+  72916  0.0  0.0  19196  3952 pts/8    S+   07:03   0:00 /usr/lib/git-core/git fetch --append --prune origin
christi+  72917  0.0  0.0 196472 22016 pts/8    S+   07:03   0:00 /usr/lib/git-core/git-remote-https origin https://...

The envrionment runs on a remote enterprise setup, but no proxy involved, just SSH connection.

When I extend the refresh intervall the automatic refresh does not cause the issue anymore.

I didn't store credentials in with the helper.
I noticed, as soon as I kill all git processes, it's prompting for credentials in the terminal, where jupyterlab is running.

[E 2021-08-25 11:06:48.376 ServerApp] 500 POST /git/remote/fetch?1629882235718 (10.23.10.13) 172617.79ms referer=http://[machine address]:8882/lab
Password for [repo]: 

But it did not prompt for it in the browser. Website is shown as "not secure" in Chrome. So I allowed pop-ups and unsecure contents for the address. It now did prompt for username/password and doesn't block anymore. so far so good.
Storing credentials (local) in helper also helps, but only temporarily.

If I set back refresh intervall to 3000 and restart JL3 the issue continues unless I kill all git processes again and enter the credentials in the JL-Terminal, again no prompt in the browser.
If I leave the refresh intervall incredibly high, I have no issues again, manual refresh, pull, push works fine.
It looks like the refresh caused by the intervall, doesn't prompt for credentials.

By the way, if I set the intervall to higher than 2000000000 or so the git panel disappears, when reloading the page.

[fcollonval]

Thanks for the detailed feedback.

[hstarmans]

I had the same problem, two options;

• ssh keys; you need to add the key to the manager!!
  see https://stackoverflow.com/questions/1595848/configuring-git-over-ssh-to-login-once
• https login use something like git config --global credential.helper cache

[bhavyalavarthi]

Is this issue resolved? Are there any suggested workarounds?

What are exact steps that I should follow to get past the following error, while trying to pull.

Error: Unable to get the lock on the directory

Can someone help me on this. Thank you.

[hstarmans]

I get the problem if l try to use jupyter lab with git on another computer than my own. The workaround is set up git with ssh keys

[nsingl00]

I raised a PR to fix this. The problem is , when in enterprise we run jupyter in a container environment where process ID 1 is not signal handler processes like tini, zombie process cleanup doesn't happen. Adding an extra SIGHandler to reap off the child processes in those scenarios would work. Added the PR for those cases.