Tests do not run: missing file, wrong Python version

[gward]

I'm trying to run the tests using tox. Here's what I did to setup:

$ virtualenv -p /usr/bin/python3 venv
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /data/src/fortiosapi/venv/bin/python3
Also creating executable in /data/src/fortiosapi/venv/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

$ ./venv/bin/pip install tox

And here is how the test run went:

$ ./venv/bin/tox
GLOB sdist-make: /data/src/fortiosapi/setup.py
py27 inst-nodeps: /data/src/fortiosapi/.tox/.tmp/package/1/fortiosapi-0.10.3.zip
py27 installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.,asn1crypto==0.24.0,bcrypt==3.1.6,certifi==2018.11.29,cffi==1.11.5,chardet==3.0.4,cryptography==2.5,enum34==1.1.6,fortiosapi==0.10.3,idna==2.8,ipaddress==1.0.22,nose==1.3.7,oyaml==0.7,packaging==19.0,paramiko==2.4.2,pexpect==4.6.0,ptyprocess==0.6.0,pyasn1==0.4.5,pycparser==2.19,PyNaCl==1.3.0,pyparsing==2.3.1,PyYAML==3.13,requests==2.21.0,six==1.12.0,urllib3==1.24.1
py27 run-test-pre: PYTHONHASHSEED='2774607791'
py27 runtests: commands[0] | /data/src/fortiosapi/.tox/py27/bin/python -m unittest discover -v
test_fortiosapi_virsh (unittest.loader.ModuleImportFailure) ... ERROR

======================================================================
ERROR: test_fortiosapi_virsh (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_fortiosapi_virsh
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/data/src/fortiosapi/tests/test_fortiosapi_virsh.py", line 49, in <module>
    conf = yaml.load(open(virshconffile, 'r'))
IOError: [Errno 2] No such file or directory: 'virsh.yaml'


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)
ERROR: InvocationError for command '/data/src/fortiosapi/.tox/py27/bin/python -m unittest discover -v' (exited with code 1)
py35 create: /data/src/fortiosapi/.tox/py35
ERROR: InterpreterNotFound: python3.5
________________________________________ summary _________________________________________
ERROR:   py27: commands failed
ERROR:  py35: InterpreterNotFound: python3.5

I hacked tox.ini to use Python 3.6, since that is what my OS (Ubuntu 18.04) includes:

diff --git a/tox.ini b/tox.ini
index edb1521..b7e8071 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,5 +1,5 @@
 [tox]
-envlist = py27,py35
+envlist = py27,py36
 
 [testenv]
 changedir=tests
@@ -15,6 +15,6 @@ commands = {envbindir}/python -m unittest discover -v []
 basepython = python2.7
 
 
-[testenv:py35]
+[testenv:py36]
 #commands = {envbindir}/python -m unittest discover -v []
-basepython = python3.5
\ No newline at end of file
+basepython = python3.6

At least the InterpreterNotFound error is gone, but the tests still fail:

$ ./venv/bin/tox
GLOB sdist-make: /data/src/fortiosapi/setup.py
py27 inst-nodeps: /data/src/fortiosapi/.tox/.tmp/package/1/fortiosapi-0.10.3.zip
py27 installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.,asn1crypto==0.24.0,bcrypt==3.1.6,certifi==2018.11.29,cffi==1.11.5,chardet==3.0.4,cryptography==2.5,enum34==1.1.6,fortiosapi==0.10.3,idna==2.8,ipaddress==1.0.22,nose==1.3.7,oyaml==0.7,packaging==19.0,paramiko==2.4.2,pexpect==4.6.0,ptyprocess==0.6.0,pyasn1==0.4.5,pycparser==2.19,PyNaCl==1.3.0,pyparsing==2.3.1,PyYAML==3.13,requests==2.21.0,six==1.12.0,urllib3==1.24.1
py27 run-test-pre: PYTHONHASHSEED='3793182550'
py27 runtests: commands[0] | /data/src/fortiosapi/.tox/py27/bin/python -m unittest discover -v
test_fortiosapi_virsh (unittest.loader.ModuleImportFailure) ... ERROR

======================================================================
ERROR: test_fortiosapi_virsh (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_fortiosapi_virsh
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/data/src/fortiosapi/tests/test_fortiosapi_virsh.py", line 49, in <module>
    conf = yaml.load(open(virshconffile, 'r'))
IOError: [Errno 2] No such file or directory: 'virsh.yaml'


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)
ERROR: InvocationError for command '/data/src/fortiosapi/.tox/py27/bin/python -m unittest discover -v' (exited with code 1)
py36 inst-nodeps: /data/src/fortiosapi/.tox/.tmp/package/1/fortiosapi-0.10.3.zip
py36 installed: asn1crypto==0.24.0,bcrypt==3.1.6,certifi==2018.11.29,cffi==1.11.5,chardet==3.0.4,cryptography==2.5,fortiosapi==0.10.3,idna==2.8,nose==1.3.7,oyaml==0.7,packaging==19.0,paramiko==2.4.2,pexpect==4.6.0,ptyprocess==0.6.0,pyasn1==0.4.5,pycparser==2.19,PyNaCl==1.3.0,pyparsing==2.3.1,PyYAML==3.13,requests==2.21.0,six==1.12.0,urllib3==1.24.1
py36 run-test-pre: PYTHONHASHSEED='3793182550'
py36 runtests: commands[0] | /data/src/fortiosapi/.tox/py36/bin/python -m unittest discover -v
test_fortiosapi_virsh (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: test_fortiosapi_virsh (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_fortiosapi_virsh
Traceback (most recent call last):
  File "/usr/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/data/src/fortiosapi/tests/test_fortiosapi_virsh.py", line 49, in <module>
    conf = yaml.load(open(virshconffile, 'r'))
FileNotFoundError: [Errno 2] No such file or directory: 'virsh.yaml'


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)
ERROR: InvocationError for command '/data/src/fortiosapi/.tox/py36/bin/python -m unittest discover -v' (exited with code 1)
________________________________________ summary _________________________________________
ERROR:   py27: commands failed
ERROR:   py36: commands failed

[gward]

Ahh: there is a symlink tests/virsh.yaml -> virsh60.yaml in my source directory. But it's a dangling symlink, since virsh60.yaml does not exist.

[tacerus]

Does anyone know what virsh60.yaml is supposed to be? The tests seem to still be broken without that file.

[thomnico]

Hi, I left the company before documenting this feature. Explore: https://github.com/fortinet-solutions-cse/ftnt-sut-manager which uses it. It is basically telling tox the FGT to connect to (IP, user etc..)

[tacerus]

Very interesting, thank you @thomnico!