Watson: Codebase review

May 2019

Project information

Python version support

At first, I had installed the Debian 10 packages (python3-watson and watson), but I wanted to review and use the latest version, preferably in a virtualenv, so I decided to get the latest code from the GitHub repository.

Just by looking at the Debian package names you can notice that Watson supports Python 3, but does it also support Python 2?

Let's take a look at the tox.ini file which is used for running tests in multiple python environments:

[tox]
envlist = flake8,py27,py34,py35,py36,py37

We can see that Watson effectively supports python 2.7 which is pretty typical these days.

Dependencies

Runtime

Being packaged in Debian, this means we can just look up for its runtime dependencies in a Debian system:

$ apt-cache depends watson python3-watson | awk '/Depends: [^<]/ {print $2}' | xargs dpkg -l | grep ii
ii  python3-arrow    0.12.1-2     all   Python3 library to manipulate dates, times, and timestamps
ii  python3-click    7.0-1        all   Wrapper around optparse for command line utilities - Python 3.x
ii  python3-requests 2.21.0-1     all   elegant and simple HTTP library for Python3, built for human beings
ii  python3-watson   1.6.0-6      all   Library for Watson (Python 3)

Development

Besides this high-level info, there are two requirements files in the root folder, but why? Let's grep around and see if we can get more info:

# some results are not shown as seemed irrelevant to me
$ grep -r requirements *
Makefile:   $(PIP) install -r requirements-dev.txt
MANIFEST.in:include requirements-dev.txt
MANIFEST.in:include requirements.txt
setup.py:    install_requires=parse_requirements('requirements.txt'),
setup.py:    tests_require=parse_requirements('requirements-dev.txt'),
docs/contributing/hack.md:        $ pip install -r requirements-dev.txt

As for the requirements-dev.txt, we can see tools for running the tests, generating the documentation, and more:

flake8, which wraps pep8, pyflakes, mccabe and more, to check the quality and style of python code.
mkdocs, to build the docs
mock, for creating mocks for testing
py, a development support library, which is now deprecated.
pytest, a (simpler than unittest) testing framework, along with some extensions (datafiles, mock and runner)
tox, which aims to standardize testing in python
twine, a utility for publishing python packages on PyPI.

Folder organization

The file structure is as follows:

docs/: markdown documentation; some of it is automatically generated with make docs.
scripts/: auxiliary scripts.
tests/: tests, mainly unit tests.
watson/: source code.
Root dir:
- travis ci
- testing with tox
- setup config for package generation
- Makefile for automation
- shell completion scripts
- python package requirements
- license
- other docs

Code organization

$ cloc --by-file watson
--------------------------------------------------------------------------------
File                              blank        comment           code
--------------------------------------------------------------------------------
watson/cli.py                       237            445            848
watson/watson.py                    110             50            395
watson/fullmoon.py                    7              6            220
watson/utils.py                      53             63            168
watson/frames.py                     38              0            117
watson/config.py                     33             47             32
watson/__main__.py                    1              0              5
watson/__init__.py                    1              0              3
watson/version.py                     1              1              1
--------------------------------------------------------------------------------
SUM:                                481            612           1789
--------------------------------------------------------------------------------

$ cloc --by-file tests
------------------------------------------------------------------------------------
File                                  blank        comment           code
------------------------------------------------------------------------------------
tests/test_watson.py                    251             44            570
tests/test_utils.py                      56              4            170
tests/test_config.py                     24             47             82
tests/__init__.py                        15              1             28
tests/test_fullmoon.py                    6              1             19
tests/conftest.py                         6              1              8
------------------------------------------------------------------------------------
SUM:                                    358             98            877
------------------------------------------------------------------------------------

Testing

You can run all tests in all available Python environments in your system like this:

$ tox

Watson uses pytest along with unittest.mock to do unit testing.

Let's understand some testing constructs used in Watson:

mock.patch.object(): patch the named member (attribute) on an object (target) with a mock object.

# In tests/__init__.py
# path.object(target, attribute, new) with new being the new object to replace the target
return mock.patch.object(dt_module, 'datetime', MockedDateTime)

@pytest.mark.parametrize(): enables parametrization of arguments for a test function documentation.
Pytest fixtures: some functions receive pre-filled arguments, like watson, mock, or config_dir. These are fixtures used as function arguments.

$ pytest --fixtures
--------------------------------------- fixtures defined from pytest_datafiles ----------------------------------------
datafiles
    pytest fixture to define a 'tmpdir' containing files or directories
    specified with a 'datafiles' mark.

------------------------------------------ fixtures defined from pytest_mock ------------------------------------------
mocker
    return an object that has the same interface to the `mock` module, but
    takes care of automatically undoing all patches after each test method.
mock
    Same as "mocker", but kept only for backward compatibility.

---------------------------------------- fixtures defined from tests.conftest -----------------------------------------
watson
    tests/conftest.py:14: no docstring available
config_dir
    tests/conftest.py:9: no docstring available

--------------------------------------- fixtures defined from tests.test_watson ---------------------------------------
json_mock
    tests/test_watson.py:31: no docstring available