I was writing a RestructuredText document with bunch of code. Having the document doctested seemed like a good idea.

Tox and pytest were the natural choices here as I don't have to futz around with bootstrapping and testing scripts.

So I make this tox.ini configuration:

[tox] envlist = py34 # because I only have bunch of documents for now ... skipsdist = true [testenv] deps = pytest process-tests mock tornado aspectlib commands = {posargs:py.test -vv} [pytest] norecursedirs = .git .tox addopts = -rxEfs --strict --doctest-modules --doctest-glob *.rst --tb short

So I was using Python 3.4 because I want to include some asyncio examples. Unfortunately horror followed:

py34 runtests: PYTHONHASHSEED='2663730781' py34 runtests: commands[0] | py.test -vv ============================= test session starts ============================= platform win32 -- Python 3.4.0 -- py-1.4.20 -- pytest-2.5.2 -- C:\docs\.tox\py34\Scripts\python.exe collected 1 items document.rst: [doctest] document.rst ERROR: InvocationError: 'C:\\docs\\.tox\\py34\\Scripts\\py.test.EXE -vv' ________________________________________________________________________________________________________ summary _________________________________________________________________ ERROR: py34: commands failed

Now at this point I'm wondering what the hell is going on. Silent failure? Trying to run py.test manually:

C:\docs>.tox\py34\Scripts\py.test -vv ============================= test session starts ============================= platform win32 -- Python 3.4.0 -- py-1.4.20 -- pytest-2.5.2 -- C:\docs\.tox\py34\Scripts\python.exe collected 1 items document.rst: [doctest] document.rst

Gah!

Now at this point I remember that py.test captures output by default. Could that cause problems? Seems so:

C:\docs>.tox\py34\Scripts\py.test -vv --capture no ============================= test session starts ============================= platform win32 -- Python 3.4.0 -- py-1.4.20 -- pytest-2.5.2 -- C:\docs\.tox\py34\Scripts\python.exe collected 1 items document.rst: [doctest] document.rst INTERNALERROR> Traceback (most recent call last): INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\main.py", line 81, in wrap_session INTERNALERROR> doit(config, session) INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\main.py", line 118, in _main INTERNALERROR> config.hook.pytest_runtestloop(session=session) INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 377, in __call__ INTERNALERROR> return self._docall(methods, kwargs) INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 388, in _docall INTERNALERROR> res = mc.execute() INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 289, in execute INTERNALERROR> res = method(**kwargs) INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\main.py", line 138, in pytest_runtestloop INTERNALERROR> item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem) INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 377, in __call__ INTERNALERROR> return self._docall(methods, kwargs) INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 388, in _docall INTERNALERROR> res = mc.execute() INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 289, in execute INTERNALERROR> res = method(**kwargs) INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\runner.py", line 64, in pytest_runtest_protocol INTERNALERROR> runtestprotocol(item, nextitem=nextitem) INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\runner.py", line 74, in runtestprotocol INTERNALERROR> reports.append(call_and_report(item, "call", log)) INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\runner.py", line 110, in call_and_report INTERNALERROR> report = hook.pytest_runtest_makereport(item=item, call=call) INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\main.py", line 162, in call_matching_hooks INTERNALERROR> return hookmethod.pcall(plugins, **kwargs) INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 381, in pcall INTERNALERROR> return self._docall(methods, kwargs) INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 388, in _docall INTERNALERROR> res = mc.execute() INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 289, in execute INTERNALERROR> res = method(**kwargs) INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\capture.py", line 246, in pytest_runtest_makereport INTERNALERROR> rep = __multicall__.execute() INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 289, in execute INTERNALERROR> res = method(**kwargs) INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\runner.py", line 208, in pytest_runtest_makereport INTERNALERROR> longrepr = item.repr_failure(excinfo) INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\_pytest\doctest.py", line 61, in repr_failure INTERNALERROR> filelines = py.path.local(filename).readlines(cr=0) INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\py\_path\common.py", line 134, in readlines INTERNALERROR> content = self.read('rU') INTERNALERROR> File "C:\docs\.tox\py34\lib\site-packages\py\_path\common.py", line 126, in read INTERNALERROR> return f.read() INTERNALERROR> File "C:\docs\.tox\py34\lib\encodings\cp1250.py", line 23, in decode INTERNALERROR> return codecs.charmap_decode(input,self.errors,decoding_table)[0] INTERNALERROR> UnicodeDecodeError: 'charmap' codec can't decode byte 0x83 in position 309: character maps to <undefined> ============================== in 0.20 seconds ===============================

It would appear I had some non-ascii characters in there. It's as unavoidable as my own surname (Mărieș - with a ș, not ş) - the document is in Română (Romanian). But cp1250 ?! My document is in utf-8 - what's going on here?

Turns out Python 3 will use GetACP to get the preferred encoding - which never seems to be the utf-8 code page (cp65001).

At this point I'm very intrigued and want to know if I could change that preferred encoding . But I can't:

C:\docs>py -3 -c "import locale; print(locale.setlocale(locale.LC_ALL, 'eng_US.65001'))" Traceback (most recent call last): File "<string>", line 1, in <module> File "C:\Python34\lib\locale.py", line 592, in setlocale return _setlocale(category, locale) locale.Error: unsupported locale setting

Note I've tried other stuff. Still could change the preferred encoding ...

Was the argument format incorrect? I think not. With cp1252 it works fine:

C:\docs>py -3 -c "import locale; print(locale.setlocale(locale.LC_ALL, 'eng_US.1252'))" English_United States.1252

I just give up and just monkey patch that open builtin. Because I still want pytest's test discovery and I don't want to write a script reading up all the documents myself. I should make a pytest bug report at least ...

Made a conftest.py and threw this in:

Note You think this is ugly? Try patching a builtin yourself.

import aspectlib aspectlib . weave ( open , lambda open : lambda name , mode = 'r' , * args , encoding = 'utf8' , ** kwargs : open ( name , mode , * args , ** kwargs ) if 'b' in mode else open ( name , mode , * args , encoding = encoding , ** kwargs ) )

You can argue that's like killing mosquitoes with a bazooka, but I can get back to my doctests:

py34 runtests: PYTHONHASHSEED='1866354792' py34 runtests: commands[0] | py.test -vv ============================= test session starts ============================= platform win32 -- Python 3.4.0 -- py-1.4.20 -- pytest-2.5.2 -- C:\docs\.tox\py34\Scripts\python.exe collected 1 items document.rst: [doctest] document.rst FAILED ================================== FAILURES =================================== __________________________ [doctest] document.rst ___________________________ 001 Silly stuff:: 002 003 >>> print(1) Expected: 2 Got: 1 C:\docs\document.rst:3: DocTestFailure =========================== short test summary info =========================== FAIL document.rst ========================== 1 failed in 0.08 seconds =========================== ERROR: InvocationError: 'C:\\docs\\.tox\\py34\\Scripts\\py.test.EXE -vv' ________________________________________________________________________________________________________ summary _____________________________ ERROR: py34: commands failed

Yay!