[pytest-dev] monkeypatch.setattr of functions and from imports

[Daniel Nouri]

Dear all

I have a function 'somefunc' in module 'a'. Another module 'b' imports
that function with a "from import":

a.py:

def somefunc():
return 'hey!'

b.py:

from a import somefunc

def someotherfunc():
return somefunc() + 'there'


I want to now test 'someotherfunc' while patching away 'somefunc':

def test_someotherfunc(monkeypatch):
from b import someotherfunc
monkeypatch.setattr('a.somefunc', lambda: 'eh?')
someotherfunc()

But this will fail, since 'b' imported somefunc from 'a' before we
monkey patched the module's attribute. Without a "from import", this
would work:

b.py:

import a

def someotherfunc():
return a.somefunc() + 'there'

What I would normally resort to is patch somefunc's func_code, so that
even code that used a "from import" before I could patch will use the
patched version.

Thoughts?


Cheers
Daniel

[holger krekel]

Hi Daniel,

Post by Daniel Nouri
Dear all
I have a function 'somefunc' in module 'a'. Another module 'b' imports
return 'hey!'
from a import somefunc
return somefunc() + 'there'
from b import someotherfunc
monkeypatch.setattr('a.somefunc', lambda: 'eh?')
someotherfunc()
But this will fail, since 'b' imported somefunc from 'a' before we
monkey patched the module's attribute. Without a "from import", this
import a
return a.somefunc() + 'there'
What I would normally resort to is patch somefunc's func_code, so that
even code that used a "from import" before I could patch will use the
patched version.

I would probably patch b's somefunc. Note that setattr() will by
default raise an exception if "b.somefunc" does not exist.

As to patching func_code: good idea, i had forgotten about assigning func_code.
In earlier Python2.X versions assigning to func_code didn't work but i just
checked that it does on py27 and py33. So it's definitely worthwhile to
pursue. In your example you could do:

monkeypatch.setattr('a.somefunc.func_code', (lambda: 'eh?').func_code)

and it works (needs to use __code__ on py3). I guess we could think about a

monkeypatch.setcode("a.somefunc", lambda: 'eh?')

helper. You could either pass in a function or a code object.
We could even think about allowing non-string targets:

monkeypatch.setcode(a.somefunc, lambda: 'eh?')

Or would "monkeypatch.setfunc" be a better name?

best,
holger

[Daniel Nouri]

Post by holger krekel
Hi Daniel,

Post by Daniel Nouri
Dear all
I have a function 'somefunc' in module 'a'. Another module 'b' imports
return 'hey!'
from a import somefunc
return somefunc() + 'there'
from b import someotherfunc
monkeypatch.setattr('a.somefunc', lambda: 'eh?')
someotherfunc()
But this will fail, since 'b' imported somefunc from 'a' before we
monkey patched the module's attribute. Without a "from import", this
import a
return a.somefunc() + 'there'
What I would normally resort to is patch somefunc's func_code, so that
even code that used a "from import" before I could patch will use the
patched version.

I would probably patch b's somefunc. Note that setattr() will by
default raise an exception if "b.somefunc" does not exist.

Good point. But this is where my example above actually differs from
where I was trying to do this; namely in a fixture. So ideally all code
paths that called somefunc would get the dummy. (I'm using this fixture
in a functional test.)

Post by holger krekel
As to patching func_code: good idea, i had forgotten about assigning func_code.
In earlier Python2.X versions assigning to func_code didn't work but i just
checked that it does on py27 and py33. So it's definitely worthwhile to
monkeypatch.setattr('a.somefunc.func_code', (lambda: 'eh?').func_code)
and it works (needs to use __code__ on py3).

Nice. I hadn't thought about using setattr in this way!

It's still a little unfortunate that monkeypatch.setattr works fine for
very similar cases (e.g. class methods); just not for this one. At the
same time, it's clearly following Python's own semantics and works just
like setting an attribute.

Post by holger krekel
I guess we could think about a
monkeypatch.setcode("a.somefunc", lambda: 'eh?')
helper. You could either pass in a function or a code object.
monkeypatch.setcode(a.somefunc, lambda: 'eh?')
Or would "monkeypatch.setfunc" be a better name?

Hmm, I'm not sure how obscure my case is. And I guess even with a
setcode in the API, people would still need to learn the hard way that
the case of patching a module's attributes is special. I would
certainly use setcode if it existed though. :-)


Daniel

[Floris Bruynooghe]

Post by holger krekel
As to patching func_code: good idea, i had forgotten about assigning func_code.
In earlier Python2.X versions assigning to func_code didn't work but i just
checked that it does on py27 and py33. So it's definitely worthwhile to
monkeypatch.setattr('a.somefunc.func_code', (lambda: 'eh?').func_code)
and it works (needs to use __code__ on py3). I guess we could think about a
monkeypatch.setcode("a.somefunc", lambda: 'eh?')
helper. You could either pass in a function or a code object.
monkeypatch.setcode(a.somefunc, lambda: 'eh?')
Or would "monkeypatch.setfunc" be a better name?

Why does it have to be a new method? Can't setattr simply see if the
object being patched as well as the object it is being patched with
has a .func_code or .__code__ attribute and automatically patch the
code if so? This would be additional to the current patching it does
so that it's more obvious in a debugger for the current cases. It
would have solved the OPs problem and I can't think of any real
downside currently.

Regards,
Floris

[holger krekel]

Post by Floris Bruynooghe

Post by holger krekel
As to patching func_code: good idea, i had forgotten about assigning func_code.
In earlier Python2.X versions assigning to func_code didn't work but i just
checked that it does on py27 and py33. So it's definitely worthwhile to
monkeypatch.setattr('a.somefunc.func_code', (lambda: 'eh?').func_code)
and it works (needs to use __code__ on py3). I guess we could think about a
monkeypatch.setcode("a.somefunc", lambda: 'eh?')
helper. You could either pass in a function or a code object.
monkeypatch.setcode(a.somefunc, lambda: 'eh?')
Or would "monkeypatch.setfunc" be a better name?

Why does it have to be a new method? Can't setattr simply see if the
object being patched as well as the object it is being patched with
has a .func_code or .__code__ attribute and automatically patch the
code if so? This would be additional to the current patching it does
so that it's more obvious in a debugger for the current cases. It
would have solved the OPs problem and I can't think of any real
downside currently.

Wouldn't it violate what you expect from a "setattr" operation?
Anyway, i played with setting __code__ the other day and there is
a bad limitation::

def test_hello(monkeypatch):
x = "/tmp"
def myabs(somepath):
return x

Post by Floris Bruynooghe
monkeypatch.setattr("os.path.abspath.__code__", myabs.__code__)

E ValueError: abspath() requires a code object with 0 free vars, not 1

This means that setting __code__ with closure-using code does not work.
This reduces the usefullness of setting code objects for practical
purposes and speaks against introducing any support at this point.
Setting non-closures might still be useful and the example above is
other compatible to py26++ so it's probably good enough.

best,
holger

[Carl Meyer]

Post by Daniel Nouri
from b import someotherfunc
monkeypatch.setattr('a.somefunc', lambda: 'eh?')
someotherfunc()

This doesn't look like correct syntax for monkeypatch.setattr. I think
monkeypatch.setattr(a, 'somefunc', lambda: 'eh?')

Never mind this! I checked in pytest 2.3.5 on the project I had handy,
didn't realize this syntax had been added in pytest 2.4. Nice!

Carl

[Carl Meyer]

Hi Daniel,

Post by Daniel Nouri
I have a function 'somefunc' in module 'a'. Another module 'b' imports
return 'hey!'
from a import somefunc
return somefunc() + 'there'
from b import someotherfunc
monkeypatch.setattr('a.somefunc', lambda: 'eh?')
someotherfunc()

This doesn't look like correct syntax for monkeypatch.setattr. I think
you mean:

monkeypatch.setattr(a, 'somefunc', lambda: 'eh?')

Post by Daniel Nouri
But this will fail, since 'b' imported somefunc from 'a' before we
monkey patched the module's attribute. Without a "from import", this
import a
return a.somefunc() + 'there'
What I would normally resort to is patch somefunc's func_code, so that
even code that used a "from import" before I could patch will use the
patched version.
Thoughts?

The simpler approach is to patch in the module where the function is
used, not the module where it is defined. This has the added advantage
of limiting the scope of your monkeypatching to only the module where
you actually need it. So monkeypatch b.somefunc instead of a.somefunc:

monkeypatch.setattr(b, 'somefunc', lambda: 'eh?')

The key point here is to think of monkeypatching as just "reassign some
name in some namespace", which makes it natural to reassign the name
within the namespace where your code-under-test is actually using the name.

Carl