Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Some tests are dependant on Python's hashing randomization #50

Closed
multani opened this issue Dec 5, 2014 · 0 comments
Closed

Some tests are dependant on Python's hashing randomization #50

multani opened this issue Dec 5, 2014 · 0 comments

Comments

@multani
Copy link
Member

multani commented Dec 5, 2014

The problem basically is:

gourami:~/.../nevow$ trial nevow.test.test_flatsax.Basic.test_otherns                
nevow.test.test_flatsax
  Basic
    test_otherns ...                                                       [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 0.017s

PASSED (successes=1)


gourami:~/.../nevow$ PYTHONHASHSEED=42 trial nevow.test.test_flatsax.Basic.test_otherns
nevow.test.test_flatsax
  Basic
    test_otherns ...                                                     [FAIL]

===============================================================================
[FAIL]
Traceback (most recent call last):
  File ".../nevow/nevow/test/test_flatsax.py", line 184, in test_otherns
    self.failUnlessEqual(xml, flatten(parseString(xml)))
  File "/usr/lib/python2.7/dist-packages/twisted/trial/_synctest.py", line 356, in assertEqual
    % (msg, pformat(first), pformat(second)))
twisted.trial.unittest.FailTest: not equal:
a = '<html xmlns="http://www.w3.org/1999/xhtml" xmlns:xf="http://www.w3.org/2002/xforms"><p>in default namespace</p><xf:input><xf:label>in another namespace</xf:label></xf:input></html>'
b = '<html xmlns:xf="http://www.w3.org/2002/xforms" xmlns="http://www.w3.org/1999/xhtml"><p>in default namespace</p><xf:input><xf:label>in another namespace</xf:label></xf:input></html>'


nevow.test.test_flatsax.Basic.test_otherns
-------------------------------------------------------------------------------
Ran 1 tests in 0.017s

FAILED (failures=1)

This means that some tests will break some because they depend on some ordering which is not stable . This currently blocks #47 since Tox sets a different hash seed for each run and the probability of getting the "right" seed is probably very low.

I would propose to hunt down the places where such ordering is not guaranteed and use sorted() or something appropriate to fix the problem. (The fix for this test case is too wrap this iteritems() into sorted()). This will make the output more predictable obviously, ease the tests as the tester can expect to have a reliable output, but it has a slight cost performance though.

Other possibilities that I can think of:

  • change the tests so they are smarter: instead of doing a basic string equality (or a assert expected in result as I saw in other failing tests where expected content is not in the same order as the one in result), parse the result (they are XML-like strings each times) and assert that the structure is properly formed.
  • find the right hash :) (and set it somewhere once and for all, like this for Tox.
multani added a commit to multani/nevow that referenced this issue Dec 5, 2014
@multani
Serialize tag attributes in a stable order
63f20a1 
`attributes.iteritems()` doesn't have a guaranteed order and depends on
the underlying implementation.
This caused some tests to fail under specific circumstances because they
were expecting a specific order which wasn't the one returned in this case.

This fix makes attributes serialization stable, and fix some tests which
were expecting a different order than the new, stable, one.

Fix twisted#50
@multani multani mentioned this issue Dec 5, 2014
Merged
@multani multani closed this as completed in f88939a Oct 6, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@multani