enhance del_mapinfo/change_mapinfo

[HyperSine]

Checklist

Which kind of PR do you create?

• This PR only contains minor fixes.
• This PR contains major feature update.
• This PR introduces a new function/api for Qiling Framework.

Coding convention?

• The new code conforms to Qiling Framework naming convention.
• The imports are arranged properly.
• Essential comments are added.
• The reference of the new code is pointed out.

Extra tests?

• No extra tests are needed for this PR.
• I have added enough tests for this PR.
• Tests will be added after some discussion and review.

Changelog?

• This PR doesn't need to update Changelog.
• Changelog will be updated after some proper review.
• Changelog has been updated in my PR.

Target branch?

• The target branch is dev branch.

One last thing

• I have read the contribution guide

---

I was trying to emulate a ELF packed by virbox protector recently. I found there were some error logs when handling syscall_mprotect. Because I cannot post the ELF file on github, here's the minimum code to reproduce:

#!/usr/bin/env python3
import qiling
import qiling.os.posix.syscall

ql = qiling.Qiling([ '/bin/ls' ], rootfs = '/')

ql.mem.map(0xdead0000, 0x4000, 3, 'manual')
ql.mem.map(0xdead4000, 0x4000, 7, 'manual')
ql.mem.map(0xdead8000, 0x4000, 1, 'manual')

print('before:')
print('\n'.join(filter(lambda s: 'manual' in s, ql.mem.get_formatted_mapinfo())))

qiling.os.posix.syscall.ql_syscall_mprotect(ql, 0xdead2000, 0x4000, 0x1)

print('after:')
print('\n'.join(filter(lambda s: 'manual' in s, ql.mem.get_formatted_mapinfo())))

Output:

$ ./code.py 
before:
0000000000dead0000 - 0000000000dead4000   rw-     manual                 
0000000000dead4000 - 0000000000dead8000   rwx     manual                 
0000000000dead8000 - 0000000000deadc000   r--     manual                 
[x]     Cannot change mapinfo at 0xdead2000-0xdead6000
after:
0000000000dead0000 - 0000000000dead4000   rw-     manual                 
0000000000dead4000 - 0000000000dead8000   rwx     manual                 
0000000000dead8000 - 0000000000deadc000   r--     manual

After some investigation, I found that the function change_mapinfo() in qiling/os/memory.py could only change only one MapInfoEntry at a time and that MapInfoEntry must be fully contained by memory range [mem_s, mem_e). If multiple MapInfoEntry just overlap but all of them are not fully contained by the memory range, such error would appear. So I did some enhancement to change_mapinfo and other functions. Here is what this PR contains:

1. Add a function find_mapinfo() as it would be used in both del_mapinfo() and change_mapinfo().
2. Use find_mapinfo() to calculate overlap_ranges in del_mapinfo(). It has less iterations so we can have better performance.
3. Use direct insert when add new entries in the end of del_mapinfo(). The new entries to be added are all parts of entries removed before. So just insert new entries at index from i0, no need to call bisect.insort.
4. Rewrite change_mapinfo(). Now it could change multiple overlapping entries' permissions/label at a time.
5. Swap the order of mem_unmap() and del_mapinfo() in unmap() function for better exception safety.

After applying this PR, the code above wound have correct output:

$ ./code.py
before
0000000000dead0000 - 0000000000dead4000   rw-     manual                 
0000000000dead4000 - 0000000000dead8000   rwx     manual                 
0000000000dead8000 - 0000000000deadc000   r--     manual                 
after:
0000000000dead0000 - 0000000000dead2000   rw-     manual                 
0000000000dead2000 - 0000000000dead6000   r--     manual                 
0000000000dead6000 - 0000000000dead8000   rwx     manual                 
0000000000dead8000 - 0000000000deadc000   r--     manual   

[xwings]

Cool, that make sense.

[xwings]

Hi,

The code seems to break the test. I need to revert it. Will you be able to make another PR ?

[HyperSine]

Hi,

The code seems to break the test. I need to revert it. Will you be able to make another PR ?

Can you tell me which test is broken?

[xwings]

https://github.com/qilingframework/qiling/actions/runs/7208092719/job/19636291311

======================================================================
ERROR: test_qdb_mips32el_hello (__main__.DebuggerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_qdb.py", line 19, in test_qdb_mips32el_hello
    ql.run()
  File "/home/runner/work/qiling/qiling/tests/../qiling/core.py", line 590, in run
    debugger = debugger(self)
  File "/home/runner/work/qiling/qiling/tests/../qiling/debugger/qdb/qdb.py", line 49, in __init__
    self.dbg_hook(list(filter(lambda d: int(d, 0) != self.ql.loader.entry_point, init_hook)))
  File "/home/runner/work/qiling/qiling/tests/../qiling/debugger/qdb/qdb.py", line 120, in dbg_hook
    run_qdb_script(self, self._script)
  File "/home/runner/work/qiling/qiling/tests/../qiling/debugger/qdb/utils.py", line 159, in run_qdb_script
    func()
  File "/home/runner/work/qiling/qiling/tests/../qiling/debugger/qdb/utils.py", line 273, in magic
    p_st = self.rr._save()
  File "/home/runner/work/qiling/qiling/tests/../qiling/debugger/qdb/utils.py", line 219, in _save
    return self.State(self.ql.save())
  File "/home/runner/work/qiling/qiling/tests/../qiling/core.py", line 641, in save
    saved_states["mem"] = self.mem.save()
  File "/home/runner/work/qiling/qiling/tests/../qiling/os/memory.py", line 366, in save
    data = self.read(lbound, ubound - lbound)
  File "/home/runner/work/qiling/qiling/tests/../qiling/os/memory.py", line 403, in read
    return self.ql.uc.mem_read(addr, size)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/unicorn/unicorn.py", line 579, in mem_read
    raise UcError(status)
unicorn.unicorn.UcError: Invalid memory read (UC_ERR_READ_UNMAPPED)

[HyperSine]

I think the problem is not from my code. It is due to the bug from upstream: unicorn-engine/unicorn#1877.

In test_qdb_mips32el_hello, ql would map memory above 0x90000000 because of ql.loader.mmap_address. But it would call ql.mem.unmap_all() later which relies on ql.uc.mem_regions(). The function mem_regions() doesn't return a correct result and causes the test failed. Specifically,

1. Some method in ql calls ql.mem.map(0x90000000, ...), but unicorn engine treats it as memory starting from 0x10000000 because of the bug I mentioned above. Our ql.mem.map_info will have the record starting from 0x90000000.
2. Later, some method in ql calls ql.mem.unmap_all(), which retrieves memory regions from unicorn. So there will be a memory region starting from 0x10000000, and ql.mem.del_mapinfo(0x10000000, ...) will be called. Our ql.mem.map_info doesn't have such record, so del_mapinfo does nothing. But unicorn will still unmap such memory region.
3. Later, some method in ql calls ql.mem.save(). It iterates ql.mem.map_info and tries to read the undeleted memory record starting from 0x90000000 which, in unicorn's perspective, is 0x10000000, and of course, is already unmapped. So a UcError is raised.

If you modify the test code tests/test_qdb.py:

# from line 13
    def test_qdb_mips32el_hello(self):
        rootfs = "../examples/rootfs/mips32el_linux"
        path = rootfs + "/bin/mips32el_hello"

        ql = Qiling([path], rootfs)
        ql.loader.mmap_address = 0x20000000     # <-- add just this line
        ql.debugger = "qdb::rr:qdb_scripts/mips32el.qdb"
        ql.run()
        del ql

you will find my PR passes the test.