Functionality questions

[BFergerson]

Thanks for making this project. A non-interactive python debugger is exactly what I'm looking for. I'd like to use your project to support the Python probe for Source++. I've been able to successfully implement some portions of the functionality, but I'm having some trouble understanding some Python debugging relating things. I think some of this trouble might be alleviated with new functionality for nopdb and I'm more than willing to implement that functionality. My background is with JVM languages so I'm just looking for a bit of guidance on how things work the Python way.

Breakpoint doesn't hit on Flask app

I don't think this issue is a nopdb issue as I'm unable to hit the breakpoint using import pdb; pdb.set_trace() in the main method either. I believe this issue is because of either compiled code within Flask or pdb is not hitting because of separate threads. Either way, I'm able to set breakpoints via PyCharm so it appears to be possible in some way. Hopefully, it's just a threading thing and breakpoints can be placed for every thread. Do you know how this functionality could be implemented with nopdb?

Here is some code that shows a breakpoint that's never hit:

import os

from flask import Flask
from nopdb import nopdb

app = Flask(__name__)


@app.route('/')
def hello():
    provider = str(os.environ.get('PROVIDER', 'world'))
    return 'Hello ' + provider + '!'


if __name__ == '__main__':
    with nopdb.breakpoint(file="FlaskTest.py", line=12) as bp:
        bp.exec("print(\"provider: \" + provider)")
        port = int(os.environ.get('PORT', 5000))
        app.run(host='0.0.0.0', port=port)

Ability to add breakpoints without calling code via context manager

This might be related to the issue above, but how do you add breakpoints to code that you don't want to call via the context manager? As in, code that is already running or that will run by itself sometime in the future? I have a small example here where I want to set a breakpoint before a function is called in the future via a delay. Can nopdb address this type of scenario?

Code which shows a breakpoint that's never hit:

import threading

from nopdb import nopdb


def fun():
    x = 5
    y = x


if __name__ == '__main__':
    start_time = threading.Timer(10, fun)
    start_time.start()

   # I'd like to set the breakpoint ahead of time and just output x once it's hit
    with nopdb.breakpoint(file="SeparateThread.py", line=8) as bp:
        bp.exec("print(\"x: \" + str(x))")

Does cpdb make sense to extend?

This might be a bit off-topic, but while looking into Python debuggers I ran across https://pypi.org/project/CPdb/. Does this project make sense to extend for nopdb? The reason I ask is because of the claim cpdb makes about "long-running" applications running better with it. I would like to use nopdb in "long-running" applications so I'm curious if it makes sense to integrate the technologies.

[cifkao]

Thanks for your questions!

Let me answer the second question first, which is actually about two things: using NoPdb without a context manager, and threads. Yes, you can use NoPdb without a context manager: you just need to call the start() method of a NoPdb object, e.g.:

import nopdb


def fun():
    x = 5
    y = x


if __name__ == '__main__':
    dbg = nopdb.get_nopdb()  # Get the default NoPdb object for this thread
    dbg.breakpoint(file="myscript.py", line=6).exec("print(\"x: \" + str(x))")  # nopdb.breakpoint() would also work here
    dbg.start()  # Start debugging

    fun()

    dbg.stop()

About threads: A debugger is normally attached to only one thread (the one that started it), so that explains why your breakpoint is not getting hit. I don't know if it's even possible to attach it to a different thread that is already running. What NoPdb does is call sys.settrace(), which only affects the thread from which it is called. There is also threading.settrace(), which will cause the debugger to be automatically attached to every new thread. So you could theoretically do something like threading.settrace(sys.gettrace()) after starting NoPdb in the current thread and before starting any other threads. However, NoPdb is not thread-safe (that's why there is one NoPdb object per thread, and not a global one), so this will probably break something.

So while multi-thread support should be possible to implement (and I agree it would be useful), I think it would be complicated because of thread safety (maybe not that much for breakpoints, but definitely for the capture_call functionality) and I don't have a clear idea at the moment how I would do it. When I was designing NoPdb, I was mostly modeling it after Pdb, which AFAIK also doesn't work across threads.

I'm not sure about the Flask question, but I guess threading will be one issue.

Finally, regarding CPdb, I haven't heard about it. Which of its functionalities do you think it could be interesting to have here?

[BFergerson]

Thanks for the comprehensive answer. Using the start()/stop() and threading.settrace(sys.gettrace()) approach solves all the issues I've raised. I think the only thing left I could ask for would be the ability to remove individual breakpoints. It looks like with the start()/stop() approach I can't remove one breakpoint at a time. Is this the functionality you mention with A way to disable breakpoints.?

As far as threading.settrace(sys.gettrace()) goes, how difficult would you imagine it is for nopdb to support multithreading? I think this could be an area where I could contribute.

And for cpdb, it says:

Pdb basically hooks on sys.settrace to analyze each line being executed, running pdb on a large app, significantly slows it down. cpdb has a c-api extension method to use as the sys.settrace method which filters out lines on which there are no breakpoints set and calls the pdb dispatch only if there is a break point on the line.

The logic behind what's being said there makes sense to me. Do you have any experience running pdb-based tools on long-running/large apps? I'm not very knowledgeable on this so I'm not in a position to refute whether or not the C-API extension is necessary.

[cifkao]

As far as threading.settrace(sys.gettrace()) goes, how difficult would you imagine it is for nopdb to support multithreading? I think this could be an area where I could contribute.

I guess having a completely thread-safe NoPdb class would basically mean adding locks to all the methods, which could slow things down. Maybe there could be a thread-safe subclass ConcurrentNoPdb for this purpose.

The logic behind what's being said there makes sense to me. Do you have any experience running pdb-based tools on long-running/large apps? I'm not very knowledgeable on this so I'm not in a position to refute whether or not the C-API extension is necessary.

I don't really know about long-running apps, I'm mostly using it for quick analyses... I haven't done any performance benchmark, but from reading Pdb's code, it seems like it's tracing every function line by line, while NoPdb only does this with functions that have a breakpoint or capture, so it should already be much faster.