connection freezes in gym environment

[LaurensNeinders]

Hello,
When I'm in the gym environment for trackmania lidar, the incoming information doesn't seem to always update too well, there are random small periods of freezes where the position is not updated for a few iterations.
To demonstrate this I recorded the position of the car while drive the tmrl-test track.
Is there anything that can be done about this problem?

p.s. I saw in another thread [https://github.com//discussions/27] about "minor changes in the OpenPlanet script and trackmania environments", does this have to do with it?

connection.issues.position.drown.mp4

[LaurensNeinders]

oh nevermind, I just found the changes to tools.py. Those fixed the issues

[yannbouteiller]

Hi, this looks like Windows freezes, does it come with freezes of the game window and/or a warning about the time-step timing out for the corresponding number of seconds?

On my old laptop I get a similar issue since a TrackMania update a couple monthes ago: the game would sometimes freeze so badly that the OpenPlanet script times out and stops (which results in the gym environment throwing an error since tmrl 0.4.0. Before 0.4.0 the gym environment wouldn't know that the OpenPlanet script stopped and would happily return trash samples).

(With regard to the thread you link, the OpenPlanet script has not changed finally in fact. I have just changed the code of the environment such that it throws an error when the OpenPlanet script crashes due to one of these freezes)

It is worth noting that I seem to get the freezes I am talking about on my old laptop only (Windows 10), not my new (overpowered and Windows 11) laptop. Also I am not 100% sure this is related to the stutters you are describing?

[LaurensNeinders]

Thanks for the reply,
I don't suspects the freezes to be because of the hardware, my pc should be powerful enough (ryzen 3600, 3070).
I also found that, when logging the position in the plugin, the freezes do not seem to be there and the position is updated correctly.
So my guess would be that the connection between the openplanet plugin and the rtgym environment is not 100% stable all the time.
I use, among others, the position of the car for training and I feel like this issue could cause the algorithm to have a more difficult time, when these freezes occur.

[yannbouteiller]

Strange, the issue I get on my old laptop looks quite different.

So, if I understand correctly you modded the OpenPlanet script to log positions, i.e., you modded this script:

#name "TMRL grab data "
#author "tmrl team"
#category "TMRL"


bool send_data_float(Net::Socket@ sock, float val)
{
	if (!sock.Write(val)) {
		// If this fails, the socket might not be open. Something is wrong!
		print("INFO: Disconnected, could not send data.");
		return false;
	}
	return true;
}

void Main()
{
	while(true)
	{
		auto sock_serv = Net::Socket();
		if (!sock_serv.Listen("127.0.0.1", 9000)) {
			print("Could not initiate server socket.");
			return;
		}
		print(Time::Now + "Waiting for incomming connection...");

		while(!sock_serv.CanRead()){
			yield();
		}
		print("Socket can read");
		auto sock = sock_serv.Accept();

		print(Time::Now + "Accepted incomming connection.");

		while (!sock.CanWrite()) {
			yield();
		}
		print("Socket can write");
		print(Time::Now + " Connected!");

		bool cc = true;
		while(cc)
		{
			CTrackMania@ app = cast<CTrackMania>(GetApp());
			CSmArenaClient@ playground = cast<CSmArenaClient>(app.CurrentPlayground);
			CSmArena@ arena = cast<CSmArena>(playground.Arena);
			auto player = arena.Players[0];
			CSmScriptPlayer@ api = cast<CSmScriptPlayer>(player.ScriptAPI);

			auto race_state = playground.GameTerminals[0].UISequence_Current;
			
			// Sending data
			cc = send_data_float(sock, api.Speed);
			send_data_float(sock, api.Distance);
			send_data_float(sock, api.Position.x);
			send_data_float(sock, api.Position.y);
			send_data_float(sock, api.Position.z);
			send_data_float(sock, api.InputSteer);
			send_data_float(sock, api.InputGasPedal);
			if(api.InputIsBraking) send_data_float(sock, 1.0f);
			else send_data_float(sock, 0.0f);
			// old code: if(race_state == ESGamePlaygroundUIConfig__EUISequence::Finish) send_data_float(sock, 1.0f);
			// can use CGamePlaygroundUIConfig::EUISequence::Finish or CGameTerminal::ESGamePlaygroundUIConfig__EUISequence::Finish
			if(race_state == SGamePlaygroundUIConfig::EUISequence::Finish) send_data_float(sock, 1.0f);
			else send_data_float(sock, 0.0f);
			send_data_float(sock, api.EngineCurGear);
			send_data_float(sock, api.EngineRpm);
			
			// send_data_float(sock,race.Finished);

			yield();  // this statement stops the script until the next frame
		}
		sock.Close();
		sock_serv.Close();
	}
}

All this script is doing is grabbing the info from TrackMania such as api.Position.x, api.Position.y, api.Position.z (that you logged without seeing these freezes, if I understand correctly) and send them to a local socket via localhost (127.0.0.1) and port 9000.

This information is then retrieved by the rtgym environment by this python thread in the TM2020OpenPlanetClient, whose current code is:

class TM2020OpenPlanetClient:
    def __init__(self, host='127.0.0.1', port=9000, struct_str='<' + 'f' * 11):
        self._struct_str = struct_str
        self.nb_floats = self._struct_str.count('f')
        self.nb_uint64 = self._struct_str.count('Q')
        self._nb_bytes = self.nb_floats * 4 + self.nb_uint64 * 8

        self._host = host
        self._port = port

        # Threading attributes:
        self.__lock = Lock()
        self.__data = None
        self.__t_client = Thread(target=self.__client_thread, args=(), kwargs={}, daemon=True)
        self.__t_client.start()

    def __client_thread(self):
        """
        Thread of the client.
        This listens for incoming data until the object is destroyed
        TODO: handle disconnection
        """
        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
            s.connect((self._host, self._port))
            data_raw = b''
            while True:  # main loop
                while len(data_raw) < self._nb_bytes:
                    data_raw += s.recv(1024)
                div = len(data_raw) // self._nb_bytes
                data_used = data_raw[(div - 1) * self._nb_bytes:div * self._nb_bytes]
                data_raw = data_raw[div * self._nb_bytes:]
                self.__lock.acquire()
                self.__data = data_used
                self.__lock.release()

    def retrieve_data(self, sleep_if_empty=0.1, timeout=10.0):
        """
        Retrieves the most recently received data
        Use this function to retrieve the most recently received data
        If block if nothing has been received so far
        """
        c = True
        t_start = None
        while c:
            self.__lock.acquire()
            if self.__data is not None:
                data = struct.unpack(self._struct_str, self.__data)
                c = False
                self.__data = None
            self.__lock.release()
            if c:
                if t_start is None:
                    t_start = time.time()
                t_now = time.time()
                assert t_now - t_start < timeout, f"OpenPlanet stopped sending data since more than {timeout}s."
                time.sleep(sleep_if_empty)
        return data

This thread creates a TCP socket connected to the aforementioned 127.0.0.1:9000 and continuously reads whatever the OpenPlanet script sends. This loop:

while len(data_raw) < self._nb_bytes:
    data_raw += s.recv(1024)

is supposed to exit whenever a full packet sent by the OpenPlanet script is received, then the thread truncates the said packet from its local buffer:

div = len(data_raw) // self._nb_bytes
data_used = data_raw[(div - 1) * self._nb_bytes:div * self._nb_bytes]
data_raw = data_raw[div * self._nb_bytes:]

waits for a lock:

self.__lock.acquire()

and finally puts the packet in self.__data and releases the lock:

self.__data = data_used
self.__lock.release()

The only issue that I can possibly see would either be some unlikely issue with localhost (127.0.0.1:9000) communication on your PC, or this lock randomly taking a long time to be freed, which should not happen because this lock is only acquired otherwise by retrieve_data, which releases it and sleeps for 0.1 second before retrying when self.__data is empty.

In my case, it is the game itself that freezes occasionally, resulting in the OpenPlanet script crashing (there is a timeout of 1s by default), which in turns results in an exception thrown by:

assert t_now - t_start < timeout, f"OpenPlanet stopped sending data since more than {timeout}s."

after 10 seconds.

[yannbouteiller]

Or maybe for some reason the GIL randomly decides not to run this thread for a long time?

We could get rid of it and read from the socket directly in retrieve_data, but then the TCP buffer would fill up when retrieve_data is not called.

[LaurensNeinders]

I found that /tmrl/custom/utils/tools.py in my branch was still the version from apr,16 2022. After updating to the latest version, the issue with the stutters went away. It seems like the bug has already disppeared.

A possible explanation for the freezes could be this piece of code in retrieve_data()
a part of the code changed from :

if self.__data is not None:
                data = struct.unpack(self._struct_str, self.__data)
                c = False

to this:

if self.__data is not None:
                data = struct.unpack(self._struct_str, self.__data)
                c = False
                self.__data = None

The change resets self.__data to None, which the older code didn't.
Without this reset it might have been possible for the retrieve_data() method to send the same data multiple times in a row.

[yannbouteiller]

Oh yes indeed, I thought you were using the newer version