Mettle segfaults attempting to write to a closed channel upstream

[acammack-r7]

While reproducing rapid7/metasploit-framework#12888 on Linux, I also discovered that at least with the HTTP transport, it is easy to catch mettle in a bad state when it does not recognize that the connection on the upstream of a network connection is closed, causing it to attempt a socket write and it ends up segfaulting. This can be reproduced by trying to send data across the connection in step 4 of rapid7/metasploit-framework#12888.

It appears that this is a possibility across all connection behaviors as a race condition, but it is especially bad when the client sends some data and closes the connection right away (the server on the other end closed the connection right away, but ~30s later the channel was not marked closed):

[01-28-2020 16:32:56.782s] [tlv.c:566] processing method: 'core_channel_open' id: '31324158387953557263351596830481'
[01-28-2020 16:32:56.782s] [network_client.c:467] resolving 'tcp://127.0.0.1:8333'
[01-28-2020 16:32:56.782s] [network_client.c:345] connecting to tcp://127.0.0.1:8333
[01-28-2020 16:32:56.782s] [network_client.c:278] connected to 'tcp://127.0.0.1:8333'
[01-28-2020 16:32:59.696s] [mettle.c:75] Heartbeat
[01-28-2020 16:33:04.696s] [mettle.c:75] Heartbeat
[01-28-2020 16:33:09.697s] [mettle.c:75] Heartbeat
[01-28-2020 16:33:14.696s] [mettle.c:75] Heartbeat
[01-28-2020 16:33:19.695s] [mettle.c:75] Heartbeat
[01-28-2020 16:33:23.799s] [tlv.c:566] processing method: 'core_channel_write' id: '82929997445586913188508952014690'
Segmentation fault

In a bi-directional case:

Server:

$  socat -v TCP-LISTEN:8333,reuseaddr SYSTEM:"echo hello && sleep 5 && echo there && sleep 1"
< 2020/01/28 17:17:02.075830  length=6 from=0 to=5
hello
< 2020/01/28 17:17:07.077770  length=6 from=6 to=11
there

Client:

$  socat -v SOCKS4A:localhost:127.0.0.1:8333 -
> 2020/01/28 17:17:07.374887  length=6 from=0 to=5
there
there
a;slkdfja
< 2020/01/28 17:17:08.520644  length=10 from=0 to=9
a;slkdfja
af
< 2020/01/28 17:17:09.104787  length=3 from=10 to=12
af

Mettle:

[01-28-2020 17:17:02.072s] [tlv.c:566] processing method: 'core_channel_open' id: '15664141260921926190853307761506'
[01-28-2020 17:17:02.073s] [network_client.c:467] resolving 'tcp://127.0.0.1:8333'
[01-28-2020 17:17:02.073s] [network_client.c:345] connecting to tcp://127.0.0.1:8333
[01-28-2020 17:17:02.073s] [network_client.c:278] connected to 'tcp://127.0.0.1:8333'
[01-28-2020 17:17:03.386s] [mettle.c:75] Heartbeat
[01-28-2020 17:17:08.386s] [mettle.c:75] Heartbeat
[01-28-2020 17:17:09.477s] [tlv.c:566] processing method: 'core_channel_write' id: '14731815549335732031172341115657'
Segmentation fault