"Your request has been blocked due to a network policy."

[FloydW]

Hello Mr. Kane,

I have grabbit installed on OSX through homebrew and it works great.
I wanted to tweak the code a little to allow it to download files where the extension is in caps (".JPG"). But after compile, grabbit returns error messages that Reddit is blocking the transactions "due to a network policy."
Steps to Reproduce:
In a Terminal window:

mkdir grabbit
cd grabbit
git init
git clone https://github.com/bbkane/grabbit.git
cd grabbit
mkdir tmp
mkdir tmp/grabbit
go run . grab -c test-config.yml
go run . grab -c test-config.yml >output.txt 2>&1

Please provide guidance.
output.txt
Error messages can be seen in the attached file "output.txt".

[bbkane]

Hi @FloydW ! I was able to reproduce this locally, but I'm also not sure why Reddit doesn't like the go run version but is ok with the homebrew version.

I thought it might be a change since the last released version so I checked out the repo at the v4.4.19 tag, but that also gave the error.

So maybe it's a Go runtime version update?

Maybe the next step is to try different older versions of Go to build and run it.

If that doesn't work, maybe taking a packet capture of both versions to see how the HTTPS requests differ.

To be honest, this might take some real time to investigate and fix and I'm not sure when I'm going to have that time. I'll leave this issue open until it gets fixed.

If you have the time, I'd love any help you can offer (see next steps above). If you find anything, comment on this issue.

[bbkane]

Ok! grabbit v4.4.19 (the homebrew version) was built with Go 1.20.5.

https://go.dev/doc/manage-install let me install that and then the download worked.

Still have to figure out what changed between go1.20.5 and the current version (go1.22.0), but we know more than we did!

[bbkane]

https://www.benburwell.com/posts/intercepting-golang-tls-with-wireshark/

[FloydW]

I used Proxyman to catch the network traffic in the command line. The crazy thing is that once I turned on the Proxyman proxy to capture the HTTP responses everything worked. I will attach the HTTP transactions with and without the proxy. Without, I get 301 redirect responses.
Think this could be a timing issue?

grabbit-no-proxy-03-06-2024-12-35-43.csv
grabbit-proxy-03-06-2024-12-34-09.csv

[bbkane]

Maybe proxyman forwarding the request is changing something?

[JosiahBradley]

I've root caused this and built a working version of Grabbit on go1.21.0 and go1.22.1 for win11 amd64:

HTTP2 was force disabled and downgraded to HTTP1

This patch fixes it

index 85863e2..d116057 100644
--- a/grab.go
+++ b/grab.go
@@ -169,28 +169,17 @@ func getTopPosts(ctx context.Context, timeout time.Duration, logger *logos.Logge
                ReadBufferSize:         0,
                Proxy:                  http.ProxyFromEnvironment,
                OnProxyConnectResponse: nil,
-               DialContext: (&net.Dialer{
-                       Timeout:        30 * time.Second,
-                       KeepAlive:      30 * time.Second,
-                       Deadline:       time.Time{},
-                       LocalAddr:      nil,
-                       DualStack:      false,
-                       FallbackDelay:  0,
-                       Resolver:       nil,
-                       Cancel:         nil,
-                       Control:        nil,
-                       ControlContext: nil,
-               }).DialContext,
+               DialContext:            nil,

                // change from default
-               ForceAttemptHTTP2: false,
+               ForceAttemptHTTP2: true,

                MaxIdleConns:        100,
                IdleConnTimeout:     90 * time.Second,
                TLSHandshakeTimeout: 10 * time.Second,

                // use an empty map instead of nil per the link above
-               TLSNextProto: make(map[string]func(authority string, c *tls.Conn) http.RoundTripper),
+               TLSNextProto: nil,

                ExpectContinueTimeout: 1 * time.Second,
        }

Reference from redditdev
https://www.reddit.com/r/redditdev/comments/uncu00/comment/i8gyfmx/?utm_source=share&utm_medium=web2x&context=3

[bbkane]

I'll try to merge this tonight!

[bbkane]

@FloydW , I've also just pushed a commit to that does a case-insensitive check against the allowed suffixes. I'll release a new version of grabbit now you can download.

[bbkane]

Oh, and thanks @JosiahBradley , how did you solve this? GODEBUG=??? go run . ?