Cloudformation 1.1.7 : missing hubs folder in prod-assets bucket

[Decode1415]

Hello,

I got a curious issue. I got this issue with the modified version of the template 1.1.5 (when AWS changed the way to acces buckets), with 1.1.6 and now with 1.1.7 : my prod-assets buckets is missing the hubs folder. And all the time, my stack is building correctly.

If I look at /var/log/cloud-init-output.log, I see that everything is working fine until it gets to a control app named Supervisor :

Supervisor starting mozillareality/ita. See the Supervisor output for more details.
Waiting for ita.
Supervisor starting mozillareality/polycosm-static-assets. See the Supervisor output for more details.
Waiting for polycosm assets.
Waiting for certs.
Supervisor starting mozillareality/certbot. See the Supervisor output for more details.
mv: cannot stat '/storage/aws-backup-lost+found*': No such file or directory
mv: cannot stat '/storage/aws-backup-restore*/*': No such file or directory
Supervisor starting mozillareality/pgbouncer. See the Supervisor output for more details.
Waiting for pgBouncer.
Waking up the database.
 ?column?
----------
        1
(1 row)

Starting app clients.
Supervisor starting mozillareality/hubs. See the Supervisor output for more details.
Waiting for Hubs.
Supervisor starting mozillareality/spoke. See the Supervisor output for more details.
Waiting for Spoke.
Starting stream services.
Supervisor starting mozillareality/janus-gateway. See the Supervisor output for more details.
Waiting for Janus.
Starting app services.
Supervisor starting mozillareality/reticulum. See the Supervisor output for more details.
Waiting for Reticulum.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    21  100    21    0     0    396      0 --:--:-- --:--:-- --:--:--   388
Internal server errorSupervisor starting mozillareality/postgrest. See the Supervisor output for more details.
Waiting for PostgREST.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 3000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 3000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 3000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 3000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 3000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 3000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 3000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 3000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 3000: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    64    0    64    0     0     16      0 --:--:--  0:00:03 --:--:--    16
{"message":"Database connection lost, retrying the connection."}Supervisor starting mozillareality/youtube-dl-api-server. See the Supervisor output for more details.
Waiting for YT-DL.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 8080: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 8080: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 8080: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 8080: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    58  100    58    0     0      8      0  0:00:07  0:00:06  0:00:01    13

I also notice that Supervisor, in itself, was not installed. And I was able to install it manualy with apt-get ... Does someone knows where is this cloud-init script? So I can try to make it works or at least try to see why it doesn't complete properly? Any advise will be very appreciated. Thank you!

[Decode1415]

If I execute manually /opt/polycosm/polycosm_boot.sh aws aws, I get :

[...]

Waiting for bio.
Loading app services.
Waiting for ita.
Waiting for polycosm assets.
Waiting for certs.
mv: cannot stat '/storage/aws-backup-lost+found*': No such file or directory
mv: cannot stat '/storage/aws-backup-restore*/*': No such file or directory
Waiting for pgBouncer.
Waking up the database.
 ?column?
----------
        1
(1 row)

Starting app clients.
Waiting for Hubs.
Waiting for Spoke.
Starting stream services.
Waiting for Janus.
Starting app services.
Waiting for Reticulum.
Internal server errorWaiting for PostgREST.
{"swagger":"2.0","info":{"version":"5.2.0 (1e732ac)","title":"PostgREST API","description":"This is a dynamic API generated by PostgREST"},"host":"127.0.0.1:3000","basePath":"/","schemes":["http"],"consumes":["application/json","application/vnd.pgrst.object+json","text/csv"],"produces":["application/json","application/vnd.pgrst.object+json","text/csv"],"paths":{"/":{"get":{"tags":["Introspection"],"summary":"OpenAPI description (this document)","produces":["application/openapi+json","application/json"],"responses":{"200":{"description":"OK"}}}},"/rpc/create_or_replace_admin_view":{"post":{"tags":["(rpc) create_or_replace_admin_view"],"produces":["application/json","application/vnd.pgrst.object+json"],"parameters":[{"required":true,"schema":{"required":["name"],"type":"object","properties":{"name":{"format":"text","type":"string"},"extra_columns":{"format":"text","type":"string"},"extra_clauses":{"format":"text","type":"string"}}},"in":"body","name":"args"},{"$ref":"#/parameters/preferParams"}],"responses":{"200":{"description":"OK"}}}}},"parameters":{"preferParams":{"name":"Prefer","description":"Preference","required":false,"in":"header","type":"string","enum":["params=single-object"]},"preferReturn":{"name":"Prefer","description":"Preference","required":false,"in":"header","type":"string","enum":["return=representation","return=minimal","return=none"]},"preferCount":{"name":"Prefer","description":"Preference","required":false,"in":"header","type":"string","enum":["count=none"]},"select":{"name":"select","description":"Filtering Columns","required":false,"in":"query","type":"string"},"order":{"name":"order","description":"Ordering","required":false,"in":"query","type":"string"},"range":{"name":"Range","description":"Limiting and Pagination","required":false,"in":"header","type":"string"},"rangeUnit":{"name":"Range-Unit","description":"Limiting and Pagination","required":false,"default":"items","in":"header","type":"string"},"offset":{"name":"offset","description":"Limiting and Pagination","required":false,"in":"query","type":"string"},"limit":{"name":"limit","description":"Limiting and Pagination","required":false,"in":"query","type":"string"}},"externalDocs":{"url":"https://postgrest.org/en/v2/api.html","description":"PostgREST Documentation"}}Waiting for YT-DL.
{"youtube-dl":"2021.12.17","youtube-dl-api-server":"0.4"}
Starting TURN services.
Waiting for Coturn.

[pattersonbl2]

An issue with PostgRest is falling with is vital for the app to build. Let me test on my end and see what I get.

[pattersonbl2]

Run the sudo bio sup status command. I want to see what packages you are running.

[Decode1415]

@pattersonbl2 :

# sudo bio sup status
package                                                     type        desired  state  elapsed (s)  pid     group
mozillareality/pgbouncer/1.12.0/20191106234457              standalone  up       up     71610        20283   pgbouncer.default@mozhubs
mozillareality/postgrest/5.2.0/20190130035107               standalone  up       up     71608        20475   postgrest.default@mozhubs
mozillareality/certbot/1.0.0/20191224043510                 standalone  up       up     71616        20198   certbot.default@mozhubs
mozillareality/hubs-discord-bot/0.0.1/20200215023448        standalone  up       down   21           <none>  hubs-discord-bot.default@mozhubs
mozillareality/janus-gateway/2.0.1/20211220195030           standalone  up       up     71610        20358   janus-gateway.default@mozhubs
mozillareality/youtube-dl-api-server/0.5/20230303041110     standalone  up       up     71654        18175   youtube-dl-api-server.default@mozhubs
mozillareality/reticulum/1.0.1/20230109224638               standalone  up       up     71596        20635   reticulum.default@mozhubs
mozillareality/coturn/4.5.3.0/20200422002445                standalone  up       up     13198        25327   coturn.default@mozhubs
mozillareality/hubs/1.0.0/20230109221430                    standalone  up       up     71610        20258   hubs.default@mozhubs
mozillareality/ita/0.0.1/20211117212826                     standalone  up       up     71653        18243   ita.default@mozhubs
mozillareality/polycosm-static-assets/1.0.0/20200320162152  standalone  up       up     71609        20427   polycosm-static-assets.default@mozhubs
mozillareality/spoke/1.0.0/20220405230707                   standalone  up       up     71603        20517   spoke.default@mozhubs

[Decode1415]

@pattersonbl2

By any chance, would you have an advice about what I should or could try? Thank you!

[Decode1415]

(still working on it, I did found, I think, what was wrong in my setup - no need to investigate now, the issue is probably on my side, will keep this ticket updated)

[Decode1415]

There is clearly an issue with the hosted zone. CloudFormation does not remove created NS in it when deleting the MozillaHub stack.

I did buy two new domains, just to start it over and have fresh data. After 3 stacks creation/deleting (1 failed complettly, 1 failed partially), I still got 27 records in hosted zone for the internal domain. So there is an extra 25 that should not be there if we subtract the two that are created automatically by AWS at the creation of the hosted zone itself.

And if I look at the hosted zone of the shortlink domain, I got one extra cname record.

(testing/debugging continue ...)

[Decode1415]

I confirm the issue with the hosted zone with CloudFormation, I had manually delete the entries in order to be able to build my stack again.

[Decode1415]

Apparently, cleaning the DNS did not changed a thing, and after all, I still got the very same issue :

Missing file admin.html. Please try again. on my admin page, and my hubs folder is still missing.

sudo bio sup status

package type desired state elapsed (s) pid group
mozillareality/polycosm-static-assets/1.0.0/20200320162152 standalone up up 702 2928 polycosm-static-assets.default@mozhubs
mozillareality/hubs/1.0.0/20230109221430 standalone up up 645 3130 hubs.default@mozhubs
mozillareality/spoke/1.0.0/20220405230707 standalone up up 623 3280 spoke.default@mozhubs
mozillareality/ita/0.0.1/20211117212826 standalone up up 784 2765 ita.default@mozhubs
mozillareality/reticulum/1.0.1/20230109224638 standalone up up 566 3532 reticulum.default@mozhubs
mozillareality/postgrest/5.2.0/20190130035107 standalone up up 539 3801 postgrest.default@mozhubs
mozillareality/certbot/1.0.0/20191224043510 standalone up up 696 2957 certbot.default@mozhubs
mozillareality/coturn/4.5.3.0/20200422002445 standalone up up 518 3897 coturn.default@mozhubs
mozillareality/pgbouncer/1.12.0/20191106234457 standalone up up 649 3101 pgbouncer.default@mozhubs
mozillareality/youtube-dl-api-server/0.5/20230303041110 standalone up up 528 3851 youtube-dl-api-server.default@mozhubs
mozillareality/janus-gateway/2.0.1/20211220195030 standalone up up 611 3369 janus-gateway.default@mozhubs

[Decode1415]

I started from sratch and even with another domain name (I was trying with a subdomain on a second level domain) and I got exactly the same problem : no hubs folder in the assets bucket and the error message about missing index.html.

Please advise. Thanks.

[Decode1415]

There was an issue with my AWS account. I do not know what is it, but I created myself a brand new account and it worked on the first try. So the issue was on my side.