New: Add worker service to serverless #6

sarvaje · 2019-09-11T17:28:28Z

No description provided.

sarvaje · 2019-09-24T17:17:51Z

I need to clean dependencies and '.*ignore', but I think this is ready for review

utsavized

Just a few sanity-check questions.

utsavized · 2019-09-24T20:23:26Z

host.json

+    "serviceBus": {
+      "messageHandlerOptions": {
+        "maxAutoRenewDuration": "00:01:00",
+        "maxConcurrentCalls": 5


These settings apply to all functions, right? Could there by any impact on the other functions, or they all ran fine? Just wanted to double check.

That file doesn't exists in that path anymore, but yes, it would apply to all functions I think.
For know, worker has this config, and the other functions use the default values.

When everything is in the same function, we will need to figure out if each function can have their own config, because for sync I think we can have more concurrent calls

utsavized · 2019-09-24T20:24:57Z

package.json

-  "dependencies": {
-    "@azure/functions": "^1.0.3",
-    "@destinationstransfers/ntp": "^2.0.0",
-    "@octokit/rest": "^16.28.7",


We don't need any of these as dependencies?

not in the root. remember that this is now a monorepo, each package has its own package.json

utsavized · 2019-09-24T20:28:46Z

package.json

@@ -55,16 +40,27 @@
  "private": true,
  "scripts": {
    "ava": "ava",
-    "build": "npm run clean && npm-run-all build:* && npm run build-function-templates",
+    "build": "yarn clean && yarn update:references && node scripts/test-all.js build",


We are having all the functions as packages now, instead of just being folder at the root? Why is that? Nothing wrong with it, I was just curious. Also, since the structure changes, I suppose the current pipeline will fail to deploy these functions, right? Have you tried the new structure is okay with Azure, by manually deploying these, or were you testing the workers only with existing already-deployed functions? I am asking because in my experience, the functions have been really particular about folder structures.

As we talk a few days ago, we are turning this in a monorepo, in this case, all functions except 'workers' are going to a folder, utils are going to another and then workers.

The deployment process is something that we need to figure out, but each "function" folder ('services' and 'worker') should contains everything necessary to be deployed in azure functions. We need to think about the utils, do we need to publish it in npm? can we use it without deploy it?

About utils for now I'm referencing to the path: "@online-service/utils": "file:../utils" so we don't need to publish it. The only think is that we need to be sure that 'utils' is compiled before use it.

We could look into bundle everything with webpack. It should decrease the load time as there will not be hundreds of smaller files. Maybe for another update?

utsavized · 2019-09-24T20:32:37Z

packages/services/functions.json

@@ -17,7 +17,7 @@
            "scriptFile": "../dist/src/functions/azure/syncservice.js"
        }
    },
-    {   
+    {
        "name": "statusservice",
        "template": {
            "disabled": false,


Do we need this file anymore?

this file is the one used to build the function.json for each function. Do you want me to delete it?

utsavized · 2019-09-24T20:33:14Z

packages/services/scripts/create-function-templates.js

+        }
+    });
+};
+


We probably don't need this file either, right?

utsavized · 2019-09-24T20:34:58Z

packages/services/serverless.yml

 functions:
  sync-service:
    handler: dist/functions/azure/sync-service.run
    events:
-      - serviceBus: 
+      - serviceBus:
        x-azure-settings:
          queueName: webhint-results
          name: message


Don't worry about changes in this file for now. To make Serverless.yml work, step 1 was to convert everything to functions, which we are close to completing. Step to is for me to either send a PR to them, or fork our own version and update all the Azure dependencies and some logic because they are lagging behind by more than a year.

sarvaje · 2019-10-31T21:20:02Z

move architecture.md to this repo

There is no architecure.md in the previous online scanner. I have created a task to add it.

sarvaje · 2019-10-31T21:28:11Z

@molant @antross this is ready for another review.

.vscode/launch.json

utsavized

Looks good to me.

molant

Let's ship it!

molant · 2019-11-18T20:45:45Z

🚀🎉🥳

sarvaje force-pushed the worker-serverless branch from 1e48113 to 739275e Compare September 12, 2019 21:54

utsavized reviewed Sep 24, 2019

View reviewed changes

sarvaje requested review from antross and molant September 25, 2019 15:55

sarvaje marked this pull request as ready for review September 25, 2019 15:55

sarvaje force-pushed the worker-serverless branch 24 times, most recently from 6e288e0 to fb2e3db Compare October 1, 2019 23:17

sarvaje force-pushed the worker-serverless branch 2 times, most recently from f594840 to d1de30e Compare October 31, 2019 21:02

sarvaje added 19 commits November 1, 2019 13:14

New: Add worker service to serverless

768e649

New: Change to monorepo

bb50d1b

Fix: Update and clean dependencies

8979c71

Chore: Remove unnecessary typing

10d465f

Fix: Warning in mongoose and config manager

d4bed52

Fix: Dependencies, Dockerfile and worker function.json

6b81038

Fix: Services build

4399b2e

Fix: Added chrome to Docker and clean up dependencies

9aa2506

Fix: Pipelines configuration to deploy functions using the monorepo

02cb549

New: Terraform script to deploy Infrastructure

2be14a4

Fix: Install only production packages when deploying consumption plan

74b9430

Fix: Address some feedback

06f8904

Fix: Use json for HTTP Post trigger

dd94ff5

Fix: Explain why we copy the tsconfig.json in Docker

ba7722c

Fix: Remove and refactor cmdline@2 from azure pipelines

103e381

Fix: Remove status function

d4232cf

Fix: Use trackEvent instead of trackMetrics

8419681

Fix: Add note about why we use fork to run webhint

0994e1c

Fix: Move telemetry to sync service

b1fd1ee

sarvaje force-pushed the worker-serverless branch from 6bbbd08 to b1fd1ee Compare November 1, 2019 20:15

antross approved these changes Nov 11, 2019

View reviewed changes

utsavized reviewed Nov 12, 2019

View reviewed changes

.vscode/launch.json Show resolved Hide resolved

utsavized approved these changes Nov 12, 2019

View reviewed changes

molant approved these changes Nov 12, 2019

View reviewed changes

sarvaje merged commit 62528ec into webhintio:master Nov 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New: Add worker service to serverless #6

New: Add worker service to serverless #6

sarvaje commented Sep 11, 2019

sarvaje commented Sep 24, 2019

utsavized left a comment

utsavized Sep 24, 2019

sarvaje Sep 24, 2019

utsavized Sep 24, 2019

sarvaje Sep 24, 2019

utsavized Sep 24, 2019

sarvaje Sep 24, 2019

sarvaje Sep 25, 2019

molant Oct 28, 2019

utsavized Sep 24, 2019

sarvaje Sep 24, 2019

sarvaje Sep 25, 2019

utsavized Sep 24, 2019

sarvaje Sep 25, 2019

utsavized Sep 24, 2019

sarvaje commented Oct 31, 2019

sarvaje commented Oct 31, 2019

utsavized left a comment

molant left a comment

molant commented Nov 18, 2019

New: Add worker service to serverless #6

New: Add worker service to serverless #6

Conversation

sarvaje commented Sep 11, 2019

sarvaje commented Sep 24, 2019

utsavized left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sarvaje commented Oct 31, 2019

sarvaje commented Oct 31, 2019

utsavized left a comment

Choose a reason for hiding this comment

molant left a comment

Choose a reason for hiding this comment

molant commented Nov 18, 2019