How does it work?

Funktor is a framework for running background job processing in AWS Lambda. Lambda can scale almost infinitely almost instantly and you only pay when you code is running. It's perfect for bursty jobs.

Funktor provides Ruby code for pushing jobs onto a queue and for executing the jobs in Lambda. It also provides a means to deploy your application and provision AWS resources via Serverless, but Serverless is not a requirement for using the basic library code. You could provision the required resouces any way that you'd like, and deploy you application by other means.

There are 3 main components to using Funktor:

• Provisioning SQS queues and a DynamoDB table
• Pushing jobs onto the Incoming Job Queue
• Executing jobs from one or more work queues

SQS & DynamoDB

Funktor uses an Incoming Job Queue as the single interface to the outside world. When you push a job to the Incoming Job Queue it will then distribute the job to the appropriate work queue.

SQS queues allow jobs to be scheduled for only up to 15 minutes in the future. Funktor works around this limitation by using DynamoDB table to store jobs that are scheduled farther in the future.

A lambda function is executed once per minute that looks for jobs in the DynamoDB table that are scheduled to happen in the next couple of minutes and it pushes those jobs onto the appropriate work queue where they are executed at the appropriate time.

The DynamoDB table is also used to store basic stats about the number of jobs in each possible state ('queued', 'processing', etc...), as well as infomation about the current state of each job that has yet to complete.

Provisioning queues and tables

The following commands will provision queues and your table for you.

funktor bootstrap my-funktor-app
cd my-funktor-app
funktor init
serverless deploy -v

TODO : Add some details (possibly in another doc) about what is required of these queues and how one might go about provisioning them outside of serverless.

Pushing jobs onto the Incoming Job Queue

To push a job to the queue you can create a simple worker:

class HelloWorker
  include Funktor::HelloWorker

  def perform(greeting)
    puts greeting
  end
end

And then you can push a job to be performed as soon as possible:

HelloWorker.perform_async("Hello world!")

Or you can schedule a job to run in the future:

HelloWorker.perform_in(5.minutes, "Hello world!")

Note that the arguments passed to your worker will be serialized to JSON before being pushed to SQS. This means that your job arguments should be simple data types (numbers, strings, boolean, array, hash) and not complex Ruby objects. Complex objects like Date or ActiveRecord models will not work properly.

Executing jobs

Funktor will provision Lambda funtions for you, and will connect them to your SQS work queues.

This means that you don't need to do anything to get your jobs to run. When a job is pushed to a queue AWS will handle pushing that job to Lambda and executing your handler.

If your job raises an error Funktor will automatically retry you job using an exponential backoff schedule. (TODO : Link to the doc about retries...)