Skip to content

Latest commit

 

History

History
 
 

isolate

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
doc
doc
 
 
proto
proto
 
 
static
static
 
 
templates
templates
 
 
tools
tools
 
 
ui
ui
 
 
OWNERS
OWNERS
 
 
PRESUBMIT.py
PRESUBMIT.py
 
 
README.md
README.md
 
 
acl.py
acl.py
 
 
app.yaml
app.yaml
 
 
appengine_config.py
appengine_config.py
 
 
cloudstorage
cloudstorage
 
 
components
components
 
 
config.py
config.py
 
 
config_test.py
config_test.py
 
 
cron.yaml
cron.yaml
 
 
gae_ts_mon
gae_ts_mon
 
 
gcs.py
gcs.py
 
 
graphy
graphy
 
 
gviz
gviz
 
 
handlers_backend.py
handlers_backend.py
 
 
handlers_endpoints_v1.py
handlers_endpoints_v1.py
 
 
handlers_endpoints_v1_test.py
handlers_endpoints_v1_test.py
 
 
handlers_frontend.py
handlers_frontend.py
 
 
handlers_test.py
handlers_test.py
 
 
index.yaml
index.yaml
 
 
main_backend.py
main_backend.py
 
 
main_frontend.py
main_frontend.py
 
 
mapreduce
mapreduce
 
 
mapreduce.yaml
mapreduce.yaml
 
 
mapreduce_jobs.py
mapreduce_jobs.py
 
 
model.py
model.py
 
 
model_test.py
model_test.py
 
 
module-backend.yaml
module-backend.yaml
 
 
pipeline
pipeline
 
 
queue.yaml
queue.yaml
 
 
stats.py
stats.py
 
 
stats_test.py
stats_test.py
 
 
template.py
template.py
 
 
test_env.py
test_env.py
 
 
test_support
test_support
 
 
tool_support
tool_support
 
 

README.md

Isolate Server

An AppEngine service to efficiently cache large set of large files over the internet with high level of duplication. The cache is content addressed and uses Cloud Storage as its backing store.

Isolate enables sending temporary files around. It is a pure cache, files will be deleted.

Isolate can be used standalone when only files need to be transfered but no task scheduler is needed.

Documentation

Setting up

  • Visit http://console.cloud.google.com and create a project. Replace <appid> below with your project id.
  • Visit Google Cloud Console
    • IAM & Admin, click Add Member and add someone else so you can safely be hit by a bus.
    • IAM & Admin, Service accounts, click Create service account:
      • Name it "server@<appid>.iam.gserviceaccount.com".
      • Check Furnish a new private key and select P12.
        • TODO(vadimsh): switch to JSON keys.
      • Click Create.
    • Storage, click Create bucket, name it with the same . Do not use any pre-created bucket, they won't work.
      • Click on Browser on the left.
      • Click the 3 dots on the right of your new bucket and select Edit bucket permissions.
      • Enter the user "server@<appid>.iam.gserviceaccount.com" in the Add members field and select Storage Legacy Bucket Writer and Storage Legacy Object Reader from the Storage Legacy group.
      • Click Add.
    • Pub/Sub, click Enable API.
  • Upload the code with: ./tools/gae upl -x -A <appid>
  • If you plan to use an auth_service,
  • else
    • Visit "https://<appid>.appspot.com/auth/bootstrap" and click Proceed.
  • Visit "https://<appid>.appspot.com/auth/groups":
    • Create access groups as relevant. Visit the "IP Whitelists" tab and add bot external IP addresses if needed.
  • Visit "https://<appid>.appspot.com/restricted/config"
    • Set "API access client email address" to "server@<appid>.iam.gserviceaccount.com".
    • Follow the on-screen instructions to generate the base64 encoded DER private key.
    • Click Submit.
  • Tweak settings:
    • Visit Google Cloud Console
      • App Engine, Memcache, click Change:
        • Chose Dedicated.
        • Set the cache to Dedicated 5Gb.
        • Wait a day of steady state usage.
        • Set the limit to be lower than the value read at "Total cache size" in "Memcache Viewer".
    • App Engine, Settings, click Edit:
      • Set Google login Cookie expiration to: 2 weeks, click Save.