Skip to content

Amazon EC2

Jump to bottom
fred-wang edited this page Aug 8, 2012 · 19 revisions

The testing framework is being experimented on Amazon EC2. A more complete documentation will be available when the installation is done.

Amazon EC2

Pricing

AWS Management Console

  • Instances

    • Windows and Linux instances in use

  • Images

    • AMIs: images of the machines.
      • Linux
      • Windows

  • Elastic Block Store

    • Volumes:

      • Windows and Linux volumes of instances in use

    • Snapshots:

      • Windows (30GiB)
      • Linux (8GiB)

  • Network & Security

    • Security Groups

      • mathjaxtest
      • ICMP, TCP, UDP from mathjax-test: allow total communication between member of the security groups.
      • HTTP allow connection to the Web interface
      • SSH: allow connection to the Linux machine
      • RDP: allow connection to the Windows machine
      • The VNC port is disallowed. Use VNC over a SSH session for better security.
      • The Windows machine also has its own security configuration in addition to the above setting. In particular, "10.0.0.30" is in the list of Internet Explorer trusted sites and the Windows Firewall has specific rules to accept connection from the task handler.

    • Key Pairs

    • mathjaxtest: key to use for remote access (SSH and RDP)

Test Execution Overview

  • It may be a good idea to check that the programs on the machines and the testing framework are up-to-date, especially if the machines have not been used for a long time. See "Maintenance of the Testing Machine"

  • Create Amazon EC2 instances. See "Initializing the Testing Framework".

  • Control the Amazon EC2 instances. See "Remote Access".

  • Start the Selenium servers

    • Linux: In a separate terminal (e.g. a "screen" window or a terminal from the GUI), execute "~/seleniumServer.sh".
    • Windows: Open "Windows PowerShell" and execute ".\seleniumServer.bat"
    • If you have any problem to interrupt Selenium instances you may stop the server and run it again.

  • Use the QA Web Interface to create and run tasks. See http://devel.mathjax.org/testing/web/docs/html/qa-web-interface.html for details. TODO: give more explanation here?

  • Save the test results. Terminate the Amazon EC2 instances and release elastic IPs so that we won't be charged anymore. See "Stopping the Testing Framework".

Maintenance of the Testing Machine

  • "Micro On-Demand Instances" can be used to do maintenance tasks. They can be executed as normal Amazon EC2 instances, except that you can not use a VPC and thus can not assign an elastic IP.

  • Linux

    • Updating the programs: "sudo apt-get update && sudo apt-get upgrade"
    • Updating the testing framework. cd MathJax-test/ git pull etc

  • Windows

    • Updating the programs: use the graphical interface.

  • Preserving the state of the Linux or Windows machines Instances Actions => Create an AMI file. After a moment, a new AMI image and Snapshot will be created. You can now terminate the instance and delete the former AMI and Snapshot. Problem: it may be easy to unintentionally delete the Snapshot of a machine.

Initialization of the Testing Framework

  • Running an Instance

    • Click on Images => AMIs. Right click on an instance (Linux or Windows) and select "Launch Instance"
    • Choose an Instance Type, select VPC and Continue. Suggestion: use "High-CPU On-Demand Instances" for intensive testing. "Standard On-Demand Instances" may be fine otherwise.
    • Enter a private IP address for your instance, and Continue. The framework is configured to assume that IPs from 10.0.0.30 to 10.0.0.34 correspond to Linux machines while IPs from 10.0.0.40 to 10.0.0.44 correspond to Windows machines. Also the main machine (with the test suite, MathJax branches and Task Controller) is assumed to be 10.0.0.30.
    • Skip the two next screens (Storage Device Configuration, Custom Tags).
    • Choose the mathjaxtest Key Pairs and Continue.
    • Choose the mathjaxtest Security Groups and Continue.
    • Launch the instance.

  • Associating Elastic IP to an instance

    • Click on Network & Security => Elastic IPs
    • Allocate New Adress, choose VPC, Yes Allocate
    • Right Click on the Elastic IP => Associate
    • Associate IP to an instance, Yes Associate

Remote Access

  • Some SSH/VNC/RDP clients

    • Linux: vinagre (SSH, VNC), rdeskop (RDP)
    • Windows, Mac: ?

  • Linux Instance

    • Use a SSH client to connect to the instance.
    • Use the "screen" command to handle several windows. Quick guide to the screen environment: use "Ctrl + a, Ctrl + c" to create a new window, "Ctrl + a, Ctrl + n" to move to the next window, "Ctrl + a, Ctrl + p" to move to the previous window, use the "exit" command to close the current window and finally "Ctrl + a, Ctrl + d" to detach the screen session. When you detach a screen session, all the servers will keep running even if you logout from the SSH session. You can restore your screen session when you connect again with the command "screen -r" (provided the machine has not been stopped since your last connection).
    • In one screen window, use "~/init.sh" to start the VNC server. On the machine 10.0.0.30, this will also mount the mathjaxtest bucket on ~/MathJax-results/ and start the task handler.
    • Use a VNC client to connect to the GUI.

  • Windows Instance

    • Use a RDP client to connect to the instance.
    • Open a Windows Session.

  • Web Interface

  • Directory Structure

    • Linux
      • ~/init.sh: Bash script to mount Amazon S3 buckets, start the GUI, the VNC server and (if the host is 10.0.0.30) start the task handler.
      • ~/mount-AmazonS3.sh, ~/mount-AmazonS3.sh: Bash scripts to mount/umount Amazon S3 buckets
      • ~/start-gui.sh: Bash script to start the GUI and the VNC server.
      • ~/seleniumServer.sh: Bash script to start the selenium server
      • ~/start-task-handler.sh: Bash script to start the task handler
      • ~/MathJax-test/: MathJax testing framework
      • ~/AmazonS3/: mount points for Amazon S3 buckets
      • ~/Desktop/Webdrivers/: Drivers for Chrome. Added to $PATH by ~/.bashrc.
    • Windows
      • ~\selenium-server.bat: Bat script to start the Selenium server.
      • ~\selenium-server.jar: Selenium server.
      • ~\Desktop\Webdrivers: Drivers for Chrome and Internet Explorer. Added to $PATH.

  • Installation (see Platforms Supported)

    • Browsers: Firefox, Chrome, Internet Explorer, Safari, Opera, Konqueror.
    • Fonts: STIX and Mathjax fonts.
    • Plugin: MathPlayer
    • Selenium Servers (see directory structure)
    • Webdrivers (see directory structure)

Stopping the Testing Framework

  • Saving the test results to Amazon S3 The test results are saved on the main machine in MathJax-test/web/results/. They will be cleared when the instance terminates. To save a copy to Amazon S3, just copy the relevant files in ~/MathJax-test/web/results/public/. You need to use the Amazon S3 interface to make the new files public.

  • Terminating an Instance Warning: this will totally clear the instance and its associated volume. Be sure to save unfinished task before doing so. In particular save a copy of the test results to Amazon S3. See also "Maintenance of the Testing Machine" if you need to update the AMIs.

    • Right Click on an instance and select terminate.
    • Check "Release Elastic IPs", Yes Terminate

Organization of test execution

In this section, we find a way to optimize the test execution on EC2. We would like to test the four main rendering engines (Gecko, Presto, Webkit, Trident) and their supported output modes (HTML-CSS+STIX, HTML-CSS+TeX, SVG). For Internet Explorer, we also have 4 document compatibily modes to test. This gives 3+2+3+9=17 tasks to execute:

  • {Firefox} × {HTML-CSS+STIX, HTML-CSS+TeX, SVG}: Linux
  • {Opera} × {HTML-CSS+TeX, SVG}: Linux
  • {Chrome} × {HTML-CSS+STIX, HTML-CSS+TeX, SVG}: Linux (with Selenium 1 API) or Windows (Webdriver)
  • {Internet Explorer} × [ {HTML-CSS+STIX, HTML-CSS+TeX} × {Quirks, IE7, IE8, IE9} ∪ {(SVG, IE9)} ]: Windows

Some points to consider:

  • Pricing is per instance-hour consumed for each instance, from the time an instance is launched until it is terminated. Each partial instance-hour consumed will be billed as a full hour.
  • Execution of Linux is cheaper but not possible for all browsers. Moreover the task handler is installed on Linux, so when you execute a test on a Windows instance you also need to run a Linux instance at the same time.
  • More powerful hardware is faster but more expensive.
  • One can execute several EC2 instances in parallel but that's less convenient to set up and manage.
  • Webdriver is faster, more reliable and allow parallel execution on the same machine. However, it is not supported yet for Chrome+Linux and only one instance of Internet Explorer can be executed at a time.
  • Execution on Internet Explorer is the slowest and there are many configurations to test.
  • The testing framework can not handle too many tasks at once, for otherwise it automatically stops unresponsive tasks. We have to oversee the execution and run interrupted tasks manually again.
  • In addition to test execution, you need time to set up the testing framework, save the results to AmazonS3 and stop the testing framework.

Combinations experimented:

  • Running Linux Standard Small On-Demand Instances, the 5 Firefox&Opera tasks. Interruption: It seems hard for the testing framework to handle everything. Tasks are interrupted too often.
  • Running Linux Standard Small On-Demand Instances, the 3 Firefox tasks, followed by the 2 Opera tasks. Interruption: Opera tasks have been interrupted twice. Estimated Execution time: ~100min (Firefox) + ~45min (Opera). Estimated cost: $0.240
  • Running Linux High-CPU Medium On-Demand Instances , the 5 Firefox&Opera tasks. Interruption: one Opera task has been interrupted twice. Estimated Execution time ~40min. Estimated cost: $0.165
  • Running Linux Standard On-Demand Instances and 2 Windows High-CPU On-Demand Instances. Linux (5 tasks): Firefox, Opera Windows1 (6 tasks): {Quirks, IE7, IE8, IE9}×{HTML-CSS+STIX}, {Chrome}×{HTML-CSS+STIX, HTML-CSS+TeX} Windows2 (6 tasks): {Quirks, IE7, IE8, IE9}×{HTML-CSS+TeX}, {IE9,Chrome}×{SVG}

MathJax Wiki

Clone this wiki locally