Standard Service (w/o Realtime)

By default, your jobs take advantage of cores that are shared by all of our users. We do not have a set number of total cores in our system, and there are no set number of cores per user. Instead, PiCloud continually and automatically estimates the workload we have and will have in the next hour. There are two components to this estimation:

1. Periodic Jobs: A large class of users runs jobs periodically, whether it’s every minute, hour, or day. Over time our system learns the amount of load each of these users contributes, and increases server capacity in anticipation of periodic jobs.
2. Aperiodic Jobs: For users who do not have a predictable pattern, we scale our worker nodes once their jobs have been added to our queue. The number of workers we bring up depends on how long we estimate their jobs will take.

How do we estimate the runtime of a job?

We estimate the runtime of enqueued jobs by the average and variance of runtimes of similar completed jobs; a similar job is any job by the user that executed the same Python function with the same PiCloud-specific reserved keyword arguments. Keyword arguments such as _type are important since they could affect the speed at which a job is processed.

How does your workload affect the number of workers we bring up?

Since we use Amazon Web Service’s servers, we are charged for every hour we have a server up. To manage our costs, we try to make sure our servers spend as much time as possible processing jobs, rather than sitting idle. Here’s a rule of thumb for how this affects you: If each of your jobs takes an hour to process, PiCloud will automatically scale such that all of your jobs are running simultaneously in parallel. We’re happy to bring up as many as you need since you’re keeping our servers busy for the full hour increment that we rent from Amazon. If each of your jobs takes a half hour, we’ll run about half of your jobs in parallel at a time, so that each core will be busy for the full hour running two jobs each. In practice, our scaling algorithm is more liberal than this, but this should give you a good conservative ballpark.

How much compute power should you expect?

If you’re just running a small batch of short jobs, you’ll generally get less than 10 cores. If you’re running a large number of long-running jobs, then using the above algorithm, we could potentially bring up hundreds of cores for you.

Realtime Service

The model of our standard service is ideal for users trying out PiCloud for the first time, users with relaxed response-time requirements, and users with large batch jobs. Our realtime service is for our other class of users who need N cores to be ready at a moment’s notice, even if that means those cores will be sitting idle at times.

How it Works

You tell us the type of core you want, and the number you want, and we’ll bring them online just for you. Your jobs will be placed in both your own realtime queue, as well as our general queue. So if you have reserved 80 cores, and you check in 200 jobs, at least 80 jobs will be immediately be dequeued and begin processing on your realtime cores. Potentially more than 80 may run simultaneously since you still have access to the cores available in our standard service.

How we (want to) Charge

Right now, we charge a fixed cost for every core hour (rounded up). If you reserve realtime cores, and you run no jobs, you’ll be charged that amount. If you run jobs, you’ll be charged both the cost of the realtime cores, as well as the cost of our standard service for the number of compute hours your jobs run for. For example, if you reserve 10 c1 cores for an hour, and run 1 compute hour worth of jobs on c1 cores during that time, you’ll pay $0.15 + $0.05 = $0.20 cents total (See our pricing page). The problem we have with this is two fold:

1. The fixed cost of each realtime reservation needs to be sufficiently high to cover our expenses in the event a user does not run any jobs. Charging our elastic rate on top of that is unfairly expensive.
2. If our standard service scaled a user to 100 cores, and the user then reserved 30 cores, they would not see any additional compute power, while paying substantially more. While it’s true that they would be guaranteed that compute power without any risk of crowding from other users, we don’t believe users were getting the best bang for their buck.

We love users who use realtime because they offer our automatically-scaling system a level of predictability; we don’t want to dis-incentivize this behavior. So we’re going to change our policy as follows: Each realtime core, rather than being an hourly charge, will instead be an hourly minimum. Hypothetically, if you reserved a realtime core that has a minimum of $0.10/hour, then after an hour you would be charged the greater of $0.10 or the bill you accrued from using our standard service. If you don’t run any jobs, that’s $0.10. If you run some jobs, but the standard service bill doesn’t exceed $0.10, then you pay $0.10. If you run enough jobs to exceed $0.10, say $0.15, then you pay just $0.15. We’ll let you all know when we’ve made this change to our pricing!

How to Reserve Realtime Cores

Go to the realtime tab on our web dashboard.

Select the type of core, and the number of cores you want for us to keep available for you, and then hit submit.

When the cores are ready, you’ll receive an email and the “start time” will be set.

When you run jobs, make sure that you set the _type keyword to the core type that you have reserved. In this example, we reserved 160 c2 cores.

cloud.call(f, _type='c2')

Alternatively, reserving realtime cores will be possible through the Python console using the cloud library in its next release:

cloud.realtime.request('c2', 160)

How long does it take to allocate Realtime Cores?

We’ve been working hard making allocation time faster and more consistent, which has been a common issue raised by users. With our latest design, it should take no more than 15 minutes for cores to be provisioned!

How many cores can you allocate?

We have users who provision hundreds and thousands of cores using our realtime service. If you’re worried you might break our backs, file a support ticket and let us know what you’re up to.

Conclusion

The takeaway we want to leave with you is as follows: The PiCloud platform will not only get you up and running in no time with no configuration, but will also be able to meet your exact computational requirements if necessary—even if you need 1000 cores in 15 minutes. It’s not a question of if, but how, and we’re working to make how as easy as possible.

We’ve included a list of recipients below so you can see some of the great work that will be benefiting from our platform.

If you missed out on our first cycle of grants, don’t sweat! We plan on having a second cycle in the near future.

List of Recipients

Chris Beaumont
PhD candidate at the Institute for Astronomy
University of Hawaii at Manoa

Developing an automated technique to search astrophysical images for “bubbles” blown by massive young stars. This will help astronomers handle their rapidly growing datasets, which we currently examine by eye when looking for morphologically complex objects.

Damian Borth
PhD candidate
University of Kaiserslautern
German Research Center for Artificial Intelligence (DFKI)

Video concept detection using web-based training sources like YouTube. I particularly focus my interest on developing learning approaches which adopt to weak labels and deal with the domain change problem.

Liang Yuxian Eugene
Research Assistant
National Cheng Chi University

Modeling the Lifecycle of Natural Disasters.

Patrick Henaff
Associate Professor of Finance
Institut d’Administration des Entreprises, Université Paris I Panthéon-Sorbonne

Motivated by the current financial crisis, my research focus is to understand and measure model risk in computational finance, ie. the risk associated with the use of an inappropriate model to price and manage financial derivatives. Inspired by other scientific disciplines, in particular computational biology, this research project will also experiment with a new type of scientific production, based on the publication and peer review of of self-contained, reproducible experimental protocols, as opposed to summary results.

Roy Keyes
Phd candidate in the Department of Physics and Astronomy
University of New Mexico

Developing faster, more accurate methods of calculating radiation dose to improve cancer treatment along with associated research tools.

Meredith Lehmann
La Jolla High School

SARS and the swine flu rekindled interest in the spread of epidemics and a large literature appeared to conclusively prove the intuitive proposition that modern epidemics spread disproportionately through large hub airports near large population centers. My research challenges the conventional wisdom using a different model of the US transportation network. A different picture of epidemic propagation in the continental US emerges from this analysis. Primarily auto, not air, travel seeds counties with a modest number of infecteds and the subsequent explosion at the county level dominates further seeding by infected travelers. No preferred epidemic pathways arise in my simulated world and may not be present in the real world either.

Eric Lofgren
PhD candidate in the Department of Epidemiology, Gillings School of Global Public Health
University of North Carolina at Chapel Hill

The mathematical modeling and simulation of the transmission of Clostridium difficile in healthcare settings.

Ariel Rokem
Post-Doctoral Researcher at the Department of Psychology under Professor Brian Wandell
Stanford University

Using Magnetic Resonance Imaging (MRI) to measure physiological and anatomical properties of the human visual system. In particular, trying to understand the biological underpinnings of differences in perception between the center of the visual field and the periphery.

Thomas Wiecki
PhD candidate under Professor Michael J. Frank
Brown University

My research aims at elucidating the cognitive and neurobiological mechanisms underlying decision making in the healthy and diseased brain. Towards this goal I am simulating parts of the intact brain in the computer and model how different brain lesions and psychiatric diseases influence brain activity and behavior. Ultimately, this research will help in developing better diagnosis and treatment options for brain disorders such as schizophrenia and Parkinson’s disease.

• If you ran out of your original 5 core hour credits, you can come back and play around some more!
• If you have minimal computing needs, this means that you can now use PiCloud regularly without even having to enter a credit card.

Looking for more? Don’t forget, we’re giving away $500 worth of credits as part of our Academic Research Program. Applications are due this Thursday, October 27th.

Since this is the first time we’re doing this, we want to keep it simple. If you wish to apply, send an e-mail to research-funding@picloud.com by Thursday, October 27th with the following:

1. Full name
2. Organization or educational Institution
3. Your position
4. Short biography
5. A summary of your research field and project. Feel free to include conference papers, publications, and links to project websites. Please emphasize how PiCloud’s compute power will facilitate your research.

We will be awarding two submissions with free core hours. Winning researchers will also have an opportunity to get larger exposure for their projects on our blog and website. We’re looking forward to your submissions!

Background Information

We are developing various models that help define the life cycles of natural disasters. In order to develop these models, we need to process large amounts of data; the data comes from social networks such as Twitter and official government sources.

As you can imagine, computing and comparing data across different sources can be difficult, not to mention the scale of the data that needs to be processed in a moment’s notice. In addition, the research project is undertaken by social scientists; there’s a constraint in terms of budget and technical personnel; I am one of the few technical personnel working on this research project.

My objective is to develop a software system that helps discover and detect the life cycles of natural disasters, all within the constraints mentioned above.

The general process of my software system is as such:

1. Scrape data and store it in a MongoDB cluster hosted on Amazon EC2.
2. Retrieve the data and perform analysis on it, such as natural language processing.
3. Store the results of step 2.

I am currently running my application on Ubuntu 10.04 and using MongoDB as my main data store. Python is my primary programming language. I’ll be sharing two ways in which PiCloud helped me cut down development time and costs tremendously.

How PiCloud Saved My Day with the s1 Core Type

As mentioned earlier, I am required to scrape data using Twitter’s API. I originally used PiCloud’s c1 core type; it worked for a while but it quickly ran into throttling issues. Fortunately, PiCloud announced the s1 core type just a day before I faced throttling issues.

PiCloud’s s1 core type is optimized for mining and scraping operations since each core uses a different IP address; the effects of IP throttling limits are minimal. Should I attempt to replicate a similar architecture to perform mining operations, I might not be able to complete the alpha version of the software system.

I now find myself leveraging the s1 core type; which helps me to scrape and mine for various data sources, in particular Twitter and other news sources.

Switching to s1 core type is extremely easy, all we needed to do is change the type keyword:

# from
_type = 'c1'
# to
_type = 's1'

Thanks to PiCloud’s s1 core type, I can continue to collect data with minimal worries.

How PiCloud’s Serverless Architecture Saves Development Time

As I am storing my data on Amazon EC2 via a MongoDB cluster, I need a way to for PiCloud to ’speak’ with my database. My initial application has a REST interface that allows PiCloud to ’speak’ with MongoDB; each REST endpoints represents a cost in terms of development time, and ongoing maintenance.

On average, I need to spend 20 minutes or more to develop a specific REST endpoint for a specific use. Now using PiCloud, I do not even need to develop or maintain these REST endpoints. PiCloud’s serverless paradigm means all I need to do is allow PiCloud to read my database directly and calculations can be performed immediately. Here’s how it works:

We typically connect to MongoDB on a localhost by doing the following:

from pymongo import Connection
connection = Connection('localhost')

However, we can run the above code on PiCloud by simply changing localhost to the external IP address of the Mongo server so that PiCloud knows how to reach it.

from pymongo import Connection
# 'xxx.xxx.xxx.xxx' is the IP address and 12345 is the port number
connection = Connection('xxx.xxx.xx.xxx', 123456)

If we want the same code to work both locally and on PiCloud, we use cloud.running_on_cloud() to check how we should connect to MongoDB.

from pymongo import Connection
if cloud.running_on_cloud():
    # 'xxx.xxx.xxx.xxx' is the IP address and 12345 is the port number
    connection = Connection('xxx.xxx.xx.xxx', 123456)
else:
    connection = Connection('localhost')

This means that I can leverage hundreds of cores automatically without any server management or configuration on my part. All I need to do is to write Python code and PiCloud takes care of the rest.

I am also leveraging PiCloud’s other core types, which includes c1, c2 and m1 core types, depending on the amount of computing required. As I am required to compute the data frequency, keyword frequency (and many other metrics) within a moments notice, PiCloud’s serverless paradigm fits perfectly as I do not need to spin up an entire EC2 cluster or even a single instance just to perform my calculations.

Summary

PiCloud has taken care of my server management and configuration needs. The ability to call up computing power without the need to spin up new EC2 instances is a huge time saver. It has saved me tremendous amount of time and effort, allowing me to focus on what is truly important, algorithms and calculations of data.

Eugene enjoys solving difficult problems creatively in the form of building web applications using Python/Django and JavaScript/JQuery. He also enjoys doing research related to the areas of social computing, social media analysis, recommendation algorithms, link analysis, data visualization, data mining, information retrieval, business intelligence, and intelligent user interfaces. You can find him at http://www.liangeugene.com.

Why is this important?

Using unique IPs is necessary to minimize the automated throttling most sites will impose when seeing fast, repeated access from a single IP.

How do I use it?

If you’re already using our c1 cores, all you’ll need to do is set the _type keyword.

cloud.call(func, _type=’s1′)

How much?

$0.04/core/hour

Why don’t other cores have individual IPs?

For other core types, such as c2, multiple cores may be running on a single machine that is assigned only a single IP address. When using s1 cores, you’re guaranteed that each core sits on a different machine.

Suggestions?

We’re excited to move the s1 core type out of beta for our customers. If you have any suggestions for other core types you would like to see, please let us know.

Cloud computing is a revolutionizing technology. It sidesteps the need for buying, setting up and maintaining powerful computers for people who would rather spend time and effort directly on their high-level innovations.

In our particular use case, we are using cloud computing for bioinformatics, specifically on building a database and in developing a pipeline for comparative analysis of a subset of genes across different E. coli genomes. I work in a lab that does not do bioinformatics as a primary research interest so I don’t have immediate access to an in-house computer cluster to do some computationally intensive tasks. This led me to explore cloud computing, and since I write scripts in Python, a search on “cloud computing” and “Python” in Google led me to PiCloud.

After experimenting with Amazon EC2 myself trying to figure out how to build a smooth interface to it from my pipeline, it was a sigh of relief to know about a Python library that does exactly this…and it does it in a distinctly Pythonic way – simple and elegant. With PiCloud’s library, you can deploy your computations to the cloud, in a massively parallel mode if need be, with minimal modifications to your Python code.

At the moment, I mainly use PiCloud on two fronts. I am using it for some critical functions in a web-based biological database of our genes/proteins of interest. The database is running on Google App Engine (which in itself is another breed of cloud computing service). App Engine is great but it has certain limitations that prevent biological databases from running in its platform alone. I needed to use NumPy, matplotlib and Biopython among other Python packages as well as command line programs written in C and C++ (e.g. ClustalW, EMBOSS and HMMER). All of which I couldn’t install or use in App Engine. In the pipeline development aspect of my work, which I do on my laptop; while I do have all these packages and programs installed, certain computations take a lot of time and can slow down my system. I now submit these tasks to the cloud for processing where I can choose to run them a lot faster.

Python packages and/or programs written in other languages that you need can be used within PiCloud with their Environments feature. Configuring an environment is essentially setting up a filesystem to include all of the packages and programs you need. You configure a filesystem by SSH-ing into a Linux virtual machine, where you can install packages and programs through the command line as you would on any machine. After you log off of the machine, you save the environment, and then you’re ready for action. As long as you’ve installed all the things you need, you never have to do the configuration again. However, if you need to install more, you just connect again to the VM through SSH and modify it…it’s that easy.

Running functions on PiCloud is just as easy as calling:

>>> cloud.call(your_function, 'argument')

and if you need to use your preconfigured environment you just add the _env keyword argument:

>>> cloud.call(your_function, 'argument', _env='your_saved_environment')

For our database that runs on Google App Engine, invoking cloud.call() won’t work because, just like the other Python packages mentioned, I also can’t install the PiCloud library there. Luckily, PiCloud has a way to talk with any web app through its REST API. For this you need to register functions that you would like to call through REST:

>>> cloud.rest.register(your_function, 'name_of_your_function')

which will give you the URL where you will send your HTTP GET or POST requests to invoke the function, which then returns the job ID as a response. The status of the job and the results are fetched by making GET requests to another set of URLs given the job ID as a parameter. (Please refer to the docs for more details.)

In our database, we use the REST API for generating operon maps and for fetching sequences from databases both using functions that depend on Biopython. We also have REST functions for doing sequence alignment (using ClustalW or EMBOSS) in the cloud from inputs received from the web app. With PiCloud’s REST API and Google App Engine, being both cloud-based platforms, we made ourselves a working prototype of how to leverage cloud computing for building biological databases that can potentially scale without the need for a major rewrite of the code to adapt to any future spike in usage and traffic.

“this would normally take 20 to 25 hours… I was able to run it in PiCloud in only 50 minutes”

In my latest run of our offline bioinformatics pipeline, I analysed more than a 100 E. coli genomes where we found around 1400 of our protein of interest, in total. I needed to do sequence alignment of all the pairwise combinations of those 1400 proteins which is around a million protein pairs. In my laptop (2.26 GHz Intel Core 2 Duo with 4GB RAM), this would normally take 20 to 25 hours to complete without parallelization. I was able to run it in PiCloud in only 50 minutes and I could have run it faster with a more optimized code. All steps in the pipeline can be done from a web app, again through the REST API. Therefore, a plan for the final “publicly consumable” form of the program is to make it completely web-based…completely running on the cloud.

With an easy access to the cloud, the possibilities are unlimited. Now, ordinary biologists can do computationally demanding tasks without the need to delve deep into server and database management and without the complexities and hassles of running jobs in grids and clusters.

Joemar Taganna is a PhD candidate at the Flanders Institute for Biotechnology (VIB) in Belgium. He works in a collaborative project between the Laboratory of Structural and Molecular Microbiology in Vrije Universiteit Brussel and the Laboratory of Medical Biotechnology in Universiteit Gent. The project aims to have a holistic view of the major adhesive protein arsenal of various E. coli pathotypes in order to elucidate pathogen-specific targets for anti-adhesive therapeutics.

1. Install any non-Python software package you need via apt-get or make.
2. Install any Python module that we do not automatically extract from your machines, which are typically those that require compilation or depend on external libraries.

In this post, we’ll show you how to create and use your first environment. We’ll be installing the ObsPy package, which is a Python toolbox for processing seismological data.

Why Environments?

We strive to make moving your computation to the cloud as easy as possible. That’s why our cloud Python package automatically detects and transfers dependencies over to our cloud.

import cloud
from your_expansive_library_of_functions import complex_function
# cloud.call transfers all the modules needed to run complex_function on PiCloud
cloud.call(complex_function)

Unfortunately, automatic dependency transfer only works for pure Python modules. The ObsPy package requires both a .pth file and C-code compilation for proper operation. So the following simple function quickly runs into problems:

def simple_function():
    import obspy

>>> jid = cloud.call(simple_function)
>>> cloud.result(jid)
[Mon Sep 19 16:39:13 2011] - [WARNING] - Cloud: Job 1337 threw exception:
 Could not depickle job
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cloud/serialization/cloudpickle.py", line 679, in subimport
    __import__(name)
ImportError: No module named obspy

Importing ObsPy fails because it could not be transferred to PiCloud in working form. You might be wondering how you’ve been able to use NumPy, SciPy, and other natively-compiled libraries on PiCloud. The answer is we have many libraries pre-installed on our systems. Here are the respective links for what we have pre-installed for Python 2.6, and Python 2.7.

Creating a new Environment

Step 1: Go to the Environments tab in the Control Panel.

Step 2: Click “create new environment”.

A popup box will appear. The Base Environment option allows you to choose what distribution of Ubuntu Linux you would like to use as the base filesystem. It’s important to understand why we give you this option. If you use Python 2.7 on your local machine to offload computation to PiCloud, we will run your functions in the Python 2.7 interpreter from the Ubuntu 11.04 (Natty) base. If you use Python 2.6 on your local machine to offload computation to PiCloud, we will run your functions in the Python 2.6 interpreter from the Ubuntu 10.10 (Maverick) base. We are consistent about which interpreter we use since the modules you install in your environment may compile against a specific version of Python. In short, if you’re using Python 2.6 on your machines, but you use the Natty base, or vice versa, you will most likely run into compatibility issues.

The Environment Name is the name you’ll use to reference the environment in your jobs. The Environment Description is for yourself and/or your team to keep track of the purpose and contents of each environment.

Step 3: Click submit.

When you click submit, your environment will appear under the “Environments being configured” tab. You may have to wait a minute or two while we boot and configure a server with the appropriate base environment for you.

For our example, we’ve named the environment seismology_env.

Connecting to your Environment Setup Server

When the server is ready, click the connect link. Note that the instructions are currently tailored towards *nix environments. If you are using Windows and do not have an SSH client, we recommend Tunnelier.

Download the private key we have generated for you. You will use this same private key for all future environment setup servers. SSH enforces that only the owner should have access to the file, which is why we instruct you to run chmod 400 privatekey.pem. Once you’ve done that, SSH into the provided server using the private key by using the -i flag as shown in the instructions.

Getting Around Your Environment

Once you’ve SSH-ed in, you’ll find yourself in a Ubuntu Linux filesystem environment.

picloud@ip-10-46-223-4:~$ ls /
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  sbin  selinux  srv  sys  tmp  usr  var

Your current working directory is /home/picloud:

picloud@ip-10-46-223-4:~$ pwd
/home/picloud

You can verify the distribution of Ubuntu you’re using:

picloud@ip-10-46-223-4:~$ cat /etc/issue
Ubuntu 11.04 \n \l

We give you sudo access so that you have the freedom to install anything anywhere.

# this does not produce an error
picloud@ip-10-46-223-4:~$ sudo touch /root/i_can_be_root

Important: The owner and group for files and directories in your environment do not matter. While you’ll be using the setup and root user accounts, your jobs will be run with an entirely different user account that will have access to the entire filesystem environment.

Setting Up Your Environment

We’ll use sudo access to install the ObsPy library.

picloud@ip-10-46-223-4:~$ sudo pip install obspy.core obspy.signal
Downloading/unpacking obspy.core
  Downloading obspy.core-0.4.8.zip (186Kb): 186Kb downloaded
  Running setup.py egg_info for package obspy.core

    no previously-included directories found matching 'docs/other/*'
Downloading/unpacking obspy.signal
  Downloading obspy.signal-0.4.9.zip (4.0Mb): 4.0Mb downloaded
  Running setup.py egg_info for package obspy.signal

Requirement already satisfied (use --upgrade to upgrade): numpy>1.0.0 in /usr/local/lib/python2.7/dist-packages (from obspy.core)
Requirement already satisfied (use --upgrade to upgrade): scipy in /usr/local/lib/python2.7/dist-packages (from obspy.signal)
Installing collected packages: obspy.core, obspy.signal
  Running setup.py install for obspy.core

    no previously-included directories found matching 'docs/other/*'
    Skipping installation of /usr/local/lib/python2.7/dist-packages/obspy/__init__.py (namespace package)
    Installing /usr/local/lib/python2.7/dist-packages/obspy.core-0.4.8-nspkg.pth
    Installing obspy-runtests script to /usr/local/bin
  Running setup.py install for obspy.signal

    building 'libsignal' extension
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c obspy/signal/src/recstalta.c -o build/temp.linux-x86_64-2.7/obspy/signal/src/recstalta.o
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c obspy/signal/src/xcorr.c -o build/temp.linux-x86_64-2.7/obspy/signal/src/xcorr.o
    ...
I/usr/include/python2.7 -c obspy/signal/src/fft/fftpack_litemodule.c -o build/temp.linux-x86_64-2.7/obspy/signal/src/fft/fftpack_litemodule.o
    gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions build/temp.linux-x86_64-2.7/obspy/signal/src/recstalta.o build/temp.linux-x86_64-2.7/obspy/signal/src/xcorr.o build/temp.linux-x86_64-2.7/obspy/signal/src/coordtrans.o build/temp.linux-x86_64-2.7/obspy/signal/src/pk_mbaer.o build/temp.linux-x86_64-2.7/obspy/signal/src/filt_util.o build/temp.linux-x86_64-2.7/obspy/signal/src/arpicker.o build/temp.linux-x86_64-2.7/obspy/signal/src/bbfk.o build/temp.linux-x86_64-2.7/obspy/signal/src/fft/fftpack.o build/temp.linux-x86_64-2.7/obspy/signal/src/fft/fftpack_litemodule.o -o build/lib.linux-x86_64-2.7/obspy/signal/lib/libsignal.so
    Skipping installation of /usr/local/lib/python2.7/dist-packages/obspy/__init__.py (namespace package)
    Installing /usr/local/lib/python2.7/dist-packages/obspy.signal-0.4.9-nspkg.pth
Successfully installed obspy.core obspy.signal
Cleaning up...

As you can see, installing obspy.signal requires compiling C code with references to the NumPy library. We would not have been able to automatically extract this package from your machine.

Save the Environment

When you click “save” from the Environment Panel, your SSH connection will be closed. The length of time it takes to save your environment depends on how much you’ve installed. Once it’s ready, your new Environment will be listed under the “Your environments” section.

Using Your Environment

To use an environment, use the _env keyword argument to specify the environment you want to use by name. _env is valid for cloud.call, cloud.map, cloud.cron.register, or cloud.rest.publish.

To demonstrate, we will run a beamforming algorithm using the ObsPy library that we just installed. Beamforming is a technique used in seismology for geolocating seismic events. In this case, the event is the demolition of the AGFA skyscraper in Munich. We’ve derived the example from here.

Step 1: Upload the recorded dataset from the demolition to cloud.files

cloud.files.put('agfa.dump')

Step 2: Define the beamforming function.

Note that our version pulls the data from cloud.files directly into memory using the getf function.

import cloud
import pickle, urllib
from obspy.core import UTCDateTime
from obspy.signal.array_analysis import sonic
from obspy.signal import cornFreq2Paz

def beamforming(file_name):
    st = pickle.loads(cloud.files.getf(file_name).read())

    # Instrument correction to 1Hz corner frequency
    paz1hz = cornFreq2Paz(1.0, damp=0.707)
    st.simulate(paz_remove='self', paz_simulate=paz1hz)

    # Execute sonic
    kwargs = dict(
	# slowness grid: X min, X max, Y min, Y max, Slow Step
	sll_x=-3.0, slm_x=3.0, sll_y=-3.0, slm_y=3.0, sl_s=0.03,
	# sliding window propertieds
	win_len=1.0, win_frac=0.05,
	# frequency properties
	frqlow=1.0, frqhigh=8.0, prewhiten=0,
	# restrict output
	semb_thres=-1e9, vel_thres=-1e9, verbose=True, timestamp='mlabhour',
	stime=UTCDateTime("20080217110515"), etime=UTCDateTime("20080217110545")
    )

    return sonic(st, **kwargs)

Step 3: Run it on PiCloud using the _env keyword.

Since this is a computationally intensive task, we use the c2 core type to take advantage of 2.5 compute units of power.

>>> jid = cloud.call(beamforming, 'agfa.dump', _env='seismology_env', _type='c2')
>>> out = cloud.result(jid)
>>> out
array([[  7.33088462e+05,   6.52313948e-01,   4.77909058e-17,
          1.63009177e+02,   1.12929181e+00],
       [  7.33088462e+05,   6.54728115e-01,   4.92721051e-17,
          1.58838740e+02,   9.97246208e-01],
       [  7.33088462e+05,   6.65887892e-01,   5.40099541e-17,
          1.58552264e+02,   9.02496537e-01],
       ...,
       [  7.33088462e+05,   7.97200561e-01,   2.54941247e-16,
          1.84349488e+01,   1.23328829e+00],
       [  7.33088462e+05,   7.93642402e-01,   2.90117096e-16,
         -1.20947571e+01,   8.59069264e-01],
       [  7.33088462e+05,   8.08987796e-01,   3.06803433e-16,
         -2.65650512e+01,   8.72066511e-01]])

To better visualize the result, we can plot the output:

The following code was used to generate the plot:

import matplotlib.pyplot as plt
labels = 'rel.power abs.power baz slow'.split()

fig = plt.figure()
for i, lab in enumerate(labels):
    ax = fig.add_subplot(4, 1, i + 1)
    ax.scatter(out[:, 0], out[:, i + 1], c=out[:, 1], alpha=0.6,
               edgecolors='none')
    ax.set_ylabel(lab)
    ax.xaxis_date()

fig.autofmt_xdate()
fig.subplots_adjust(top=0.95, right=0.95, bottom=0.2, hspace=0)

Pricing

Creating environments is free. There are no additional charges associated with this feature. From our perspective, environments allow you to run more cycles on PiCloud, which is where we get our pay off.

Conclusion

With Environments, every programming language and software package is now a viable tool to use on PiCloud. While we love Python, we’re excited that it is no longer the “sole language of the PiCloud Platform.” Coupled with our function publishing feature, we envision Python serving as the glue language through which users can orchestrate their computation on our otherwise language-agnotistic platform.

Thanks to the select group of users who beta tested this feature for us, and to Ken Park from PiCloud who envisioned and shepherded this project to launch!

1. To call Python functions from a programming language other than Python. For example, if you’re integrating the PiCloud platform into a Java codebase, or even into a smartphone application (Android or iPhone).
2. To use PiCloud from Google AppEngine, since our cloud client library is not supported on GAE.
3. Because you’re tired of setting up web application projects when what you really need is a scalable RPC system.

In this post, we’ll give you your first taste of publishing functions on the web.

Define your Function

Just like when you offload regular computation to PiCloud, feel free to do anything in your function including importing custom libraries and making external connections.

def add(x, y):
    """This function adds!"""
    return x+y

Publish It

>>> import cloud
>>> cloud.setkey(key, secret_key)
>>> cloud.rest.publish(add, 'addition')
'https://api.picloud.com/r/2/addition'

The first argument, add, is your function. The second argument, addition, is a label so you can reference the function later; it’s also present in the returned URL for clarity. For a list of all other arguments, refer to the cloud.rest module documentation.

Let’s get information about the function we just published by making a GET request on the returned url. We recommend curl to do this from a shell. We authenticate requests using basic authentication. In curl, use “-u” as shown below to specify your key as your username, and secret key as your password. Note that we automatically extract the function’s doc string as the description.

$ curl -k -u 'key:secret_key' https://api.picloud.com/r/2/addition/
{"output_encoding": "json", "version": "0.1", "description": "This function adds!", "signature": "addition(x, y)", "uri": "https://api.picloud.com/r/2/addition", "label": "addition"}

You can also see your published functions from your account control panel.

Call the Published Function

Now let’s call the function by using a POST request to the same URL. To specify arguments to the function add, you simply pass them in as JSON encoded POST values. In this case, you would specify the POST values x and y.

$ curl -k -u 'key:secret_key' https://api.picloud.com/r/2/addition/ -d x=1 -d y=1
{'jid': 809730}

Get the Result

There are two ways we can grab the result of this job. The standard way is through your Python console:

>>> import cloud
>>> cloud.setkey(key, secret_key)
>>> cloud.result(809730)
2

The language-agnostic way to do this using our REST API is to query the following URL: https://api.picloud.com/job/{job_id}/result/.

$ curl -k -u 'key:secret_key' https://api.picloud.com/job/809730/result/
{"result": 2}

The difference between these two methods is that cloud.result will block until the result is ready; our REST API will return a “job not done” error, so you’ll have to keep querying until it’s ready.

For a full specification of our API, please see our REST API documentation.

Taking Advantage of JSON Arguments

Since arguments are specified as JSON, you can easily pass in strings, lists, and dictionaries into your published functions. For example, we can concatenate two strings using our addition function:

$ curl -k -u 'key:secret_key' https://api.picloud.com/r/2/addition -d "x=\"Hello, \"" -d "y=\"World\""
{'jid': 809731}
$ curl -k -u 'key:secret_key' https://api.picloud.com/job/809731/result/
{"result": "Hello, World"}

We can also merge two lists using our addition function:

$ curl -k -u 'key:secret_key' https://api.picloud.com/r/2/addition -d "x=[1,2,3]" -d "y=[4,5,6]"
{'jid': 809732}
$ curl -k -u 'key:secret_key' https://api.picloud.com/job/809732/result/
{"result": [1, 2, 3, 4, 5, 6]}

These work, of course, because in Python the addition operator can be applied to strings and lists, not just numbers.

Handling Raw Data

JSON does not natively support binary data. While you can encode the data to base64, and decode it in your function, we offer a more straightforward and efficient method. Binary data can be passed into a published function by using multipart/form-data as a file upload (MIME Content-Disposition sub-header has a filename parameter).

Example

To showcase raw data handling, we’re going to publish a function to create thumbnails. We’ll use this picture of Albert Einstein.

Here’s the function we’ll use to create a thumbnail of an image. We use StringIO so that we can open and save the image in a memory buffer, rather than to a file.

from PIL import Image
from cStringIO import StringIO

def thumbnail(raw_img_data, width=50, height=50, output_format='JPEG'):
    im = Image.open(StringIO(raw_img_data))
    im.thumbnail((width, height))
    out_data = StringIO()
    im.save(out_data, output_format)
    return out_data.getvalue()

import cloud
# be sure to set the output encoding to raw
cloud.rest.publish(thumbnail, 'thumbnail', out_encoding='raw')

Call the function. Use -F in conjunction with the @ symbol to POST an image file as a file upload, which will be treated as raw data by PiCloud. We can adjust the width and height by passing in POST values, or if we omit them, the default value of 50 will be used.

$ curl -k -u 'key:secret_key' -F width=60 -F height=76 -F "raw_img_data=@albert_einstein.jpg" https://api.picloud.com/r/2/thumbnail/
{'jid': 809737}

The content of the result is the binary data representing the thumbnail image. Unlike JSON encoded results, there is no enclosing dictionary. Thus, all you have to do to see the image is pipe the result of the job into a file.

$ curl -k -u 'key:secret_key' https://api.picloud.com/job/809737/result/ > albert_einstein.thumb.jpg

Open the thumbnail in your favorite image program!

Conclusion: Take a rest, and then give it a spin!

We’re particularly excited by function publishing because it bridges PiCloud with the world outside of Python, and in doing so, brings all the computing benefits of our standard service. You can publish functions without any care for the amount of hardware running underneath. As your functions get called more frequently, we automatically scale our servers to meet demand. You can also reserve real-time cores if they want to guarantee a certain number of cores at all times. Lastly, you can be confident that your computation is being run on a system built with performance, robustness, and redundancy at its core.

If this technology captivates you, follow us on Twitter, or go above and beyond and join our team!

You can now select the type of core you want your job to be run on.

• c1: Replaces our standard option as default.
• c2: Replaces the high cpu option.
• m1: Our new high-memory core with 3.25 compute units and 8GB of memory.

For more details, see our updated pricing page.

How to Use It

We’re committed to maintaining an extraordinarily simple API for you. With our old library you would do the following:

cloud.call(func, _high_cpu=True)

With our new library (available here), you do this instead:

cloud.call(func, _type='c2')

Additional Information

In conjunction with these changes, real-time cores are now reserved by type. You can see the new interface in your control panel.

We’ll be releasing more cores as we hear demand for them. Our next core, s1, which is beta, is a solution for users on our platform who scrape the web. When running jobs in parallel on s1 cores, each job will have its own IP, minimizing throttling effects. However, consecutive jobs may share the same IP address.