New Base Environment — Ubuntu Precise

October 24th, 2012 by Ken Elkabany

Since its introduction, Environments have become a staple feature of the PiCloud Platform, enabling users to deploy custom libraries and binaries. It was our first step outside of the Python box.

Up until now, we’ve offered two base environments that you could customize, or use directly:

  1. Ubuntu 10.10 Maverick for Python 2.6
  2. Ubuntu 11.04 Natty for Python 2.7

Enter Ubuntu Precise 12.04

Our latest environment is pre-configured with many of the latest libraries, making it easier than ever to move your computation to the cloud. Here are some of the notable packages:

  • NumPy 1.6.2
  • SciPy 0.11
  • Pandas 0.9.0
  • Scikits Learn 0.8.1
  • OpenCV 2.4.2
  • Java 7
  • R 2.14.1
  • Ruby 1.9.1
  • PHP 5.3.10

Click here for a list of all contents.

How do I use the new Base?

To maintain backwards compatibility with users dependent on our Natty base, by default all Python 2.7 jobs still use Natty. To use Precise, specify the environment of a job as ‘base/precise’. In Python: 

cloud.call(f, _env='base/precise')

In the shell: 

$ picloud exec -e base/precise program

Of course, you can create an environment based off of the Precise base, and use that too.

Need Another Base?

If Precise isn’t enough for you, let us know what other distribution of Linux you’d like to see us support in the comments.

Tags: , ,
Posted in What's New | 3 Comments »

Real-time Data Feed for Jobs and More!

September 18th, 2012 by Ken Elkabany

We’ve just completed a major overhaul of the job dashboard. The overhaul marks a fundamental shift in our belief of what information you should be able to see about your job. We had originally striven for minimalism, revealing only what we thought was necessary: standard output & error, exception traceback, and profile. After all, if software ran bug free, you wouldn’t need anything else. But as our users kept reminding us with support tickets, when things aren’t working, there is exceptional value to under-the-hood data.

This post covers the new visibility we offer into your computation. You can see it in action by running a new job, and viewing it through the Job Dashboard.

Count of Memory Allocation Failures

We now report the number of times a memory allocation was requested, but failed, under the “MemAlloc Fails” column.

If you haven’t run out of memory on PiCloud before, the count may not seem all that important. But up until now, users would generally get a cryptic “None” Exception when they ran out of memory, indicating that their process had unexpectedly died. The reason is because most programs don’t handle out of memory errors gracefully, typically segfaulting instead. Now if your program crashes, it’s easy to check for allocation failures, the most likely culprit.

Once you know you’re running out of memory, you can take advantage of our other features. Try using a core type with more RAM, such as our f2 core, or take advantage of our new multicore support.

CPU Usage

A common question is how effectively a job is utilizing a CPU. If your job is supposed to be crunching numbers, but is only using the CPU 30% of its allotted time, then you probably have an unsuspecting bottleneck. Before this update, there was really no way to know, other than profiling the job locally.

Now, you can access three different views of a job’s CPU usage to give you maximum visibility.

Total Usage

Runtime refers to wall-clock time. CPU Time is divided into two categories based on whether time is being spent in user or kernel space. With the data above, we can determine that the CPU was being utilized (7613.14+280.85)/8239.6 = 95.8% of the time. But where’d the other 4% go?

Usage over Time

Using the following chart, we can see what happened.

This hypothetical job was at 99%+ utilization for the majority of the time. However, it spent the first 500 seconds loading data from external sources before crunching it. Depending on your job, a graph like this might look ideal, or it might be an impetus to reduce the data loading time.

The chart is generated by taking a snapshot of a job’s CPU Time every 30 seconds.

Realtime Usage

If you select a job from the Job Dashboard, while it is processing, you’ll be able to see a realtime graph of CPU usage in the right hand column. The graph actively samples your running job every few seconds.

This soothes the anxiety some developers feel when their precious algorithms are all grown up and running on the cloud. Never again will they anxiously wonder, “how’s my job doing?”

Memory Usage

We offer a similar set of views for a job’s memory usage, as we do for its CPU usage. While swap usage is shown, no job can currently use swap.

Peak Usage

Peak usage is the most amount of memory that was ever allocated for the job. Peak usage is viewable while the job is running, or after it has completed.

Current Usage

Current usage is the amount of memory currently being used by the job. Current usage can only be seen for a job that is currently being processed.

Usage over Time

Extending the example of the hypothetical job from the CPU Usage section, we can confirm that the first 500 seconds were spent loading data into memory.

Realtime Usage

Not shown for brevity. Just like the Realtime CPU Usage chart, you can see the memory usage of a job, while it is processing.

Python Logging

The Python logging module is popular for its simplicity and extensibility. Until now, users have had to output their loggers to standard output or error to have their messages retained by PiCloud. We now automatically grab your logger messages, and store them separately for later viewing.

Pi Log

As your job runs, our system sometimes has comments for you. For example, if you spawn subprocesses, it will notify you that you cannot take advantage of our persistent processes optimization. Before, we used to write these messages to standard error, but that unnecessarily worried some users, and others complained that it polluted their output. Now, we have a separate section specifically for messages from PiCloud.

System Log

Since more users are using PiCloud for non-Python computation, we have decided to expose the Syslog associated with a job. If your job is segfaulting, and there are no indications why, this is the place to look.

In the Works

We plan to offer metrics for disk, and network usage.

Suggestions?

Need more visibility? Let us know by leaving a comment!

Tags: ,
Posted in What's New | No Comments »

Case Study: Aggregating Daily Content from 500+ Sites

September 13th, 2012 by Ken Elkabany

We know that PiCloud is primarily associated with scientific workloads, and high-performance computing. So it may be surprising to hear that many of our earliest adopters were in fact web companies who aggregate content from all over the web. In fact, the s1 (scraping) core was in direct response to the needs of this class of users.

Zinc.TV (Division of TiVo) signed up only a couple of months after our first release. They’ve built an awesome service they call your “Internet Television Dashboard.” It’s a central location where you can search for any TV show or movie, and it’ll show you all the online sources–both free and paid–that offer it. They even have a “leanback” mode for when you aren’t feeling particularly decisive.

Zinc.TV maintains its comprehensive show catalogue by regularly scraping over 500 web properties using PiCloud. They take advantage of PiCloud’s easy parallelism for aggregating and curating data, and our interoperability with other clouds; Zinc.TV stores all their data in their own databases hosted on Amazon Web Services directly from PiCloud.

Gary Rose, co-founder and head of R&D at Zinc.TV, had the following kind words to share:


“PiCloud has been an extremely valuable partner for the growth of Zinc. We estimate that it cuts our operational costs for managing the infrastructure by over 50%, and allows us to release new sites in a fraction of the time.”

Gary highlights what we agree to be the core benefit of PiCloud: drastically reducing the man hours that go into “behind the scenes” work such as creating a robust, scalable distributed processing system, thereby increasing the time for unique, interesting work. We’re popular with researchers because we automate clusters so they can focus on their science. The same principle applies to all types of technological endeavors reliant on computational power.

You can download the full Zinc.TV case study, or view it in your browser below.

Tags: , , ,
Posted in Case Study, Success Story | No Comments »

Introducing Multicore Support

August 31st, 2012 by Ken Elkabany

Up until today, each job run on PiCloud has been limited to using only a single core. For those familiar with the Python global interpreter lock (GIL), this may not seem like a big deal at first. But as our users have let us know, the limitation is acute for the following reasons:

  1. Many performance-focused Python libraries including numpy release the GIL whenever possible, which means that even Python programs can leverage multiple cores.
  2. With the release of Environments, many of our users are running non-Python multithreaded programs. Some of those can use as many cores as we can throw at it.
  3. The most RAM a single core has is 8GB (m1). Up until now, a single job couldn’t break this limit. But now, you can pool multiple cores together to get access to more RAM.

How do I use it?

All you have to do is use the _cores keyword argument. 

# uses 4 cores
job_id = cloud.call(func, _type='f2', _cores=4)

# works the same for map jobs
job_ids = cloud.map(func, datapoints, _type='f2', _cores=4)

Each job gets 4 f2 cores of processing power, and 14.8GB (4 cores x 3.7GB per f2 core) of RAM. We use the f2 core because as the next section shows, the c1 core, which is default, does not support the new multicore feature.

How many cores per job?

The number depends on the type of core you select.

Core Type Supported Multiples
c1 1 core
c2 1, 2, 4 or 8 cores
f2 1, 2, 4, 8, or 16 cores
m1 1 or 2 cores
s1 1 core

How much?

Per our pricing page, a job using a single f2 core would cost $0.22/hour. A job using two f2 cores would cost $0.44/hour. In other words, the cost per core has stayed the same, and there are no additional fees. You’re still charged by the millisecond.

Suggestions?

Multicore not enough for you? Let us know by leaving a comment.

Tags: ,
Posted in What's New | 1 Comment »

Introducing the f2 Core – Our Fastest Core Yet!

June 15th, 2012 by Ken Elkabany

If you’ve been craving more speed, you’ll want to give our brand new f2 core a spin!

How fast?

The f2 core is rated at a whopping 5.5 compute units, which is 69% more than our previous leader, the m1 core with 3.25 compute units.

How about memory?

f2 cores have access to 3.7 GBs of RAM, which is a much-needed middle ground between the 800 MBs available in a c2 core, and the 8 GBs available in an m1 core.

How do I use it?

All you’ll need to do is set the _type keyword. 

cloud.call(func, _type='f2')

How much?

At $0.22/core/hour, it’s got 120% more compute units than a c2 core, but only a 69% price premium.

Suggestions?

With our collection of five core types, there’s something for everyone! If you have ideas for another core type, or want to tell us how important a core with a GPU is to you, please leave a comment.

Tags: , ,
Posted in What's New | 6 Comments »

Happy Pi Day!

March 14th, 2012 by Ken Elkabany

π is dear to our hearts. Not only is it our favorite transcendental number, and makes up the first syllable of our company name, but it is also a homophone of the first syllable of our favorite programming language. So to celebrate this glorious day, we’re giving out free core hours.

All you have to do is send us a function that can calculate π to arbitrary digits, and runs on PiCloud. We’ll be awarding three users 100, 200, and 300 core hours based on how awesome we think their solutions are.

Need inspiration? Here’s a monte carlo method for calculating π in our documentation. Also, the Wikipedia article for Numerical Approxiations of π should come in handy.

Send your submissions to [the first 314 digits of π starting with 3 and without the .]@picloud.com. Happy coding!

Posted in Just for Fun | 2 Comments »

Improved Pricing Scheme for Realtime Cores

February 29th, 2012 by Ken Elkabany

If you’ve been using realtime cores to leverage hundreds of cores in parallel, this is the cost-reducing update you’ve been waiting for. We’ve implemented the pricing change outlined in our previous post.

What’s changed?

At first glance, our realtime core prices look like they’ve been bumped up significantly. A realtime c1 core used to be priced at $0.015/hour, and now it’s $0.03/hour. Don’t be alarmed! The difference is that the old price was an additional charge added on top of your computation bill. The new price is not a charge, but an hourly minimum.

A simple example

For a single c1 realtime core, if you use at least $0.03 (the hourly minimum) of computation in a single hour, you do not pay anything extra by having the realtime core. We reserved a core for you, and since you put it to good use, you get it for free. If you don’t use as much as $0.03 of computation in the hour, we adjust your computation bill so that you paid a total of $0.03. In other words, in exchange for us reserving a core for you, you agree to pay at least the hourly minimum whether it’s by the amount of computation you run, or by an adjusted bill.

In our old scheme, if you ran $0.03 worth of computation, you would pay $0.03 + $0.015 = $0.045. In this case, our new scheme saves you 33% of your bill!

Why is this great?

If you’ve been using hundreds of realtime cores on PiCloud, you understand the pain that this solves. Instead of paying $0.015 per core per hour on top of your computation bill, you can now reserve hundreds of realtime cores for your parallel processing needs with potentially no additional fee.

How much do I need to utilize my cores for them to be free?

Just take the ratio between the core type’s realtime hourly minimum, and the standard price. For a c1 core, that’s $0.03/$0.05 = 60%. In other words, you need to be utilizing the core for 60% (36 minutes) of the hour to hit the minimum hourly bill.

How do I control which jobs are scheduled to realtime cores?

We automatically schedule your jobs to the appropriate cores. In fact, if we have slack capacity, you may utilize more cores in parallel than the number of realtimes cores you have, and those extra cores will still count towards your hourly minimum.

How will this change be phased in?

For this month, and the month of March, users who use realtime will be given the cheaper of the old and new pricing schemes.

Posted in Realtime Cores | No Comments »

How to Get More Parallel Computing Power

November 16th, 2011 by Ken Elkabany

We often get asked how much parallel computing power each user on our platform gets. The short answer is that it’s variable and left up to our algorithms by default, but if you need, it’s easy to specify that you want exactly N cores using our Realtime Cores feature—a couple of clicks on our web dashboard, or one line of code.

Standard Service (w/o Realtime)

By default, your jobs take advantage of cores that are shared by all of our users. We do not have a set number of total cores in our system, and there are no set number of cores per user. Instead, PiCloud continually and automatically estimates the workload we have and will have in the next hour. There are two components to this estimation:

  1. Periodic Jobs: A large class of users runs jobs periodically, whether it’s every minute, hour, or day. Over time our system learns the amount of load each of these users contributes, and increases server capacity in anticipation of periodic jobs.
  2. Aperiodic Jobs: For users who do not have a predictable pattern, we scale our worker nodes once their jobs have been added to our queue. The number of workers we bring up depends on how long we estimate their jobs will take.

How do we estimate the runtime of a job?

We estimate the runtime of enqueued jobs by the average and variance of runtimes of similar completed jobs; a similar job is any job by the user that executed the same Python function with the same PiCloud-specific reserved keyword arguments. Keyword arguments such as _type are important since they could affect the speed at which a job is processed.

How does your workload affect the number of workers we bring up?

Since we use Amazon Web Service’s servers, we are charged for every hour we have a server up. To manage our costs, we try to make sure our servers spend as much time as possible processing jobs, rather than sitting idle. Here’s a rule of thumb for how this affects you: If each of your jobs takes an hour to process, PiCloud will automatically scale such that all of your jobs are running simultaneously in parallel. We’re happy to bring up as many as you need since you’re keeping our servers busy for the full hour increment that we rent from Amazon. If each of your jobs takes a half hour, we’ll run about half of your jobs in parallel at a time, so that each core will be busy for the full hour running two jobs each. In practice, our scaling algorithm is more liberal than this, but this should give you a good conservative ballpark.

How much compute power should you expect?

If you’re just running a small batch of short jobs, you’ll generally get less than 10 cores. If you’re running a large number of long-running jobs, then using the above algorithm, we could potentially bring up hundreds of cores for you.

Realtime Service

The model of our standard service is ideal for users trying out PiCloud for the first time, users with relaxed response-time requirements, and users with large batch jobs. Our realtime service is for our other class of users who need N cores to be ready at a moment’s notice, even if that means those cores will be sitting idle at times.

How it Works

You tell us the type of core you want, and the number you want, and we’ll bring them online just for you. Your jobs will be placed in both your own realtime queue, as well as our general queue. So if you have reserved 80 cores, and you check in 200 jobs, at least 80 jobs will be immediately be dequeued and begin processing on your realtime cores. Potentially more than 80 may run simultaneously since you still have access to the cores available in our standard service.

How we (want to) Charge

Right now, we charge a fixed cost for every core hour (rounded up). If you reserve realtime cores, and you run no jobs, you’ll be charged that amount. If you run jobs, you’ll be charged both the cost of the realtime cores, as well as the cost of our standard service for the number of compute hours your jobs run for. For example, if you reserve 10 c1 cores for an hour, and run 1 compute hour worth of jobs on c1 cores during that time, you’ll pay $0.15 + $0.05 = $0.20 cents total (See our pricing page). The problem we have with this is two fold:

  1. The fixed cost of each realtime reservation needs to be sufficiently high to cover our expenses in the event a user does not run any jobs. Charging our elastic rate on top of that is unfairly expensive.
  2. If our standard service scaled a user to 100 cores, and the user then reserved 30 cores, they would not see any additional compute power, while paying substantially more. While it’s true that they would be guaranteed that compute power without any risk of crowding from other users, we don’t believe users were getting the best bang for their buck.

We love users who use realtime because they offer our automatically-scaling system a level of predictability; we don’t want to dis-incentivize this behavior. So we’re going to change our policy as follows: Each realtime core, rather than being an hourly charge, will instead be an hourly minimum. Hypothetically, if you reserved a realtime core that has a minimum of $0.10/hour, then after an hour you would be charged the greater of $0.10 or the bill you accrued from using our standard service. If you don’t run any jobs, that’s $0.10. If you run some jobs, but the standard service bill doesn’t exceed $0.10, then you pay $0.10. If you run enough jobs to exceed $0.10, say $0.15, then you pay just $0.15. We’ll let you all know when we’ve made this change to our pricing!

How to Reserve Realtime Cores

Go to the realtime tab on our web dashboard.

Select the type of core, and the number of cores you want for us to keep available for you, and then hit submit.

When the cores are ready, you’ll receive an email and the “start time” will be set.

When you run jobs, make sure that you set the _type keyword to the core type that you have reserved. In this example, we reserved 160 c2 cores. 

cloud.call(f, _type='c2')

Alternatively, reserving realtime cores will be possible through the Python console using the cloud library in its next release: 

cloud.realtime.request('c2', 160)

How long does it take to allocate Realtime Cores?

We’ve been working hard making allocation time faster and more consistent, which has been a common issue raised by users. With our latest design, it should take no more than 15 minutes for cores to be provisioned!

How many cores can you allocate?

We have users who provision hundreds and thousands of cores using our realtime service. If you’re worried you might break our backs, file a support ticket and let us know what you’re up to.

Conclusion

The takeaway we want to leave with you is as follows: The PiCloud platform will not only get you up and running in no time with no configuration, but will also be able to meet your exact computational requirements if necessary—even if you need 1000 cores in 15 minutes. It’s not a question of if, but how, and we’re working to make how as easy as possible.

Tags: ,
Posted in How To, Realtime Cores | 1 Comment »

Academic Research Program Grant Recipients

November 4th, 2011 by Ken Elkabany

We are excited to announce the recipients of the $500 (10,000 c1 core hours) Academic Research Program Grant. While we had originally planned to award only two grants, we were pleasantly surprised by the level of interest our program received from highly-qualified individuals. In response, we decided to award nine grants!

We’ve included a list of recipients below so you can see some of the great work that will be benefiting from our platform.

If you missed out on our first cycle of grants, don’t sweat! We plan on having a second cycle in the near future.

List of Recipients

Chris Beaumont
PhD candidate at the Institute for Astronomy
University of Hawaii at Manoa

Developing an automated technique to search astrophysical images for “bubbles” blown by massive young stars. This will help astronomers handle their rapidly growing datasets, which we currently examine by eye when looking for morphologically complex objects.

Damian Borth
PhD candidate
University of Kaiserslautern
German Research Center for Artificial Intelligence (DFKI)

Video concept detection using web-based training sources like YouTube. I particularly focus my interest on developing learning approaches which adopt to weak labels and deal with the domain change problem.

Liang Yuxian Eugene
Research Assistant
National Cheng Chi University

Modeling the Lifecycle of Natural Disasters.

Patrick Henaff
Associate Professor of Finance
Institut d’Administration des Entreprises, Université Paris I Panthéon-Sorbonne

Motivated by the current financial crisis, my research focus is to understand and measure model risk in computational finance, ie. the risk associated with the use of an inappropriate model to price and manage financial derivatives. Inspired by other scientific disciplines, in particular computational biology, this research project will also experiment with a new type of scientific production, based on the publication and peer review of of self-contained, reproducible experimental protocols, as opposed to summary results.

Roy Keyes
Phd candidate in the Department of Physics and Astronomy
University of New Mexico

Developing faster, more accurate methods of calculating radiation dose to improve cancer treatment along with associated research tools.

Meredith Lehmann
La Jolla High School

SARS and the swine flu rekindled interest in the spread of epidemics and a large literature appeared to conclusively prove the intuitive proposition that modern epidemics spread disproportionately through large hub airports near large population centers. My research challenges the conventional wisdom using a different model of the US transportation network. A different picture of epidemic propagation in the continental US emerges from this analysis. Primarily auto, not air, travel seeds counties with a modest number of infecteds and the subsequent explosion at the county level dominates further seeding by infected travelers. No preferred epidemic pathways arise in my simulated world and may not be present in the real world either.

Eric Lofgren
PhD candidate in the Department of Epidemiology, Gillings School of Global Public Health
University of North Carolina at Chapel Hill

The mathematical modeling and simulation of the transmission of Clostridium difficile in healthcare settings.

Ariel Rokem
Post-Doctoral Researcher at the Department of Psychology under Professor Brian Wandell
Stanford University

Using Magnetic Resonance Imaging (MRI) to measure physiological and anatomical properties of the human visual system. In particular, trying to understand the biological underpinnings of differences in perception between the center of the visual field and the periphery.

Thomas Wiecki
PhD candidate under Professor Michael J. Frank
Brown University

My research aims at elucidating the cognitive and neurobiological mechanisms underlying decision making in the healthy and diseased brain. Towards this goal I am simulating parts of the intact brain in the computer and model how different brain lesions and psychiatric diseases influence brain activity and behavior. Ultimately, this research will help in developing better diagnosis and treatment options for brain disorders such as schizophrenia and Parkinson’s disease.

Tags:
Posted in Academic Research Program | 1 Comment »

20 Free Core Hours Every Month!

October 24th, 2011 by Ken Elkabany

Starting this month, all users will get 20 c1 core hours worth of credits each and every month.

  • If you ran out of your original 5 core hour credits, you can come back and play around some more!
  • If you have minimal computing needs, this means that you can now use PiCloud regularly without even having to enter a credit card.

Looking for more? Don’t forget, we’re giving away $500 worth of credits as part of our Academic Research Program. Applications are due this Thursday, October 27th.

Tags:
Posted in What's New | 4 Comments »

« Older Entries
Newer Entries »